Announcement Announcement Module
Collapse
No announcement yet.
file is not get created Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • file is not get created

    Hi ,

    I have two queues that are montired by seprate queue manager Now when I put 50000 messages those 5000 messages are internally divided into half means each queue holding 2500 messages each..the queue's data looks like these

    RODS.DATA.IN 2500 messages (having diiferent queuee manger)
    RODS.DATA.IN 2500 messages (having different queue manager)


    Now when I try to read the data from both the queues two seprate files get created ,each file having entry of the all 2500 records of a single queue.so this works perfectly, but the issue comes when I put 10,000 records those are internally divided into 5000 but the moment when I try to read them only 1 file of 5000 records get created and the secod file is not created ...!! please guide me ,,,wats wrong..!!


    below is the configuration that is the xml I am using ....

    Code:
    		
    			
    			
    	<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> 
    		<property name="location">
    			<value>config.properties</value>
    		</property>
    	</bean>			
    			
    		
    	<!-- Websphere MQ connection factory config-->
    	<bean id="jmsQueueWriteConnectionFactory"
    		class="com.ibm.mq.jms.MQQueueConnectionFactory">
    		<property name="queueManager">
    			<value>${MQ.QueueManagerWrite}</value>
    		</property>
    		<property name="hostName">
    			<value>${MQ.HostNameWrite}</value>
    		</property>
    		<property name="port">
    			<value>${MQ.Port}</value>
    		</property>
    		<property name="channel">
    			<value>${MQ.Channel}</value>
    		</property>
    		<property name="transportType">
    			<value>${MQ.TransportType}</value>
    		</property>
    	</bean>
    	
    	
    	<bean id="jmsQueueReadConnectionFactory"
    		class="com.ibm.mq.jms.MQQueueConnectionFactory">
    		<property name="queueManager">
    			<value>${MQ.QueueManagerRead}</value>
    		</property>
    		<property name="hostName">
    			<value>${MQ.HostNameRead}</value>
    		</property>
    		<property name="port">
    			<value>${MQ.Port}</value>
    		</property>
    		<property name="channel">
    			<value>${MQ.Channel}</value>
    		</property>
    		<property name="transportType">
    			<value>${MQ.TransportType}</value>
    		</property>
    	</bean>	
    	
    	<bean id="mqConnectionFactory"
    		class="org.springframework.jms.connection.SingleConnectionFactory">
    		<property name="targetConnectionFactory">
    			<ref bean="jmsQueueWriteConnectionFactory" />
    		</property>
    	
    	</bean>
    	
    	<!-- Jms Template to send messages. Used for Unit Testing -->
    	<!--<bean id="jmsTemplate"
    		class="org.springframework.jms.core.JmsTemplate">
    		<property name="connectionFactory">
    			<ref bean="mqConnectionFactory" />
    		</property>
    		<property name="defaultDestinationName">
    			<value>${MQ.DefaultDestinationName}</value>
    		</property>
    	</bean>
    
    	
    	
    	-->
    	
    	
    	<!-- Websphere MQ queue config. This config is used only in the absence of jndi  -->
    	<bean id="odsRequestQueue_dev1" class="com.ibm.mq.jms.MQQueue">
    		<property name="baseQueueManagerName" value="${MQ.BaseQueueManagerName}" />
    		<property name="baseQueueName" value="${MQ.BaseQueueName}" />
    		<property name="targetClient" value="${MQ.TargetClient}" />
    	</bean>
    				
    		<!-- putting data into the queue commented by saral  -->	
    				
    
       <si:channel id="input">
    		
    	</si:channel>
    			
    		 <jms:outbound-channel-adapter id="channel-to-mq"
    			channel="input" destination-name="${queueNameWrite}" connection-factory="mqConnectionFactory"  />  
    							
    			<!-- Reading from the queue commented by saral -->	 		
    
     	  <jms:message-driven-channel-adapter
    			id="mq-message-listner_QManager_4" channel="aggregator-input-channel_1" connection-factory="jmsQueueReadConnectionFactory"
    			destination-name="${queueNameRead}" concurrent-consumers="${concurrent-consumers}"   />	
    		
    		<jms:message-driven-channel-adapter
    			id="mq-message-listner_QManager_3" channel="aggregator-input-channel_2" connection-factory="jmsQueueWriteConnectionFactory"
    			destination-name="${queueNameRead}" concurrent-consumers="${concurrent-consumers}"   /> 
    
    		
    
    	<!--   <si:gateway id="gateway" service-interface="com.sample.agg.Gateway"
    		default-request-channel="aggregator-input-channel">
    	</si:gateway> -->
    	
    	<si:channel id="aggregator-input-channel_1">
    		<si:queue capacity="${queueCapacity}" />
    	</si:channel>
    	
    	<si:channel id="aggregator-input-channel_2">
    		<si:queue capacity="${queueCapacity}" />
    	</si:channel>	
    	
    	<si:aggregator id="aggregator_1" input-channel="aggregator-input-channel_1" output-channel="aggregator-output-channel_1"
    		
    		ref="sampleAggregator" method="aggregateMessagaes"  		
    		correlation-strategy="correlationBean" correlation-strategy-method="correlationStrategy" 
    		release-strategy="releaseStrategyBean" release-strategy-method="releaseStrategy" 
    		message-store="messageStore"
    		order="1" 
    		send-partial-result-on-expiry="true"		
    		send-timeout = "5000"		>
    	</si:aggregator>
    	
    	<si:aggregator id="aggregator_2" input-channel="aggregator-input-channel_2" output-channel="aggregator-output-channel_2"
    		
    		ref="sampleAggregator" method="aggregateMessagaes"  		
    		correlation-strategy="correlationBean" correlation-strategy-method="correlationStrategy" 
    		release-strategy="releaseStrategyBean" release-strategy-method="releaseStrategy" 
    		message-store="messageStore"
    		order="1" 
    		send-partial-result-on-expiry="true"		
    		send-timeout = "5000"
    		>
    	</si:aggregator>
    	
    	
    	<bean id="sampleAggregator" class="com.walgreens.ods.producer.Aggregator" scope="prototype" ></bean>   
    	
    	<bean id="correlationBean" class="com.walgreens.ods.producer.CorrelationBean" scope="prototype" />
    	
    	<bean id="releaseStrategyBean" class="com.walgreens.ods.producer.ReleaseStrategyBean" scope="prototype" > 
    	<property name="recordLength" value="${recordLength}" />	
    	</bean>
    	
    	 <si:channel id="throwAwayChannel"	>		
    	<si:queue capacity="${queueCapacity}" />
    	</si:channel> 
    	
    
    		<file:outbound-channel-adapter channel="aggregator-output-channel_1"
    			directory="${aggregatorOutputDirectoryPathtemp1}" filename-generator-expression="${outputFileNameExpression}" temporary-file-suffix="_swp"
    			   />	
    		<file:outbound-channel-adapter channel="aggregator-output-channel_2"
    			directory="${aggregatorOutputDirectoryPathtemp2}" filename-generator-expression="${outputFileNameExpression}" temporary-file-suffix="_swp"
    			   />				   
    
    		<!--  <file:outbound-channel-adapter channel="throwAwayChannel"
    			directory="${aggregatorOutputDirectoryPath}" filename-generator-expression="'discard-'+headers.getTimestamp()+'.xml'"
    			temporary-file-suffix="_swp"   />	-->  
    	
    	
    	<si:channel id="aggregator-output-channel_1">
    		<si:queue capacity="${queueCapacity}" />
    	</si:channel>
    	
    	<si:channel id="aggregator-output-channel_2">
    		<si:queue capacity="${queueCapacity}" />
    	</si:channel>
    	
    	
    	<task:scheduler id="scheduler" />
    	
    	<!--  
    	<task:scheduled-tasks scheduler="scheduler">
    		<task:scheduled ref="reaper" method="run" fixed-rate="${reaperSchedulerFixedRate}" />
    	</task:scheduled-tasks> 
    	
           --> 		
    
    	<bean id="reaper" class="org.springframework.integration.store.MessageGroupStoreReaper">
    			<property name="messageGroupStore" ref="messageStore"  />
    		 	
    		<property name="timeout" value="${reaperTimeOut}" />
    		
    	</bean>
    	
     <bean id="messageStore" class="org.springframework.integration.store.SimpleMessageStore" />
    	
     	
    	<!--<si:service-activator input-channel="aggregator-output-channel" ref="printerService" method="printString"  ></si:service-activator>-->
    	
    	<!--  <bean id="printerService" class="com.sample.agg.PrinterService"></bean>-->
     	
    
    	<si:poller max-messages-per-poll="1" id="defaultPoller" default="true">
    		<si:interval-trigger interval="${pollerInterval}" />
    	</si:poller>
    	
    <!-- <si:inbound-channel-adapter id="test" ref="exampleReleaseStrategy" method="releaseStrategy" >
    <si:poller fixed-rate="5000"/>
    </si:inbound-channel-adapter> --> 	
    	
    
    </beans>
    and the properties file is .....

    Code:
    # Spring bean values
    
    #explicitly reading from seprate , queue name is decided after observing the messages  in queuezee queue by saral
    queueNameWrite=RODS.DATA.OUT
    queueNameRead=RODS.DATA.IN
    #queueNameRead=RODS.DATA.OUT
    
    
    MQ.QueueManagerWrite=DCC2_QM_3
    MQ.QueueManagerRead=DCC2_QM_4
    
    MQ.HostNameWrite=dinfmq03
    MQ.HostNameRead=dinfmq04
    
    MQ.Port=11421
    MQ.Channel=RODS.APP.SVRCONN
    MQ.TransportType=1
    MQ.DefaultDestinationName=RODS.DATA.OUT
    #MQ.DefaultDestinationName=RODS.APP.DATA.IN
    MQ.BaseQueueManagerName=RODS.APP.SVRCONN
    MQ.BaseQueueName=RODS.DATA.OUT
    MQ.TargetClient=1
    
    concurrent-consumers=10
    queueCapacity=100
    recordLength=5000
    #aggregatorOutputDirectoryPath=C:\\ODS_DEV_DIR\\temp\\
    aggregatorOutputDirectoryPathtemp1=C:\\ODS_DEV_DIR\\temp1\\
    aggregatorOutputDirectoryPathtemp2=C:\\ODS_DEV_DIR\\temp2\\
    
    outputFileNameExpression='output-'+headers.getTimestamp()+'.xml'
    reaperSchedulerFixedRate=10000
    #reaperTimeOut=30
    reaperTimeOut=500
    pollerInterval=50
    Header.Source.Key=eiaHdrSource
    Header.Destination.Key=eiaHdrDest
    Header.DataId.Key=eiaHdrDataId
    Header.Route.Key=eiaHdrRoute
    	
    Header.Source.Value=RODS
    Header.Destination.Value=RODS
    Header.DataId.Value=DATA
    Header.Route.Value=LOCAL
    pLesae guide me why the second file is not created....!!

  • #2
    Hi Guys,

    Please guide me ...As I am stuck up..I am doing R&D on this but couldn't reach to the satisfication point..!!

    Comment


    • #3
      Like I said yesterday, do not reply to your own posts because you didn't receive a reply yet.

      Comment


      • #4
        I suggest you turn on debug logging; make sure you include the thread (%t) and timestamp (%d) in your log spec; you should be able to see what's going on.

        I suggest you first do it with a few (one or two) messages in each queue, to get familiar with what the logs show you.

        Then run it with your failing test and look at the logs.

        Comment


        • #5
          Originally posted by Gary Russell View Post
          I suggest you turn on debug logging; make sure you include the thread (%t) and timestamp (%d) in your log spec; you should be able to see what's going on.

          I suggest you first do it with a few (one or two) messages in each queue, to get familiar with what the logs show you.

          Then run it with your failing test and look at the logs.
          Hi Gary Russell,
          Yeah I have configured log4j and logging channel adapter and regularly checking the logs , in the case when I put 5000 records it works perfectly and 2 files containing 2500 records each is being created but in case when I put 10,000 records then In that case only one file of 5000 records is created but the second file containing the remaining 5000 records is not created..!!!

          Comment


          • #6
            Right; you already said that; but can you not figure out the difference between the two tests when looking at the logs?

            Comment


            • #7
              Originally posted by Gary Russell View Post
              Right; you already said that; but can you not figure out the difference between the two tests when looking at the logs?
              well the logs for the second are not being created since the messages from the queue for the second one , that is all the messages are written in file and queue is being emptied but the file is not created...messages are lost ...and this is the case of worry...!!..please check my configuration it mite give any hint ...



              <bean class="org.springframework.beans.factory.config.Pr opertyPlaceholderConfigurer">
              <property name="location">
              <value>config.properties</value>
              </property>
              </bean>


              <!-- Websphere MQ connection factory config-->
              <bean id="jmsQueueWriteConnectionFactory"
              class="com.ibm.mq.jms.MQQueueConnectionFactory">
              <property name="queueManager">
              <value>${MQ.QueueManagerWrite}</value>
              </property>
              <property name="hostName">
              <value>${MQ.HostNameWrite}</value>
              </property>
              <property name="port">
              <value>${MQ.Port}</value>
              </property>
              <property name="channel">
              <value>${MQ.Channel}</value>
              </property>
              <property name="transportType">
              <value>${MQ.TransportType}</value>
              </property>
              </bean>


              <bean id="jmsQueueReadConnectionFactory"
              class="com.ibm.mq.jms.MQQueueConnectionFactory">
              <property name="queueManager">
              <value>${MQ.QueueManagerRead}</value>
              </property>
              <property name="hostName">
              <value>${MQ.HostNameRead}</value>
              </property>
              <property name="port">
              <value>${MQ.Port}</value>
              </property>
              <property name="channel">
              <value>${MQ.Channel}</value>
              </property>
              <property name="transportType">
              <value>${MQ.TransportType}</value>
              </property>
              </bean>

              <bean id="mqConnectionFactory"
              class="org.springframework.jms.connection.SingleCo nnectionFactory">
              <property name="targetConnectionFactory">
              <ref bean="jmsQueueWriteConnectionFactory" />
              </property>

              </bean>

              <!-- Jms Template to send messages. Used for Unit Testing -->
              <!--<bean id="jmsTemplate"
              class="org.springframework.jms.core.JmsTemplate">
              <property name="connectionFactory">
              <ref bean="mqConnectionFactory" />
              </property>
              <property name="defaultDestinationName">
              <value>${MQ.DefaultDestinationName}</value>
              </property>
              </bean>



              -->


              <!-- Websphere MQ queue config. This config is used only in the absence of jndi -->
              <bean id="odsRequestQueue_dev1" class="com.ibm.mq.jms.MQQueue">
              <property name="baseQueueManagerName" value="${MQ.BaseQueueManagerName}" />
              <property name="baseQueueName" value="${MQ.BaseQueueName}" />
              <property name="targetClient" value="${MQ.TargetClient}" />
              </bean>

              <!-- putting data into the queue commented by saral -->


              <si:channel id="input">

              </si:channel>

              <jms:outbound-channel-adapter id="channel-to-mq"
              channel="input" destination-name="${queueNameWrite}" connection-factory="mqConnectionFactory" />

              <!-- Reading from the queue commented by saral -->

              <jms:message-driven-channel-adapter
              id="mq-message-listner_QManager_4" channel="aggregator-input-channel_1" connection-factory="jmsQueueReadConnectionFactory"
              destination-name="${queueNameRead}" concurrent-consumers="${concurrent-consumers}" />

              <jms:message-driven-channel-adapter
              id="mq-message-listner_QManager_3" channel="aggregator-input-channel_2" connection-factory="jmsQueueWriteConnectionFactory"
              destination-name="${queueNameRead}" concurrent-consumers="${concurrent-consumers}" />



              <!-- <si:gateway id="gateway" service-interface="com.sample.agg.Gateway"
              default-request-channel="aggregator-input-channel">
              </si:gateway> -->

              <si:channel id="aggregator-input-channel_1">
              <si:queue capacity="${queueCapacity}" />
              </si:channel>

              <si:channel id="aggregator-input-channel_2">
              <si:queue capacity="${queueCapacity}" />
              </si:channel>

              <si:aggregator id="aggregator_1" input-channel="aggregator-input-channel_1" output-channel="aggregator-output-channel_1"

              ref="sampleAggregator" method="aggregateMessagaes"
              correlation-strategy="correlationBean" correlation-strategy-method="correlationStrategy"
              release-strategy="releaseStrategyBean" release-strategy-method="releaseStrategy"
              message-store="messageStore"
              order="1"
              send-partial-result-on-expiry="true"
              send-timeout = "5000" >
              </si:aggregator>

              <si:aggregator id="aggregator_2" input-channel="aggregator-input-channel_2" output-channel="aggregator-output-channel_2"

              ref="sampleAggregator" method="aggregateMessagaes"
              correlation-strategy="correlationBean" correlation-strategy-method="correlationStrategy"
              release-strategy="releaseStrategyBean" release-strategy-method="releaseStrategy"
              message-store="messageStore"
              order="1"
              send-partial-result-on-expiry="true"
              send-timeout = "5000"
              >
              </si:aggregator>


              <bean id="sampleAggregator" class="com.walgreens.ods.producer.Aggregator" scope="prototype" ></bean>

              <bean id="correlationBean" class="com.walgreens.ods.producer.CorrelationBean" scope="prototype" />

              <bean id="releaseStrategyBean" class="com.walgreens.ods.producer.ReleaseStrategyB ean" scope="prototype" >
              <property name="recordLength" value="${recordLength}" />
              </bean>

              <si:channel id="throwAwayChannel" >
              <si:queue capacity="${queueCapacity}" />
              </si:channel>


              <file:outbound-channel-adapter channel="aggregator-output-channel_1"
              directory="${aggregatorOutputDirectoryPathtemp1}" filename-generator-expression="${outputFileNameExpression}" temporary-file-suffix="_swp"
              />
              <file:outbound-channel-adapter channel="aggregator-output-channel_2"
              directory="${aggregatorOutputDirectoryPathtemp2}" filename-generator-expression="${outputFileNameExpression}" temporary-file-suffix="_swp"
              />

              <!-- <file:outbound-channel-adapter channel="throwAwayChannel"
              directory="${aggregatorOutputDirectoryPath}" filename-generator-expression="'discard-'+headers.getTimestamp()+'.xml'"
              temporary-file-suffix="_swp" /> -->


              <si:channel id="aggregator-output-channel_1">
              <si:queue capacity="${queueCapacity}" />
              </si:channel>

              <si:channel id="aggregator-output-channel_2">
              <si:queue capacity="${queueCapacity}" />
              </si:channel>


              <task:scheduler id="scheduler" />

              <!--
              <task:scheduled-tasks scheduler="scheduler">
              <task:scheduled ref="reaper" method="run" fixed-rate="${reaperSchedulerFixedRate}" />
              </task:scheduled-tasks>

              -->

              <bean id="reaper" class="org.springframework.integration.store.Messa geGroupStoreReaper">
              <property name="messageGroupStore" ref="messageStore" />

              <property name="timeout" value="${reaperTimeOut}" />

              </bean>

              <bean id="messageStore" class="org.springframework.integration.store.Simpl eMessageStore" />


              <!--<si:service-activator input-channel="aggregator-output-channel" ref="printerService" method="printString" ></si:service-activator>-->

              <!-- <bean id="printerService" class="com.sample.agg.PrinterService"></bean>-->


              <sioller max-messages-per-poll="1" id="defaultPoller" default="true">
              <si:interval-trigger interval="${pollerInterval}" />
              </sioller>

              <!-- <si:inbound-channel-adapter id="test" ref="exampleReleaseStrategy" method="releaseStrategy" >
              <sioller fixed-rate="5000"/>
              </si:inbound-channel-adapter> -->


              </beans>
              Last edited by SARAL SAXENA; Nov 22nd, 2011, 11:40 AM.

              Comment


              • #8
                Please STOP posting the same stuff over and over again - you already posted your config above.

                If the messages are being consumed, they are going somewhere.

                Please CAREFULLY look through the log again, and follow a message from each queue from start to finish be sure to have %t in your log spec,

                Is it possible that ...PathTemp1 and ...PathTemp2 point to the same directory and the second file overwrites the first one?

                Comment


                • #9
                  Originally posted by Gary Russell View Post
                  Please STOP posting the same stuff over and over again - you already posted your config above.

                  If the messages are being consumed, they are going somewhere.

                  Please CAREFULLY look through the log again, and follow a message from each queue from start to finish be sure to have %t in your log spec,

                  Is it possible that ...PathTemp1 and ...PathTemp2 point to the same directory and the second file overwrites the first one?
                  Hi Gary,

                  Yeah I will take care of that, well I have tried this option also by creating two seprate directories temp1 and temp2 and in that case file inside temp1 file is created containing 5000 records but in the folder temp2 the remaining 5000 records file is not created ....this doesn't works when you put 10,000 messages and when you try to consume them and writing them into the file , but the thing that I am amazed is that it works perfectly in the case when you put only 5000 messges so some sort of load balancing is required when you increase the quantity of messages and then try to read them..!!

                  Comment

                  Working...
                  X