Announcement Announcement Module
Collapse
No announcement yet.
batch extracting same data Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • batch extracting same data

    hi,
    i have a batch that extracts data from a database then write it to a database. But it seems that it's reeating the operation and extracting the same data
    here is my configuration
    Code:
    <beans:bean id="itemReader"
    		class="org.springframework.batch.item.database.orm.HibernateNativeQueryProvider">
    		<beans:property name="entityClass">
    			<beans:value>com.lvmh.efast.domain.CampaignItem</beans:value>
    		</beans:property>
    		<beans:property name="sqlQuery">
    			<beans:value> 
    	 <![CDATA[ 
    	 select
            campaignItem.*
        from
            efast.campaign_item campaignItem 
        inner join
            efast.item_status ci3_ 
                on campaignItem.ITEM_STATUS_ID=ci3_.ID 
        inner join
            efast.item_status_label cii4_ 
                on ci3_.ID=cii4_.ITEM_STATUS_ID     
        inner join
            efast.campaign ccampaign2_ 
                on campaignItem.CAMPAIGN_ID=ccampaign2_.ID     
        where
            cii4_.DESCRIPTION not like :desc
            and ccampaign2_.SAPFM_SENDING= :sap
    	 ]]>
    			</beans:value>
    		</beans:property>
    	</beans:bean>
    	<!--class="org.springframework.batch.item.database.HibernatePagingItemReader" -->
    	<beans:bean id="itemDatabaseReader"
    		class="org.springframework.batch.item.database.HibernatePagingItemReader">
    		<beans:property name="sessionFactory" ref="hibernateSessionFactory" />
    		<beans:property name="parameterValues">
    			<beans:map>
    				<beans:entry key="desc" value="Cancelled%" />
    				<beans:entry key="sap" value="1" />
    			</beans:map>
    		</beans:property>
    		<beans:property name="queryProvider" ref="itemReader" />
    		<beans:property name="useStatelessSession" value="false" />
    		<beans:property name="fetchSize" value="10" />
    		<beans:property name="saveState" value="true" />
    		<beans:property name="pageSize" value="10" />
    	</beans:bean>
    and here is the result in my file
    Code:
    1;a003;000142;201;
    1;a002;000142;201;
    4;a002;000332;200;
    4;a002;000343;201;
    4;a002;G068331;100;
    1;a003;000142;201;
    1;a002;000142;201;
    4;a002;000332;200;
    4;a002;000343;201;
    4;a002;G068331;100;
    i don't understand why?

  • #2
    I don't understand the question yet. You said you are copying database->database, but the output you showed is a file. Your query is quite complex, so are you sure it does what you expect? Did you look out the result in a console or SQL tool to verify that it has the right number of records?

    Comment


    • #3
      hi Dave,

      sorry i was mistaken it's database to file. but with same query used with the hibernateCursorItemReader it extract the without duplication. this happen only when using the hibernate queryProvider.

      Comment


      • #4
        Still not sure I understand. The cursor reader doesn't accept a query in SQL without using the query provider abstraction as far as I recall. So the cursor you used was with a query provider as well?

        Comment


        • #5
          no, with the cursor i use a hql query which is pretty similar as the one above with some changes. Same query but the HQL version

          Comment


          • #6
            Well, you could still try running that query in isolation to make sure it brings back the right results.

            You haven't posted your step or writer configuration. Maybe there's something to look at there?

            Comment


            • #7
              here is my step configuration

              Code:
              <step id="reader2Step">
              		<tasklet transaction-manager="transactionManager">
              			<chunk reader="ItemDatabaseReader" writer="campaignItemFileWriter"
              				commit-interval="10">
              			</chunk>
              		</tasklet>
              	</step>
              and the job configuration
              Code:
              <job id="exportDataJob">
              		<step id="step1" parent="reader1Step">
              			<next on="*" to="step2" />
              		</step>
              		<step id="step2" parent="reader2Step">
              			<next on="*" to="step3"/>
              		</step>
              		<step id="step3" parent="reader3Step">
              			<next on="*" to="step4"/>
              		</step>
              		<step id="step4" parent="reader4Step">
              			<next on="*" to="step5"/>
              		</step>
              		<step id="step5" parent="reader5Step"/>
              	</job>

              Comment


              • #8
                ...and the writer? (Not that I'm expecting to discover anything startling.)

                Comment


                • #9
                  here is my writer configuration

                  Code:
                  <beans:bean id="campaignItemFileWriter"
                  		class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step"
                  		parent="parentFileWriter">
                  		<beans:property name="lineAggregator" ref="cilineAggregator" />
                  		<beans:property name="resource"
                  			value="file:#{jobParameters['path.name']}/${campaign.item.file.name}" />
                  	</beans:bean>
                  
                  <beans:bean id="cilineAggregator"
                  		class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
                  		<beans:property name="delimiter" value=";" />
                  		<beans:property name="fieldExtractor">
                  			<beans:bean
                  				class="com.lvm¨h.efast.batch.orion.extractor.campaignItemFieldExtractor" />
                  		</beans:property>
                  	</beans:bean>
                  
                  	<beans:bean id="parentFileWriter"
                  		class="org.springframework.batch.item.file.FlatFileItemWriter"
                  		abstract="true">
                  		<beans:property name="shouldDeleteIfExists" value="true" />
                  		<beans:property name="saveState" value="true" />
                  	</beans:bean>
                  thanks

                  Comment


                  • #10
                    Nothing funny there. Is the file size precisely twice the number of records in the input result set? Can you make a test case that reproduces the problem?

                    Comment

                    Working...
                    X