Announcement Announcement Module
No announcement yet.
Writing Invalid record to a flat file and valid records to database. Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Writing Invalid record to a flat file and valid records to database.

    My requirements:
    Reading flat files in a directory and processing then with a custom validator and importing valid records in database.
    I also want to introduce one more step in which,
    i want to write rejected/invalid records in a flat file and if possible valid records to a flat file along with db.

    <bean id="multiResourceReader"
    		<property name="resources" value="file://${}" />
    		<property name="delegate" ref="reader" />
    		<bean id="inputFile"
    		class="" scope="step">
    		<constructor-arg value="#{jobParameters[inputFile]}" /> </bean>
    	  <!--  Validation -->
    		<bean id="myValidationProcessor"  
    		class="org.springframework.batch.item.validator.ValidatingItemProcessor" >
    		<property name="filter" value="false"/>
    		<property name="validator" ref="myValidator"/>
    		<bean id="myValidator"  class="com.batch.validation.myAccountValidator" />
    	<!--  Reading CSV file -->
    	<bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader" 		scope="step">
    	 <!--  <property name="resources" value="file://${}" />-->
    	  <property name="linesToSkip" value="1" /> <!--  For headers to be skipped -->
    		<property name="lineMapper">
    			<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
    				<property name="lineTokenizer">
    						<property name="names"
    							value="Entity Type,Account Type,Number of Account Holders,Entity Name,Account Number,Account Holder Name" />
    				<property name="fieldSetMapper">
    					<bean class="com.batch.solution.MyFieldSetMapper" />
    	<!--  database item writer -->
    	<bean id="writer" class="com.batch.solution.JdbcItemWriter">	
              <constructor-arg ref="dataSource" />
    	 <bean id="fileRenameTasklet"	class="com.cti.batch.tasklets.FileRenameTasklet" > 
    	 	<property name="resources" value="file://${}" />
       <batch:job id="fileLoadJob" job-repository="jobRepository">
             <batch:step id="fileLoadStep" >
                <batch:tasklet  transaction-manager="txManager" >
                   <batch:chunk processor="myValidationProcessor"   reader="multiResourceReader" writer="writer" commit-interval="100" skip-limit="10">
                   <batch:skippable-exception-classes >
    					<batch:include class="org.springframework.batch.item.validator.ValidationException" />
                 <batch:next on="COMPLETED" to="fileRenameStep"/>
             <batch:step id="fileRenameStep"> 
    			  <batch:tasklet ref="fileRenameTasklet" transaction-manager="txManager">	

  • #2

    when ItemProcessors are introduced in the spring batch reference doc chapter 6.5., the concept of a CompositeItemWriter is introduced. So you would have an ItemWriter that delegates writing to a database-writer and a csv-writer.

    Usually, I'd like to stick to simple batch-processing patterns. So, how about writing valid records to a CSV-file and adding an additional step that imports the CSV-file of valid records to the DB? Maybe you could use some DB functionality that imports the whole CSV file rather than using ItemReaders and Writers.

    For invalid items, you could use an ItemProcessListener to write items to a flat file in the method onProcessError(T item, Exception e). You could use the delegation pattern here, too.
    Last edited by BjRi; Mar 8th, 2013, 05:22 AM. Reason: Wrong reference to spring batch doc


    • #3
      There is actually a ValidatingItemProcessor for just this need: