Announcement Announcement Module
Collapse
No announcement yet.
Writing to Multiple file Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Writing to Multiple file

    Hi,
    My requirment is. I have to read records from a single file. Based on the some business logic I have to write this record to either of the three file.
    1. BCPFile (Valid record)
    2. NewRecordFile (This file will have the new record which doesnot match with the existing record in database)
    3. InvalidRecordFile (File that has invalid record).

    Can I use composite writer here. Or do i need to use some transformer. I tried using composite writer but it writes twice in single file, instead of writing on different file

    In spring batch do we have something that deals with the above scenario

    Its very urjent.

  • #2
    The CompositeItemWriter provided by the framework simply calls all injected ItemWriters for every item, you'll need to implement the dispatching logic yourself in your own composite item writer.

    Comment


    • #3
      Thanks for your reply. But I have a issue with this

      code in CompositeItemWriter is
      public void write(List<? extends T> item) throws Exception {
      for (ItemWriter<? super T> writer : delegates) {
      writer.write(item);
      }
      }

      I can understand what you are trying to say is, here I can use logic to dispatch to different writer.

      Problem 1: How I can identify the ItemWriter. Say I have three writer ItemWriter1, ItemWriter2, ItemWriter3 so I will idendify the writer class.

      Problem2: I did some ground work . I have written a configuration for composite writer in job.xml file

      <bean id="compositeWriter"
      class="org.springframework.batch.sample.domain.ord er.internal.CompositeItemWriter">
      <property name="delegates">
      <list>
      <ref bean="fileItemWriter" />
      <ref bean="fileItemWriter1" />
      </list>
      </property>
      </bean>

      Now I have defined two differend ItemWriter
      <bean class="FlatFileItemWriter" id="fileItemWriter">
      <bean class="FlatFileItemWriter" id="fileItemWriter1">

      And I have specified different file name to it.

      So if we see the compositeWriter.write(). It should write same data to both the file.
      But what I found was, It was writing twice to one file and is not writing anything on the second file.

      I ran the debug and found it is trying to invoke different FileItemWriter, but it write twice to a single file, instead of writing line to both the file.

      Will appreciate if you can explain what is happening. I tried a lot to find the reason and write to both the files.

      Comment


      • #4
        I tried to use composit writer with old version and it works fine. But when I tried using org.springframework.batch.dist-2.0.0.M3-with-dependencies.zip there is a problem with compositeItemWriter. There is some bug I guess.

        Regards

        Comment


        • #5
          I don't see how the composite writer can cause such trouble given how simple it is. We still have the compositeItemWriterSampleJob that works fine. If you were able to recreate the problem there that would be very helpful.

          Comment


          • #6
            I guess, you need to provide the definition of the following
            beans as well. Are you sure that by any chance, you are not
            mentioning same file name as the property to both the writers ?

            <bean class="FlatFileItemWriter" id="fileItemWriter">
            <bean class="FlatFileItemWriter" id="fileItemWriter1">

            I've used spring batch M3 in similar capacity and don't see any problem.

            Thanks.

            Comment


            • #7
              compositeItemWriterSampleJob works because it
              1. writes to a file
              2. insert data to database.

              But the problem is with writing to two file.

              I am giving different file names so there is no confusion with the file name. As I said , I tried with the spring-batch-dist-2.0.0.M1 and earlier version and it is working fine. But with spring-batch-dist-2.0.0.M2 and spring-batch-dist-2.0.0.M3 there is a problem.
              The configuration is below
              Regestring the stream
              Code:
              	<bean id="compositeItemWriterJob" parent="simpleJob">
              		<property name="steps">
              			<bean id="step1" parent="simpleStep">
              				<property name="streams">
              					<list>
              						<ref bean="fileItemReader" />
              						<ref bean="fileItemWriter" />
              						<ref bean="fileItemWriter1" />
              					</list>
              				</property>
              				<property name="itemReader" ref="fileItemReader" />
              				<property name="itemProcessor">
              					<bean class="org.springframework.batch.item.validator.ValidatingItemProcessor">
              						<constructor-arg ref="fixedValidator" />
              					</bean>
              				</property>	
              				<property name="itemWriter" ref="compositeWriter" />
              			</bean>
              		</property>
              	</bean>
              defining composite writer
              Code:
              	<bean id="compositeWriter"
              		class="org.springframework.batch.item.support.CompositeItemWriter">
              		<property name="delegates">
              			<list>
              				<bean
              					class="org.springframework.batch.sample.domain.trade.internal.TradeWriter">
              					<property name="dao" ref="tradeDao" />
              				</bean>
              				<ref bean="fileItemWriter" />
              				<ref bean="fileItemWriter1" />
              			</list>
              		</property>
              	</bean>
              defining the beans
              Code:
              	<bean class="org.springframework.batch.item.file.FlatFileItemWriter"
              		id="fileItemWriter">
              		<property name="resource"
              			value="file:target/test.TEMP.txt" />
              		<property name="lineAggregator">
              			<bean
              				class="org.springframework.batch.item.file.transform.PassThroughLineAggregator" />
              		</property>
              	</bean>
              
              	<bean class="org.springframework.batch.item.file.FlatFileItemWriter"
              		id="fileItemWriter1">
              		<property name="resource"
              			value="file:target/test-outputs/test.TEMP1.txt" />
              		<property name="lineAggregator">
              			<bean
              				class="org.springframework.batch.item.file.transform.PassThroughLineAggregator" />
              		</property>
              	</bean>

              Comment


              • #8
                I've tried adding another FFIW into the composite sample and ran into the same problem. I'm not sure where things go wrong yet, but you can track the issue here http://jira.springframework.org/browse/BATCH-969

                Comment


                • #9
                  I have found the error for ComositeWriter.
                  If I set the TransactionSynchronizationManager.isActualTransact ionActive() as false then it works.

                  It means there is a issue with the use of transaction.

                  In class OutputState we create new instance of FileOutputStream and encapsulate with
                  TransactionAwareBufferedWriter to get outputBufferedWriter

                  Code: TransactionAwareBufferedWriter outputBufferedWriter = new TransactionAwareBufferedWriter(Channels.newWriter( fileChannel, encoding));

                  So what we do is we keep on saving the lines read in a transaction in buffer with key
                  BUFFER_KEY.

                  And at the end we call commit using code
                  transactionManager.commit(transaction); in code TaskletStep.java .
                  So here there is some issue with binding both the Writers and the transaction

                  Thanks for jira

                  Comment


                  • #10
                    Yes, it's definitely the TransactionAwareBufferedWriter causing the issue.

                    Comment


                    • #11
                      OK, it's fixed now and the compositeItemWriterSampleJob shows how to configure the writers correctly.

                      Comment

                      Working...
                      X