Announcement Announcement Module
No announcement yet.
Two file readers for different kind of files in the same step Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Two file readers for different kind of files in the same step

    Hi all,

    Here is my pb : i have two files in input that i need to process in my step. But the files are not of the same type (one '.csv' and one '.txt'), do not contain the same information, and information is formatted differently in each case.

    For instance:
    item1 | price=xxx
    item1 | quantity=xxx
    item1 | color=xxx
    item2 | weight=xxx
    item3 | color=xxx
    item5 | price=xxx
    i need for each item contained in items.txt to retrieve its information (contained in csv file) and write two files at the end : one containing items which have no information associated and one containing items with information.

    What is the correct way to implement this? I do not see how dealing with two readers like that in the same step (i want too to be sure that i can restart the job if it fails during reading from where it fails)?

    Thanks in advance for your help.

  • #2
    basically you need a custom version of mCountingItemStreamItemReader (see spring-batch-infrastructure jar) which works with list of files, it looks quite possible to me


    • #3
      Thanks for your reply michael.lange.

      Is it the correct/best way to do it? Before trying this solution, anyone would have a better idea?


      • #4
        You can have a look at MultiResourceItemReader.


        • #5
          if you still need a solution i have a running proof of concept under my github repository

          either check out all or take a look at the online source under (java package 'composite' and composite-flatfile-job.xml)

          the solution does not write into 2 files, but for that requirement you could create 2 custom writers which check the item before really writing it, see itemwriter reference (compositeItemWriter)

          How does it work ?
          1. works similar to the MultiResourceItemReader, but does not read the resources sequentially, but simultaneously
          2. uses a CompositeItemStreamReader (almost identical to emStream)
          3. takes list of ItemStreamReaders and delegates "read" to all the reads of the registered readers
          4. copies the LineMapper concept and provides a "unifying" mapper which takes all the items and creates one

          Is it restartable?

          I am almost sure :-) a restart scenario test is there too see CompositeItemStreamReaderTest
          For a working scenario you need to set the name of each configured FlatFileItemReader, otherwise one will overwrite the restart-start-line for the other, see the composite-flatfile-job.xml.

          further links:

          github project-setup

          some more thoughts

          A more sophisticated version should be possible with some changes to the MultiResourceItemReader to make it work concurrently.

          JIRA Ticket is created under
          Last edited by michael.lange; Sep 7th, 2011, 03:46 PM.