Announcement Announcement Module
Collapse
No announcement yet.
Two file readers for different kind of files in the same step Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Two file readers for different kind of files in the same step

    Hi all,

    Here is my pb : i have two files in input that i need to process in my step. But the files are not of the same type (one '.csv' and one '.txt'), do not contain the same information, and information is formatted differently in each case.

    For instance:
    infos.csv
    Code:
    item1 | price=xxx
    item1 | quantity=xxx
    item1 | color=xxx
    item2 | weight=xxx
    item3 | color=xxx
    item5 | price=xxx
    ...
    items.txt
    Code:
    item1
    item2
    item3
    item4
    item5
    ...
    i need for each item contained in items.txt to retrieve its information (contained in csv file) and write two files at the end : one containing items which have no information associated and one containing items with information.

    What is the correct way to implement this? I do not see how dealing with two readers like that in the same step (i want too to be sure that i can restart the job if it fails during reading from where it fails)?

    Thanks in advance for your help.

  • #2
    basically you need a custom version of org.springframework.batch.item.support.AbstractIte mCountingItemStreamItemReader (see spring-batch-infrastructure jar) which works with list of files, it looks quite possible to me

    Comment


    • #3
      Thanks for your reply michael.lange.

      Is it the correct/best way to do it? Before trying this solution, anyone would have a better idea?

      Comment


      • #4
        You can have a look at MultiResourceItemReader.
        http://static.springsource.org/sprin...temReader.html

        Comment


        • #5
          if you still need a solution i have a running proof of concept under my github repository https://github.com/langmi/spring-batch-examples

          either check out all or take a look at the online source under https://github.com/langmi/spring-bat...master/readers (java package 'composite' and composite-flatfile-job.xml)

          the solution does not write into 2 files, but for that requirement you could create 2 custom writers which check the item before really writing it, see itemwriter reference (compositeItemWriter)

          How does it work ?
          1. works similar to the MultiResourceItemReader, but does not read the resources sequentially, but simultaneously
          2. uses a CompositeItemStreamReader (almost identical to org.springframework.batch.item.support.CompositeIt emStream)
          3. takes list of ItemStreamReaders and delegates "read" to all the reads of the registered readers
          4. copies the LineMapper concept and provides a "unifying" mapper which takes all the items and creates one

          Is it restartable?

          I am almost sure :-) a restart scenario test is there too see CompositeItemStreamReaderTest
          For a working scenario you need to set the name of each configured FlatFileItemReader, otherwise one will overwrite the restart-start-line for the other, see the composite-flatfile-job.xml.

          further links:

          github project-setup

          some more thoughts

          A more sophisticated version should be possible with some changes to the MultiResourceItemReader to make it work concurrently.

          JIRA Ticket is created under https://jira.springsource.org/browse/BATCH-1791
          Last edited by michael.lange; Sep 7th, 2011, 03:46 PM.

          Comment

          Working...
          X