Announcement Announcement Module
No announcement yet.
Questions about process multiple files per step and then move processed files Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Questions about process multiple files per step and then move processed files

    Hi, I'm evaluating Spring Batch and I have some problems to implement my use case.

    Use Case: An external system continuously is writing delimited files in a directory, it creates a new file per hour and the file name of these files have this pattern print-yyyyMMddhh.txt. Now I have to make a batch witch run every hour and read that files to write the content in a database. But I have the following requirements:

    1. For each execution the batch have to process all files in that directory which follow the pattern print-yyyyMMddhh.txt and are two hours older than the current time.

    2. The batch have to create a file print-yyyyMMddhh-error.txt (in an error directory) with all failures lines of a file. I mean, if the batch is processing the file print-2008061913.txt and find a failure line it have to wite that line in the file print-2008061913-error.txt (it have to create that file in the first error) and then continues with the following line.

    3. After process a file the batch have to move that file to a processed directory.

    I've this questions:

    1. How to process several files in an execution?. For this task I'm using the MultiResourceItemReader class and it seems to work well. The only problem that I have is how to filter the files witch are two hours older than the current time. I was playing a bit extending PathMatchingResourcePatternResolver and more or less it works but I dont like the solution because. It seems that PathMatchingResourcePatternResolver was not created to be extended and I dont like to change the behavior for the convertion of all Resource[]. Is there a better way to implement this requirement?

    2. How to implement the second requirement. I dont investigate this point so mutch but I think that implmenting a ItemReadListener I can get the original line using the getIntup method of the FlatFileParseException class. The problem is that, in the listener, which is the processed file. Is there any way to implement this requierement? maybe extending FlatFileItemReader class?

    3. How to move the file afer process it. Searching the forum I read that this it would be done implementing a StepExecutionListener. But I have two questions. How do I known, in the listener, which file was processed? Besides that, the listener is called one for step and not once per file. Is there another listener that is call after each file was processed?

    Thanks in advance, Claudio.

  • #2
    1. I don't see how to solve it any other way that what you've mentioned.

    2. I don't think I fully understand this question. Why won't FlatFileParseExceptino work? It's thrown from the reader and could easily be caught in 'onReadError'

    3. You could put the file names that need to be moved into the ExecutionContext and then move them all in the after step.


    • #3
      Thanks lucasward, but I have some questions.

      3. how can I put file names into the ExecutionContext?
      Sorry if this is an stupid question but I don't known to do this.
      I thought that I had to extend MultiResourceItemReader to make this but looking the code of that class I don't sure now.

      2. I'm going to try to explain this point.
      If the batch is processing the file print-2008061913.txt and there is an error in a line, that line has to be written in a file called print-2008061913-error.txt.
      I mean the error file name is the same than the original file but appending "-error" (and this file is created on the first error).
      If I use FlatFileParseException to implement this functionality, I can get the failure line but I don't known in which file that line is and I need the file name to create the error file (original file name + "error").

      For example:

      Suppose that we have to process the file product-stock-2008061913.txt (pipe delimited) and this file has the following lines. The first field is the product code and the second is the stock (it has to be a number):
      AAA123|12 --Correct
      BBB456|one --Incorrect
      CCC789|23 --Correct
      DDD123|four --Incorrect

      When the batch process this file it has to create a new file called product-stock-2008061913-error.txt with the failure lines, in this case:

      How is the best way to do this?

      Thanks, Claudio.


      • #4
        Hi claudio

        I also have the same problem with you. Basically when using multiresourcreader, i need to know the line that i am processing belongs to which particular file.

        Have you found solution to this ?