Announcement Announcement Module
No announcement yet.
Advice on File Deletion Only After Successful Processing of Records Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Advice on File Deletion Only After Successful Processing of Records


    I am currently working on a Spring Integration project where the input data starts off as a file that is retrieved from a directory and split into multiple records (one record per line in the file). What I want to do is keep the file until I know that all of the records have been completely processed and then delete the file, if and only if, all records were successfully processed.

    Each record is sent through the messaging infrastructure as a object that contains, among other things, the name of the original file. How would you suggest that I implement this? I have considered using an aggregator (paired with my initial splitter than breaks the file into records) that would keep track of the file count, but have questions about how I would communicate the file count down to the aggregator. I have considered using transactions. Currently all of the processing is on one machine, but that could easily change with JMS being used for communication between machines and I don't know if that would make the transactions more difficult. Any suggestions would be greatly appreciated.

    Thank you,
    Joshua Smith

  • #2
    When the splitter splittes the Messages it assignes sequence number to every message so default aggregation strategy knows when received messages re ready to be aggregated, so essentially you don't really have to count (framework will do it for you) and when the aggregator finally aggregates the messages you know at that time that all the records were processed. Event in distributed world where you'll use JMS sequenceNumber header will be propagated with the message.


    • #3
      Thank you Oleg.

      So then the strategy is to have an aggregator on the far end of the messaging infrastructure and when it sees that all of the records have arrived it deletes the input file? Is that right?

      How do I access the sequence number information? All of my endpoints are written so that their method signatures take the business object (e.g. Record), not a parameterized message (e.g. Message<Record>). Or do I have to write that aggregator so that it takes the pararameterized message?


      • #4
        (@Header("sequenceSize")String sequenceSize) etc... Just look at the MessageHeaders constants