Announcement Announcement Module
No announcement yet.
MultiResourceItemWriter with Chunk size is creating problems Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • MultiResourceItemWriter with Chunk size is creating problems


    I have the following usecase:

    1. Read from the database.
    2. Do some validation and processing. Skips the rows if validation failed.(Skip limit is 1000).
    3. Write to database table and flat files(file size can be max 1000 rows).
    4. The writer is in a transaction and the commit-interval is 100.

    I'm using a composite writer which uses a custom database writer and MultiResourceItemWriter (using FlatFileItemWriter as Delegate).

    Problem: We have strict restriction on the file size which can't go beyond 1000 rows.

    1. I see a problem in the file size going beyond the MAX if the MAX_ROWS_IN_FILE (file size) is not multiples of commit-interval. To fix this we are validating these two properties before we start the actual step.

    2. But the problem still exists if there are any skipped records. The file size will go beyond the MAX limit.

    I have gone through the source code. Looks like the commit-interval is used as the chunk size and when the chunk is given to the writer the size can be <= the actual chunk size because of the skipped rows. This is causing the problem. I believe the chunk size and commit-interval should be two different props.

    Can any one suggest a workaround for this?