Announcement Announcement Module
Collapse
No announcement yet.
batch modifications from a file Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • batch modifications from a file

    I have been studying Spring Batch for a while now, but there is something I am either missing or misunderstanding. I would appreciate it if someone were willing to help me.

    The application I am considering Spring Batch for consumes a file hundreds of megabytes in size, every morning. The file contains deltas to be applied to what we already have in our database. So, in order to maximize throughput, what I think I should do is read a chunk from the input file, load that chunk's worth of entities from the database, and then write that chunk's worth of changes to the database. I see how Spring Batch helps with the first and last task, but it seems like it would be working against me during the second task.

    My intention has been to put the "load and apply deltas" code into a processing task, which would be the second task above. In order to maximize throughput, I think I should load as many of the entities in the chunk as I can, using as few queries as possible. But the "process task" interface only provides a single record at a time.

    Again, I would appreciate suggestions regarding how to tackle this.

    Thanks,
    Sam Bishop

  • #2
    i see no problem in implementing this inside a single ItemWriter which works with the chunk, maybe you could even use the CompositeItemWriter for this

    Comment


    • #3
      I had not seriously considered that approach since it didn't seem like the "read from the database" and "apply the deltas" code belonged in a "write" task. But it fits if I think of it as a "writes deltas to the database" task, as you're suggesting. Thank you, Michael! I greatly appreciate your help.

      Regards,
      Sam Bishop

      Comment

      Working...
      X