Announcement Announcement Module
Collapse
No announcement yet.
Chunk processing stores items in memory until you hit commit-interval? Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Chunk processing stores items in memory until you hit commit-interval?

    If I set my commit-interval to 2000 will Spring Batch read in 2000 items and store them in an array, then process them, then write? What if these objects are large? I do not want them in memory at all. I do not want to commit more frequently either. What are my options?

    I thought the the pattern was read-process-write, repeat until commit interval reached, then commit. Now I'm seeing that it's read until commit interval reached, then process, then write all. Am I missing something? I don't want objects stored in memory between commits, that seems unscalable and requires tight control of the commit-interval.

  • #2
    A couple things:
    1. You are correct in the underlying processing. While, in theory, it would be read/process until commit, then write, it is actually read until commit interval, process then write.
    2. How else would you implement chunk based processing without keeping the objects in memory? The only alternative is to write every item out as they are processed (item based processing) but that causes both I/O bottle necks and yet, the framework still allows it by setting the commit-interval to 1.

    Comment

    Working...
    X