Announcement Announcement Module
Collapse
No announcement yet.
Some questions about chunk processing Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Some questions about chunk processing

    Hi,

    I've some questions about chunk processing:

    1. A relatively small commit interval will lead to more hits to meta data table like update execution context.
    Which means it could slow down the batch process, right?

    2. Does execution context (and some other meta info) have to be synchronously write to meta table?
    Is there any other usage for execution context other than restart? Can I write it to DB async?

    3. Any other commit policy (CheckpointAlgorithm) ? for example, I'd like it to commit every 5 minutes, or every 10M data totally wrote.


    Thanks
    Ma Ling

  • #2
    1. Yes. Tuning that commit interval appropriately is important for batch performance.
    2. The ExecutionContext can be (and is) used during the current execution as well as on restart. It cannot be written to the database asynchronously because if the write fails and the job has moved on, the repository is left in an inconsistent state.
    3. You can implement your own CompletionPolicy that allows for complex chunk completion options like you mention.

    Comment

    Working...
    X