Announcement Announcement Module
Collapse
No announcement yet.
Asynchronous Chunk Processing - use case Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Asynchronous Chunk Processing - use case

    I was looking at this use case. We are using the remote chunking.
    Within a chunk we need to process in parallel. Does spring batch support this yet? Any example or direction would be helpful.

    Thanks in advance,

    Kumar

  • #2
    There's nothing to stop you from background processing, but the workload must be non-transactional (obviously hopefully), which normally means you need it to be idempotent, in case there is a partial failure.

    Comment


    • #3
      Multi threading within a chunk

      Hi Dave,

      Thanks for your reply.

      Can an itemprocessor within a chunkHandler be configured to execute in multiple threads?

      The chunks are distributed via a JMS queue.

      So let us say in a chunk there are 1000 items to be processed and the processing of these 1000 items be handled by 10 threads so the threads are processing in parallel and each thread is processing one item at a time. It does not matter as what order these items are processed within a chunk. (as the processing is the same for all items and is independent).

      Regards,

      Kumar

      Comment


      • #4
        Originally posted by kumarcsk View Post
        Can an itemprocessor within a chunkHandler be configured to execute in multiple threads?
        Yes. There's no special way to do that. I usually use the ItemWriter to rendezvous all the items from the chunk so the tasklet can know that they are finished. There's an example in Spring Batch Integration (recently moved to the Spring Batch Admin project).

        The chunks are distributed via a JMS queue.
        Now you lost me. If the chunks are coming in via JMS, your listeners are already multithreaded by their container, so there shouldn't be anything special to do.

        Comment


        • #5
          Originally posted by Dave Syer View Post
          Yes. There's no special way to do that. I usually use the ItemWriter to rendezvous all the items from the chunk so the tasklet can know that they are finished. There's an example in Spring Batch Integration (recently moved to the Spring Batch Admin project).
          Short question: where can I find this example in the Spring Batch Admin project?

          Comment


          • #6
            https://src.springframework.org/svn/...egration/async

            Comment

            Working...
            X