Announcement Announcement Module
No announcement yet.
Multithreaded Batch Design for use case Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Multithreaded Batch Design for use case


    I would like some input on a multithreaded approach for a specific use case.

    My requirement is to have a job that parses records from relatively small files in which each record maps to a Java object (no need for partitioning or implementing a multithreaded reader), transform the java object to an XML Beans Web service Request Document, then send then call the same web service on each item.

    My idea for design uses Spring integration to

    1. poll the file System, wrap the file in a JobLaunchRequest object, then send the request on a job-requests channel.
    2. configure a job step1 to Read the file in the job request and parse records to Java objects using FlatFileItemReader, then in the writer send the object on a Spring Integration channel called staging (single threaded step).
    3. configure step 2 as a multithreaded step that polls the staging channel in the reader, builds the XMLBeans request Document in the processor, then calls the web service in the Writer. (One thread for each read-process-write)
    4. Perform cleanup operations on file system (move/delete files)

    Does anyone know of a more efficient way to handle this scenario? Should number 3 be configured as a separate job rather than a second step? Or should it be configured as a parallel step to Step 1? I would like for Messages to be read from the staging queue in Step 2 as soon as they are placed on the queue in the writer for Step 1

    For example
    Step 1 Read file --> write Message to stagin queue --> Step 2 read from staging queue (as soon as first item is placed on queue) --> transform --> send WS

    Thanks in advance.