Announcement Announcement Module
Collapse
No announcement yet.
Spring Batch Multi Threaded Step - How to make each thread read unique records? Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Spring Batch Multi Threaded Step - How to make each thread read unique records?

    This question has been asked a lot of times in many forums. But I don't see an answer that fits me. I am trying to implement Multi-threaded Step in my spring batch implementation.

    Have a staging table with 100k records

    Want to process it in 10 threads of commit interval 300 per thread- so 3000 records at any point of time.

    I defined a task executor and referred it inside the step i wanted to multi thread

    My idea is that first I would get the thread pool size (10) and update the thread_id column with a velue(can be 1-10) to each of the 100k records. In this case of 10 threads and 100k records so 10k records will be assigned one id - I am trying to implement a stagingsteplistener to do this.

    wrote a reader for this staging table. task executor will create 10 readers and each reader must read 300 different records and process them - Now how do I pass a common id between the step listener and reader so that each thread will have its own set of records to process.

    As of now I have only one JVM. So I am thinking of doing this in Multi Threaded step itself rather than thinking about partition based approach.

    Please help......

    I referred pro spring batch book and created a staging step listener which is accepting a run id from the job configuration xml using job parameters as below

    Code:
    <beans:bean id="stagingStepListener"
    class="com.apress.springbatch.statement.listener.StagingStepListener" scope="step">
    <beans:property name="dataSource" ref="dataSource"/>
    <beans:property name="tableName" value="transaction"/>
    <beans:property name="whereClause"
    value="where jobId is null and processed is null"/>
    <beans:property name="jobId" value="#{jobParameters[run.id]}"/>
    </beans:bean>
    What i dont find is this? From where is this "run.id" coming from. I dont see that in any place in the book. I copied the same implementation in my spring batch and when i run it I see exception saying that run.id is not identifiable. Please help me about how to do this?

  • #2
    take a look at https://github.com/acogoluegnes/Spring-Batch-Workshop there is an example:
    database reading partitioning: reading categorized items from a database and writing them to files
    which should be adjustable by your needs

    Comment

    Working...
    X