Announcement Announcement Module
Collapse
No announcement yet.
How to handle partitions ? Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • How to handle partitions ?

    Hi all,
    How is the partioning taking place(in Spring Batch 2.1) using the grid size ?

    For example:

    I have 10,000 records and If the grid size is 10 , so each threads take 1000 records and executes them in parallell.

    1. How the records are splitted across the 10 different threads ? . On what basis the splitting of records takes place ?

    2. If each thread reads records from the same database, will there be any object level locking taking place while partitioning ?

    3. If one thread fails, how to assign the un-processed records of the failed thread to the remaining threads ?

    <code>

    <job id="Prices_LoadJob" restartable="true"
    xmlns="http://www.springframework.org/schema/batch">
    <step id="Prices_Load_ManagerStep">
    <partition step="Prices_LoadStep" partitioner="partitioner">
    <handler grid-size="10" task-executor="taskExecutor" />
    </partition>
    </step>
    </job>
    <bean id="partitioner"
    class="org.springframework.batch.core.partition.su pport.SimplePartitioner" />

    <bean id="taskExecutor" class="org.springframework.core.task.SimpleAsyncTa skExecutor" />



    <step id="Prices_LoadStep" xmlns="http://www.springframework.org/schema/batch">
    <tasklet transaction-manager="transactionManager">
    <chunk reader="pricesLoadReader" processor="pricesLoadProcessor"
    writer="pricesLoadWriter" commit-interval="8">
    </chunk>
    <listeners>
    <listener ref="partitiontepListener" />
    </listeners>
    </tasklet>

    </step>

    <bean id="partitiontepListener"
    class="batch.core.partition.PartitionStepListener" ></bean>

    </code>

    Kindly reply at the earliest .

  • #2
    Originally posted by arun4 View Post
    1. How the records are splitted across the 10 different threads ? . On what basis the splitting of records takes place ?
    Using a Partitioner. The SimplePartitioner will not work for your use case.

    2. If each thread reads records from the same database, will there be any object level locking taking place while partitioning ?
    Not unless you do the locking yourself.

    3. If one thread fails, how to assign the un-processed records of the failed thread to the remaining threads ?
    Restart the job.

    Comment


    • #3
      Using a Partitioner. The SimplePartitioner will not work for your use case.
      Dave can you suggest how to write a custom paritioner for this purpose. or any other approach to achieve this functionality.

      Comment


      • #4
        Did you look at the sample (https://fisheye.springsource.org/vie...b.xml?r=3831)?

        Comment


        • #5
          ColumnRangePartitioner

          Hi All,
          Thanks for your reply dave.
          I have to process 1,00,000 records, my grid size is 5,
          ColumnRangePartitioner will Split the records like
          1 - 20,000 --partition1
          20,000 - 40,000 --partition2
          40,000 - 60,000 -- partition3
          60,000 -80,000 -- partition4
          80,000 -1,00,000 -- partition5

          If the number of records need to process starts from the range 80,000 to 1,00,000 means, only partition5 will initiate the process and remaining all the partition will not be initiated.
          Then this approach is not so efficient for my use case.

          Dave can you suggest any other approach. ( i am using spring batch 2.1)

          Thanks for your reply in advance..
          Last edited by arun4; Jul 10th, 2010, 10:07 AM.

          Comment

          Working...
          X