Announcement Announcement Module
Collapse
No announcement yet.
How to configure complex job flow in Spring Batch Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • How to configure complex job flow in Spring Batch

    Hi All,
    I am using Spring Batch to execute network of Store Procedures.
    In the network, every step has to execute one Store procedure.
    Some of the steps run in parallel whereas some other steps run sequentially.

    I've attached Job flow diagram.

    The flow goes as follows.

    1. "Step A" runs first.
    2. Once "Step A" executed successfully, "Step B,C,D" should run in parallel.
    3. "Step D", which is part of parallel process B,C,D if completed then "Step F,G" can be started.
    4. "Step E" can be started only after completion of whole "Step B,C,D".

    The challenge i'm facing:

    In order to refer "Step F,G" from "Step D" I put the "Split of F,G" inside the flow of "Step D".

    Now "Step E" is waiting for "Step F,G" to be completed.
    How can i make "Step E" to get started just after completion of Step B,C,D?

    Here is how i wrote configure Job in xml file:

    <job id="myJob" job-repository="jobRepository" restartable="true">
    <step id="stepA" next="parallelProcessBCD">
    <tasklet ref="taskletjob" />
    </step>
    <split id="parallelProcessBCD" task-executor="taskExecutor"
    next="stepE">
    <flow>
    <step id="stepB">
    <tasklet ref="taskletjob" />
    </step>
    </flow>
    <flow>
    <step id="stepC">
    <tasklet ref="taskletjob" />
    </step>
    </flow>
    <flow>
    <step id="stepD" next="parallelFG">
    <tasklet ref="taskletjob" />
    </step>
    <split id="parallelFG" task-executor="taskExecutor">
    <flow>
    <step id="stepF">
    <tasklet ref="taskletjob" />
    </step>
    </flow>
    <flow>
    <step id="stepG">
    <tasklet ref="taskletjob" />
    </step>
    </flow>
    </split>
    </flow>
    </split>
    </job>

  • #2
    You could put a split after B,C,D that executes E,(F,G) in parallel. Might not be the most efficient arrangement but I think it is the only one that works out of the box. To get closer to your execution plan you could write a custom backgrounding Step for F,G so that they could be executed in the flow of D without having to wait for the result. Then there would have to be a decision or step at the end to wait for G to finish.

    (Please use [code][/code] tags to post code and stack traces.)

    Comment


    • #3
      Hey Folks,

      This thread is rather old, but I'm facing the same challange so I don't want to start a separate thread.

      I have to five jobs A, B, C, E, F. A, B, C should run parallel. E should start a soon as A and B are finished and F should start as soon as B and C are finished. What I did, I created a split-flow with 5 threads an introduced five TaskletSteps that either block (before execution of E and F) or release (after execution of A, B and C) the flow. I'm not too happy with this solution, as the configuration is getting intransparent and you have there are five additional steps you don't really need.

      Does anyone have a better idea?


      Here is the tasklet:
      Code:
      package com.test;
      
      import java.util.concurrent.Semaphore;
      
      public class StepSemaphore  {
          
          private Semaphore semaphore;
          private int stepsToWaitFor;
          
          public StepSemaphore(int stepsToWaitFor) throws InterruptedException {
              this.semaphore = new Semaphore(stepsToWaitFor);
              this.semaphore.acquire(stepsToWaitFor);
              this.stepsToWaitFor = stepsToWaitFor;
          }
      
          public void acquire() throws InterruptedException {
              System.out.println("StepSemaphore.acquire() [blocked]");
              semaphore.acquire(stepsToWaitFor);
              System.out.println("StepSemaphore.acquire() [released]");
          }
          
          public void release() {
              System.out.println("StepSemaphore.release()");
              semaphore.release(1);
          }
      
      }
      And here the spring-config:

      Code:
      	<bean id="semaphoreE" class="com.test.StepSemaphore">
      	   <constructor-arg index="0" value="2" />
      	</bean>
      	<bean id="semaphoreF" class="com.test.StepSemaphore">
      	   <constructor-arg index="0" value="2" />
      	</bean>
      
      	<batch:job id="myJob">
      		<batch:split id="split1" task-executor="taskExecutor">
      			<batch:flow>
      				<batch:step id="stepA" next="triggerAtoE">
      					<batch:tasklet>
      						<batch:chunk reader="itemReaderA" writer="itemWriterA" commit-interval="1" />
      					</batch:tasklet>
      				</batch:step>
                      <batch:step id="triggerAtoE">
                          <batch:tasklet ref="semaphoreE" method="release"/>
                      </batch:step>
      			</batch:flow>
                  <batch:flow>
                      <batch:step id="stepB" next="triggerBtoE">
                          <batch:tasklet>
                              <batch:chunk reader="itemReaderB" writer="itemWriterB" commit-interval="1" />
                          </batch:tasklet>
                      </batch:step>
                      <batch:step id="triggerBtoE" next="triggerBtoF">
                          <batch:tasklet ref="semaphoreE" method="release"/>
                      </batch:step>
                      <batch:step id="triggerBtoF">
                          <batch:tasklet ref="semaphoreF" method="release"/>
                      </batch:step>
                  </batch:flow>
                   <batch:flow>
                      <batch:step id="stepC" next="triggerCtoF">
                          <batch:tasklet>
                              <batch:chunk reader="itemReaderC" writer="itemWriterC" commit-interval="1" />
                          </batch:tasklet>
                      </batch:step>
                      <batch:step id="triggerCtoF">
                          <batch:tasklet ref="semaphoreF" method="release"/>
                      </batch:step>
                  </batch:flow>
                  
                   <batch:flow>
                      <batch:step id="blockerE">
                          <batch:tasklet ref="semaphoreE" method="acquire"/>
                      </batch:step>
                      <!-- Execute Step E -->
                  </batch:flow>
                  <batch:flow >
                      <batch:step id="blockerF">
                          <batch:tasklet ref="semaphoreF" method="acquire"/>
                      </batch:step>
                      <!-- Execute Step F -->
                  </batch:flow>
      		</batch:split>
      	</batch:job>

      Comment


      • #4
        Honestly? I wouldn't use Spring Batch to orchestrate this. Spring Batch is intended for job execution, not orchestration. There are better tools out there for doing job orchestration (which is really what you're looking for). I'd start with Quartz and go from there.

        Comment


        • #5
          Hi,
          thanks for the reply. I got the point and agree. But the solution I've outlined is already implemented.
          Introducing new frameworks is not an option because any new technology has to undergo an approval-process that takes quite a while. Since there is only one case where it's getting that complex, there is no real need for a new framework. Quartz would be fine, because it is already in use and it might have made things a little easier, but everyone is happy with the solution :-)

          Comment

          Working...
          X