Announcement Announcement Module
Collapse
No announcement yet.
Complex Parallel Jobs Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Complex Parallel Jobs

    Hello,

    I have the need to implement a job that has complex dependencies between it's steps. See the attached diagram. I need for Step A to run, then B and C to run in parallel. While C continues to run, I need to run D and E after B completes, and finally to run F when C, D, and E are all complete.

    Split is fine for creating parallelism, but can I nest steps/splits? You know, Split(A,B) and after B say go to another step that splits to run (D,E) ?

    This is the logical business flow, and if i have to run them serially below the first level, they will just take too long to complete.

    Thanks for any help.

  • #2
    Yes, you can nest splits. However, you may want to consider breaking this up into multiple jobs. This would allow you to manage these dependencies in a more dynamic way.

    Comment


    • #3
      I configured the batch job as follows:

      Code:
      <?xml version="1.0" encoding="utf-8"?>
      <beans xmlns="http://www.springframework.org/schema/beans" xsi:schemaLocation="http://www.springframework.org/schema/beans &#xD;&#xA;        http://www.springframework.org/schem...-beans-3.0.xsd &#xD;&#xA;        http://www.springframework.org/schema/batch &#xD;&#xA;        http://www.springframework.org/schema/batch/spring-batch-2.1.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:batch="http://www.springframework.org/schema/batch">
      <import resource="EmployeeEnrollmentCompleteJob.xml" />
      <import resource="OpenEnrollmentReminderJob.xml" />
      <import resource="EmployeeAutoEnrollDuringEnrollmentJob.xml" />
      <import resource="sendNotificationJob.xml" />
      
        <batch:job id="MainJob">
          <batch:split id="MainJobSplit" task-executor="taskExecutor">
            <batch:flow>
              <batch:split id="NestedSplit" task-executor="taskExecutor">
                <batch:flow>
                  <batch:step id="OpenEnrollmentReminderStep">
                    <batch:job ref="OpenEnrollmentReminder" />
                  </batch:step>
                </batch:flow>
                <batch:flow>
                  <batch:step id="DiscardEmployeeOenApplicationsStep">
                    <batch:job ref="DiscardEmployeeOpenApplications" />
                  </batch:step>
                </batch:flow>
              </batch:split>
              <batch:step id="EmployeeEnrollmentCompleteStep">
                <batch:job ref="EmployeeEnrollmentComplete" />
              </batch:step>
            </batch:flow>
            <batch:flow>
              <batch:step id="EmployeeAutoEnrollDuringEnrollmentStep">
                <batch:job ref="EmployeeAutoEnrollDuringEnrollment" />
              </batch:step>
            </batch:flow>
          </batch:split>
        </batch:job>
        <bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
          <property value="8" name="corePoolSize" />
          <property value="8" name="maxPoolSize" />
        </bean>
      </beans>
      The Spring batch is throwing the SAXParseException saying the Split with in a flow not allowed. You can ref it or use the flow.

      Please let me know what is the wrong in the configuration?
      Last edited by mminella; Apr 16th, 2013, 12:32 PM. Reason: Adding Code Tags

      Comment


      • #4
        Hmmm...When I try running the above XML, I actually receive an error that the step EmployeeEnrollementCompleteStep is unreachable (you don't have a next on the split before it).

        Comment


        • #5
          <?xml version="1.0" encoding="utf-8"?>
          <beans xmlns="http://www.springframework.org/schema/beans" xsi:schemaLocation="http://www.springframework.org/schema/beans &#xD;&#xA; http://www.springframework.org/schem...-beans-3.0.xsd &#xD;&#xA; http://www.springframework.org/schema/batch &#xD;&#xA; http://www.springframework.org/schema/batch/spring-batch-2.1.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:batch="http://www.springframework.org/schema/batch">
          <import resource="EmployeeEnrollmentCompleteJob.xml" />
          <import resource="OpenEnrollmentReminderJob.xml" />
          <import resource="EmployeeAutoEnrollDuringEnrollmentJob.xm l" />
          <import resource="sendNotificationJob.xml" />

          <batch:job id="MainJob">
          <batch:split id="MainJobSplit" task-executor="taskExecutor">
          <batch:flow>
          <batch:split id="NestedSplit" task-executor="taskExecutor">
          <batch:flow>
          <batch:step id="OpenEnrollmentReminderStep">
          <batch:job ref="OpenEnrollmentReminder" />
          </batch:step>
          </batch:flow>.......

          Can I import the existing batch jobs in side the new batch job as above. If I used is way I am getting the following error:

          y: java.lang.IllegalStateException: org.springframework.batch.core.configuration.Dupli cateJobException: A job configuration with this name [EmployeeAutoEnrollDuringEnrollment] was already registered
          at org.springframework.batch.core.configuration.suppo rt.AutomaticJobRegistrar.start(AutomaticJobRegistr ar.java:164)
          at org.springframework.batch.core.configuration.suppo rt.AutomaticJobRegistrar.onApplicationEvent(Automa ticJobRegistrar.java:128)
          at org.springframework.context.event.SimpleApplicatio nEventMulticaster$1.run(SimpleApplicationEventMult icaster.java:78)
          at org.springframework.core.task.SyncTaskExecutor.exe cute(SyncTaskExecutor.java:49)
          at org.springframework.context.event.SimpleApplicatio nEventMulticaster.multicastEvent(SimpleApplication EventMulticaster.java:76)
          Truncated. see log file for complete stacktrace
          Caused By: org.springframework.batch.core.configuration.Dupli cateJobException: A job configuration with this name [EmployeeAutoEnrollDuringEnrollment] was already registered
          at org.springframework.batch.core.configuration.suppo rt.MapJobRegistry.register(MapJobRegistry.java:56)
          at org.springframework.batch.core.configuration.suppo rt.DefaultJobLoader.doLoad(DefaultJobLoader.java:1 52)
          at org.springframework.batch.core.configuration.suppo rt.DefaultJobLoader.load(DefaultJobLoader.java:114 )
          at org.springframework.batch.core.configuration.suppo rt.AutomaticJobRegistrar.start(AutomaticJobRegistr ar.java:161)
          at org.springframework.batch.core.configuration.suppo rt.AutomaticJobRegistrar.onApplicationEvent(Automa ticJobRegistrar.java:128)
          Truncated. see log file for complete stacktrace
          >

          I am loading all batch job definitions at start up using the simple job launcher:
          <bean
          class="org.springframework.batch.core.configuratio n.support.AutomaticJobRegistrar">
          <property name="applicationContextFactories">
          <bean
          class="org.springframework.batch.core.configuratio n.support.ClasspathXmlApplicationContextsFactoryBe an">
          <property name="resources" value="classpath:/jobs/*Job.xml" />
          </bean>
          </property>
          <property name="jobLoader">
          <bean
          class="org.springframework.batch.core.configuratio n.support.DefaultJobLoader">
          <property name="jobRegistry" ref="jobRegistry" />
          </bean>
          </property>
          </bean>

          Ar run time Batch is throwing the above exception.

          Comment


          • #6
            1. Please use the code tags to make your XML and stack traces readable.
            2. You are only allowed one job configured with a given name per context (the name must be unique) which is why you are getting the error you see.
            3. Keep in mind that you can externalize your flows so that they are easier to read (you are required to in some scenarios).

            Comment


            • #7
              2. You are only allowed one job configured with a given name per context (the name must be unique) which is why you are getting the error you see.
              Question) How can I change the batch job name, It is already defined in the "EmployeeEnrollmentCompleteJob.xml". If I want to reuse the job name in another dependency XML file, how can I do it?
              As per my understanding we have to use the same job name as ref in <batch:job ref="EmployeeEnrollmentComplete" /> where "EmployeeEnrollmentComplete" is defined in the EmployeeEnrollmentCompleteJob.xml file.

              3. Keep in mind that you can externalize your flows so that they are easier to read (you are required to in some scenarios).
              Question) externalize flows are in the same file it may be in different files. Please send me any example link

              Comment


              • #8
                I added the next="EmployeeEnrollmentCompleteStep" as per your suggestion earlier thread and tried to execute the job. I got the exception. Below is the updated XML file and I pasted the Exception below

                <?xml version="1.0" encoding="UTF-8"?>
                <beans xmlns="http://www.springframework.org/schema/beans"
                xmlns:batch="http://www.springframework.org/schema/batch" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xsi:schemaLocation="
                http://www.springframework.org/schema/beans http://www.springframework.org/schem...-beans-3.0.xsd
                http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd">

                <import resource="EmployeeEnrollmentCompleteJob.xml" />
                <import resource="OpenEnrollmentReminderJob.xml" />
                <import resource="EmployeeAutoEnrollDuringEnrollmentJob.xm l" />
                <import resource="DiscardEmployeeOpenApplicationsJob.xml" />

                <batch:job id="MainJob2">
                <batch:split id="MainJobSplit" task-executor="taskExecutor">
                <batch:flow>
                <batch:split id="NestedSplit" task-executor="taskExecutor" next="EmployeeEnrollmentCompleteStep">
                <batch:flow>
                <batch:step id="OpenEnrollmentReminderStep">
                <batch:job ref="OpenEnrollmentReminder" />
                </batch:step>
                </batch:flow>
                <batch:flow>
                <batch:step id="DiscardEmployeeOenApplicationsStep">
                <batch:job ref="DiscardEmployeeOpenApplications" />
                </batch:step>
                </batch:flow>
                </batch:split>
                <batch:step id="EmployeeEnrollmentCompleteStep">
                <batch:job ref="EmployeeEnrollmentComplete" />
                </batch:step>
                </batch:flow>
                <batch:flow>
                <batch:step id="EmployeeAutoEnrollDuringEnrollmentStep">
                <batch:job ref="EmployeeAutoEnrollDuringEnrollment" />
                </batch:step>
                </batch:flow>
                </batch:split>
                </batch:job>
                <bean id="taskExecutor" class="org.springframework.scheduling.concurrent.T hreadPoolTaskExecutor">
                <property value="8" name="corePoolSize" />
                <property value="8" name="maxPoolSize" />
                </bean>
                </beans>
                Exception:
                Caused By: java.lang.IllegalStateException: org.springframework.batch.core.configuration.Dupli cateJobException: A job configuration with this name [EmployeeEnrollmentComplete] was already registered
                at org.springframework.batch.core.configuration.suppo rt.AutomaticJobRegistrar.start(AutomaticJobRegistr ar.java:164)
                at org.springframework.batch.core.configuration.suppo rt.AutomaticJobRegistrar.onApplicationEvent(Automa ticJobRegistrar.java:128)
                at org.springframework.context.event.SimpleApplicatio nEventMulticaster$1.run(SimpleApplicationEventMult icaster.java:78)
                at org.springframework.core.task.SyncTaskExecutor.exe cute(SyncTaskExecutor.java:49)
                at org.springframework.context.event.SimpleApplicatio nEventMulticaster.multicastEvent(SimpleApplication EventMulticaster.java:76)
                Truncated. see log file for complete stacktrace
                Caused By: org.springframework.batch.core.configuration.Dupli cateJobException: A job configuration with this name [EmployeeEnrollmentComplete] was already registered
                at org.springframework.batch.core.configuration.suppo rt.MapJobRegistry.register(MapJobRegistry.java:56)
                at org.springframework.batch.core.configuration.suppo rt.DefaultJobLoader.doLoad(DefaultJobLoader.java:1 52)
                at org.springframework.batch.core.configuration.suppo rt.DefaultJobLoader.load(DefaultJobLoader.java:114 )
                at org.springframework.batch.core.configuration.suppo rt.AutomaticJobRegistrar.start(AutomaticJobRegistr ar.java:161)
                at org.springframework.batch.core.configuration.suppo rt.AutomaticJobRegistrar.onApplicationEvent(Automa ticJobRegistrar.java:128)
                Truncated. see log file for complete stacktrace

                Comment


                • #9
                  As I wrote in my previous post we are loading the all the batch jobs with the name ending *job.xml into the memory map. And in my dependency job xml file as specified in the above post "<import resource="EmployeeEnrollmentCompleteJob.xml" />" it is also trying to load one more time causing the problem. I have changed the batch jobs name to "EmployeeEnrollmentComplete.xml" and imported the same with the new name. This has fixed the problem and nested batch job xml defined in the above also working fine.

                  Comment


                  • #10
                    <batch:job id="DependentJobExecution">
                    <batch:split id="DependentSchedule" task-executor="taskExecutor">

                    </batch:flow>
                    <batch:flow>
                    <batch:split id="Split1" task-executor="taskExecutor"
                    next="Split2">
                    <batch:flow>
                    <batch:step id="RetryFedAnnualIncomeStep">
                    <batch:job ref="RetryFedAnnualIncome" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="RetryFedNonEsiMecStep">
                    <batch:job ref="RetryFedNonEsiMec" />
                    </batch:step>
                    </batch:flow>
                    </batch:flow>
                    </batch:split>
                    <batch:split id="Split2" task-executor="taskExecutor">
                    <batch:flow>
                    <batch:step id="RetryOPAStep">
                    <batch:job ref="RetryOPA" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="MedicaidCHIPRenewalStep">
                    <batch:job ref="MedicaidCHIPRenewal" />
                    </batch:step>
                    </batch:flow>

                    </batch:split>
                    </batch:flow>
                    <batch:flow>
                    <batch:split id="Split3" task-executor="taskExecutor"
                    next="Split4">
                    <batch:flow>
                    <batch:step id="PlanDecertificationStep">
                    <batch:job ref="PlanDecertification" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="NoPlanSelectionStep">
                    <batch:job ref="NoPlanSelection" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="EmployeeDependentAgeOutStep">
                    <batch:job ref="EmployeeDependentAgeOut" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="ProcessDisEnrollmentForAge65Step">
                    <batch:job ref="ProcessDisEnrollmentForAge65" />
                    </batch:step>
                    </batch:flow>
                    </batch:split>
                    <batch:split id="Split4" task-executor="taskExecutor">
                    <batch:flow>
                    <batch:step id="InsertTrigger834EnrollmentStep" next="Process834UpdateDataStep">
                    <batch:job ref="InsertTrigger834Enrollment" />
                    </batch:step>
                    <batch:step id="Process834UpdateDataStep" next="Process834AuditDataStep">
                    <batch:job ref="Process834UpdateData" />
                    </batch:step>
                    <batch:step id="Process834AuditDataStep">
                    <batch:job ref="Process834AuditData" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:split id="Split4a" task-executor="taskExecutor"
                    next="Split4b">
                    <batch:flow>
                    <batch:step id="Issuer820ProcessingStep">
                    <batch:job ref="Issuer820Processing" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="CheckReversalFileStep">
                    <batch:job ref="CheckReversalFile" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="KeyPayRemittanceFileStep">
                    <batch:job ref="KeyPayRemittanceFile" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="KeyPayReversalFileStep">
                    <batch:job ref="KeyPayReversalFile" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="LockBoxRemittanceFileStep">
                    <batch:job ref="LockBoxRemittanceFile" />
                    </batch:step>
                    </batch:flow>
                    </batch:split>
                    <batch:split id="Split4b" task-executor="taskExecutor">
                    <batch:flow>
                    <batch:step id="EmployerDisEnrollmentDueToNonPaymentStep">
                    <batch:job ref="EmployerDisEnrollmentDueToNonPayment" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="MonthlyAggregationEmpStep" next="UpdateEmployeeEnrollmentStatusStep">
                    <batch:job ref="MonthlyAggregationEmp" />
                    </batch:step>
                    <batch:step id="UpdateEmployeeEnrollmentStatusStep">
                    <batch:job ref="UpdateEmployeeEnrollmentStatus" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="ExchangeUsageFeeTIFileStep">
                    <batch:job ref="ExchangeUsageFeeTIFile" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="KeyPayBillingFileStep">
                    <batch:job ref="KeyPayBillingFile" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="IssuerFeeInvoiceStep" next="IssuerFeeTransactionImportFileStep">
                    <batch:job ref="IssuerFeeInvoice" />
                    </batch:step>
                    <batch:step id="IssuerFeeTransactionImportFileStep">
                    <batch:job ref="IssuerFeeTransactionImportFile" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="DailyAggregationStep" next="DailyTransactionImportFileStep">
                    <batch:job ref="DailyAggregation" />
                    </batch:step>
                    <batch:step id="DailyTransactionImportFileStep">
                    <batch:job ref="DailyTransactionImportFile" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="MonthlyAggregationIndStep" next="Split4b1">
                    <batch:job ref="MonthlyAggregationInd" />
                    </batch:step>
                    <batch:split id="Split4b1" task-executor="taskExecutor"
                    next="Split4b2">
                    <batch:flow>
                    <batch:step id="ProcessEnrollmentCompletedDataStep"
                    next="ActivateEnrollmentStep">
                    <batch:job ref="ProcessEnrollmentCompletedData" />
                    </batch:step>
                    <batch:step id="ActivateEnrollmentStep">
                    <batch:job ref="ActivateEnrollment" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="MonthlyTransactionImportFileStep">
                    <batch:job ref="MonthlyTransactionImportFile" />
                    </batch:step>
                    </batch:flow>
                    </batch:split>
                    <batch:split id="Split4b2" task-executor="taskExecutor"
                    next="Split4b3">
                    <batch:flow>
                    <batch:step id="ProcessDisEnrollNonPaymentQHPandAPTCStep">
                    <batch:job ref="ProcessDisEnrollNonPaymentQHPandAPTC" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="ProcessDisEnrollmentForAge26Step">
                    <batch:job ref="ProcessDisEnrollmentForAge26" />
                    </batch:step>
                    </batch:flow>
                    </batch:split>
                    <batch:split id="Split4b3" task-executor="taskExecutor">
                    <batch:flow>
                    <batch:step id="CoresNoActionInPEnrBeforEndOfYearStep">
                    <batch:job ref="CoresNoActionInPEnrBeforEndOfYear" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="ExpireEnrollmentStep" next="RefundMasterRecordStep">
                    <batch:job ref="ExpireEnrollment" />
                    </batch:step>
                    <batch:step id="RefundMasterRecordStep" next="RefundTransactionImportFileStep">
                    <batch:job ref="RefundMasterRecord" />
                    </batch:step>
                    <batch:step id="RefundTransactionImportFileStep">
                    <batch:job ref="RefundTransactionImportFile" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:split id="Split4b3a" task-executor="taskExecutor"
                    next="Split4b3b">
                    <batch:flow>
                    <batch:step id="IndividualInvoiceGenerationStep">
                    <batch:job ref="IndividualInvoiceGeneration" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="EmployerInvoiceGenerationStep">
                    <batch:job ref="EmployerInvoiceGeneration" />
                    </batch:step>
                    </batch:flow>
                    </batch:split>
                    <batch:split id="Split4b3b" task-executor="taskExecutor">
                    <batch:flow>
                    <batch:step id="InvoicePDFGenerationStep">
                    <batch:job ref="InvoicePDFGeneration" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="DelinquencyNoticeGenerationEprStep">
                    <batch:job ref="DelinquencyNoticeGenerationEpr" />
                    </batch:step>
                    </batch:flow>
                    <batch:flow>
                    <batch:step id="DelinquencyNoticeGenerationIndStep">
                    <batch:job ref="DelinquencyNoticeGenerationInd" />
                    </batch:step>
                    </batch:flow>
                    </batch:split>
                    </batch:flow>
                    </batch:split>
                    </batch:flow>
                    </batch:split>
                    </batch:flow>
                    </batch:split>
                    </batch:flow>
                    </batch:split>
                    </batch:job>
                    <bean id="taskExecutor"
                    class="org.springframework.scheduling.concurrent.T hreadPoolTaskExecutor">
                    <property name="corePoolSize" value="8" />
                    <property name="maxPoolSize" value="8" />
                    </bean>
                    </beans>
                    Above is the depenedency shceudle in the project.

                    6743 ExchangeUsageFeeTIFile ExchangeUsageFeeTIFileJob STARTED Jul 9, 2013 10:00:20 PM
                    6749 DailyTransactionImportFile DailyTransactionImportFileJob STARTED Jul 9, 2013 10:00:45 PM
                    6740 Process834AuditData Process834AuditDataJob STARTED Jul 9, 2013 10:00:20 PM
                    6746 IssuerFeeInvoice IssuerFeeInvoiceJob STARTED Jul 9, 2013 10:00:22 PM
                    6745 KeyPayBillingFile KeyPayBillingFileJob STARTED Jul 9, 2013 10:00:20 PM


                    But when the batch execution reached to the above batches they are always in STARTED state. Nowhere logs are not showing any exceptions. All the batch tables exit message and error message are also empty.
                    Team is claims that if the batch ran individually it works fine and also several times we see the batches are executing either complete or failed. But sporadically they are in STARTED state. Please guide us for the corrective action.

                    We are using the Spring batch 2.1.9 RELEASE version we are using


                    Another observation in case of dependency configuraiton for example : <batch:job ref="IssuerFeeInvoice" /> Unable to call the
                    IssuerFeeInvoiceReader in this case. Can some one please answer this?

                    Comment


                    • #11
                      I have resolved this issue by removing the taskExecutor configuraiton from individual batch job xml file configuration. After that all batches are working fine.
                      <bean id="taskExecutor"
                      class="org.springframework.scheduling.concurrent.T hreadPoolTaskExecutor">
                      <property name="corePoolSize" value="8" />
                      <property name="maxPoolSize" value="8" />
                      </bean>

                      Comment

                      Working...
                      X