Announcement Announcement Module
Collapse
No announcement yet.
Help: Package Spring Batch Jobs As Jar/Include As Dependency of Spring WebApp Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Help: Package Spring Batch Jobs As Jar/Include As Dependency of Spring WebApp

    Hi all,

    I have a question that I can't seem to solve alone, and I haven't found an applicable answer online for my current problem.

    Specifically, I have a very large (enterprise) Spring application that has been in production for years. Currently, we need to add functionality that is nicely solved by Spring Batch (+ Quartz), and to that end, we have successfully implemented the desired functionality using these two frameworks. Moreover, we were able to integrate these two frameworks into the project in a "self-contained" fashion.

    Given that the project is already large in scope, and that we were able to code up these features in a self-contained sub-package, we figure it would be optimal if we could define all of the Spring Batch/Quartz jobs in an independent library, packaged as a jar, and simply add this dependency to the main application. I have successfully been able to isolate the code into it's own library, and similarly, have been able to package it as an executable jar.

    When the spring batch/quartz jobs are defined within the main application, and run code that is packaged in the main application, the (four) periodic jobs that are managed by Quartz run on schedule, and similarly, the (four) jobs that we manually launch when specific REST endpoints are hit all work as expected.

    Unfortunately, when I introduce the Spring Batch / Quartz dependencies via the Jar, I have no such luck. The four periodic Quartz jobs never execute. Furthermore, the four jobs that are manually triggered by specific HTTP calls emit the following behavior: the job is launched, and the first Tasklet in the first step of the job is executed once, but flow is never passed beyond that tasklet, and no other steps execute.

    I've attempted adding a "dummy" Main class as an entrypoint that simply loads the beans I defined in the application context. Similarly, I have tried taking the applicationContext from the library, adding it as an additional xml file in the main project, and importing that in the main application context. Lastly, I have tried importing the application context from the "library" from the classpath introduced by the jar dependency. While this makes the beans defined in the independent library visible in the main application, they do not function as they did when they were part of the main application (that is, the quartz jobs don't run, and the spring-batch jobs do not execute their entire flow).

    Does anybody have any guidance as to what I am doing wrong? It seems to me that this is likely a case where a slight misconfiguration on my end is preventing me from achieving the desired behavior. However, I cannot find an example online of a non-trivial set of batch routines being self-contained in a library and imported/run in a larger application via jar packaging. I feel silly even suggesting it, but: the functionality I am seeking is surely possible using spring-batch, correct?

    Any help/guidance would be tremendously appreciated.

    Thank you kindly for your time.

    PS if interested, the plugin configuration I am using to build my jar is below:

    Code:
              <plugin>
                    <artifactId>maven-assembly-plugin</artifactId>
                    <version>2.4</version>
                    <configuration>
                        <descriptorRefs>
                            <descriptorRef>jar-with-dependencies</descriptorRef>
                        </descriptorRefs>
                        <archive>
                            <manifest>
                                <mainClass>com.redacted.redacted.helpers.Main</mainClass>
                            </manifest>
                        </archive>
                    </configuration>
                    <executions>
                        <execution>
                            <id>make-assembly</id> <!-- this is used for inheritance merges -->
                            <phase>package</phase> <!-- bind to the packaging phase -->
                            <goals>
                                <goal>single</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
    Last edited by wokkaflokka; May 17th, 2013, 12:31 AM.

  • #2
    1. Are you seeing the Spring Batch code being bootstrapped when you start the web application?
    2. If so, are you seeing your Quartz job running at all?

    Comment


    • #3
      Thanks for responding.

      1. When I run the project with all spring batch code and dependencies as part of the project, the Spring Batch code bootstraps at startup, and 2) the quartz jobs run as scheduled.

      The normal project set up is as follows: I have a subpackage named spring.batch, and all code related to Spring Batch is contained therein (save the xml configuration, which is in the resources package, and is loaded in the application context). This configuration is likely the normal configuration, and works as expected. The subpackage spring.batch has no external dependencies (aside from Spring Batch, Quartz, etc. So more specifically, it has no dependencies on the main project).

      When I create a new project and move all of the code and configuration from this "spring.batch" subpackage to the new project, and then I cut a jar of this new project, I begin to experience the described problems. Specifically, I can import the project as a jar dependency in my main POM, and I have access to the files packaged in the jar (that is, intellij allows me to import and call that code within the main project). However, in this instance:

      1. I do not see the Spring Batch code being bootstrapped.
      2. The Quartz jobs do not run at all.
      3. When I invoke RESTful endpoints that manually launch the jobs, only the first step is executed.

      I would be willing to expose more details about my specific configuration if desired, and perhaps even show some of the code. However, I would need to do this in a non-public setting due to the nature of the project.

      Once again, any and all help is greatly appreciated.

      Thanks again!

      Comment


      • #4
        So what are you doing to bootstrap spring batch in your web app? Unless you bootstrap it in some way, nothing will happen. The easiest way would be to import the config in your jar file into the ones you used in your web app.

        Comment


        • #5
          Sorry, let me be more clear:

          I have tried the following bootstrap methods:

          1) import the XML file from the jar in the application context
          2) copy/paste the XML from the config that is embedded in the jar into a config file in the main project, and import that

          Neither seems to work.

          Here's my applicationContext.xml :
          Code:
          <?xml version="1.0" encoding="UTF-8"?>
          <beans xmlns="http://www.springframework.org/schema/beans"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xmlns:context="http://www.springframework.org/schema/context"
                 xsi:schemaLocation="
                  http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
                  http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd">
          
              <context:component-scan base-package="com.redacted.redacted.resources"/>
          
              <import resource="propertyConfigurerBeans.xml"/>
          
              <import resource="jmxBeans.xml"/>
          
              <import resource="projectBeans.xml"/>
          
              <import resource="rrdBeans.xml"/>
              
              <import resource="partnerBeans.xml"/>
          
              <import resource="security.xml"/>
          
              <import resource="viewHelperBeans.xml"/>
          
              <import resource="clients.xml"/>
          
              <!--<import resource="pushBatchingBeans.xml"/>-->
          
              <import resource="classpath:com/redacted/library/spring/batch/applicationContext.xml"/>
          
          </beans>
          Note: the file "pushBatchingBeans.xml" is identical to the classpath import, except the classpath import is the xml from the jar file, and the other is the dependency when not imported via jar. The configuration above is the failing configuration. If I uncomment the pushBatchingBeans import, and instead comment out the classpath import, things work as expected.

          Comment


          • #6
            1. If you turn on debugging, can you see the batch beans being created?
            2. Are you using the AutomaticJobRegistrar to create your job registry?

            Comment


            • #7
              1) No, let me try and report back.
              2) No....I wasn't even aware of that Should I be using it? A cursory glance at the documentation suggests it might be of use in my situation.

              Comment


              • #8
                When I turn on debugging, the beans appear to get created. The output was massive, so here's a very small window that suggests the beans are indeed being initialized:

                Code:
                [05/20/13 16:25:46:241](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'org.springframework.batch.core.scope.StepScope#0'
                [05/20/13 16:25:46:241](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'taskExecutor'
                [05/20/13 16:25:46:241](main) DEBUG - DefaultListableBeanFactory - Creating shared instance of singleton bean 'urbanAirshipClient'
                [05/20/13 16:25:46:241](main) DEBUG - DefaultListableBeanFactory - Creating instance of bean 'urbanAirshipClient'
                [05/20/13 16:25:46:245](main) DEBUG - DefaultListableBeanFactory - Eagerly caching bean 'urbanAirshipClient' to allow for resolving potential circular references
                [05/20/13 16:25:46:247](main) DEBUG - DefaultListableBeanFactory - Invoking afterPropertiesSet() on bean with name 'urbanAirshipClient'
                [05/20/13 16:25:46:323](main) DEBUG - DefaultListableBeanFactory - Finished creating instance of bean 'urbanAirshipClient'
                [05/20/13 16:25:46:323](main) DEBUG - DefaultListableBeanFactory - Creating shared instance of singleton bean 'mongoDeviceDAO'
                [05/20/13 16:25:46:323](main) DEBUG - DefaultListableBeanFactory - Creating instance of bean 'mongoDeviceDAO'
                [05/20/13 16:25:46:325](main) DEBUG - DefaultListableBeanFactory - Eagerly caching bean 'mongoDeviceDAO' to allow for resolving potential circular references
                [05/20/13 16:25:46:330](main) DEBUG - DefaultListableBeanFactory - Invoking init method  'init' on bean with name 'mongoDeviceDAO'
                [05/20/13 16:25:46:433](main) DEBUG - DefaultListableBeanFactory - Finished creating instance of bean 'mongoDeviceDAO'
                [05/20/13 16:25:46:433](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'pushNotificationsQueue'
                [05/20/13 16:25:46:433](main) DEBUG - DefaultListableBeanFactory - Creating shared instance of singleton bean 'queueClearingJobDetails'
                [05/20/13 16:25:46:433](main) DEBUG - DefaultListableBeanFactory - Creating instance of bean 'queueClearingJobDetails'
                [05/20/13 16:25:46:434](main) DEBUG - DefaultListableBeanFactory - Eagerly caching bean 'queueClearingJobDetails' to allow for resolving potential circular references
                [05/20/13 16:25:46:440](main) DEBUG - DefaultListableBeanFactory - Creating shared instance of singleton bean 'pushQueueClearingJob'
                [05/20/13 16:25:46:440](main) DEBUG - DefaultListableBeanFactory - Creating instance of bean 'pushQueueClearingJob'
                [05/20/13 16:25:46:440](main) DEBUG - DefaultListableBeanFactory - Eagerly caching bean 'pushQueueClearingJob' to allow for resolving potential circular references
                [05/20/13 16:25:46:442](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'pushNotificationsQueue'
                [05/20/13 16:25:46:442](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'urbanAirshipClient'
                [05/20/13 16:25:46:442](main) DEBUG - DefaultListableBeanFactory - Finished creating instance of bean 'pushQueueClearingJob'
                [05/20/13 16:25:46:442](main) DEBUG - DefaultListableBeanFactory - Invoking afterPropertiesSet() on bean with name 'queueClearingJobDetails'
                [05/20/13 16:25:46:444](main) DEBUG - DefaultListableBeanFactory - Finished creating instance of bean 'queueClearingJobDetails'
                [05/20/13 16:25:46:444](main) DEBUG - DefaultListableBeanFactory - Creating shared instance of singleton bean 'queueClearingTrigger'
                [05/20/13 16:25:46:444](main) DEBUG - DefaultListableBeanFactory - Creating instance of bean 'queueClearingTrigger'
                [05/20/13 16:25:46:448](main) DEBUG - DefaultListableBeanFactory - Eagerly caching bean 'queueClearingTrigger' to allow for resolving potential circular references
                [05/20/13 16:25:46:454](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'queueClearingJobDetails'
                [05/20/13 16:25:46:454](main) DEBUG - DefaultListableBeanFactory - Invoking afterPropertiesSet() on bean with name 'queueClearingTrigger'
                [05/20/13 16:25:46:454](main) DEBUG - DefaultListableBeanFactory - Finished creating instance of bean 'queueClearingTrigger'
                [05/20/13 16:25:46:454](main) DEBUG - DefaultListableBeanFactory - Creating shared instance of singleton bean 'queueClearingScheduler'
                [05/20/13 16:25:46:454](main) DEBUG - DefaultListableBeanFactory - Creating instance of bean 'queueClearingScheduler'
                [05/20/13 16:25:46:457](main) DEBUG - DefaultListableBeanFactory - Eagerly caching bean 'queueClearingScheduler' to allow for resolving potential circular references
                [05/20/13 16:25:46:462](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'queueClearingJobDetails'
                [05/20/13 16:25:46:462](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'queueClearingTrigger'
                [05/20/13 16:25:46:462](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'taskExecutor'
                [05/20/13 16:25:46:462](main) DEBUG - DefaultListableBeanFactory - Invoking afterPropertiesSet() on bean with name 'queueClearingScheduler'
                [05/20/13 16:25:46:516](main) DEBUG - DefaultListableBeanFactory - Finished creating instance of bean 'queueClearingScheduler'
                [05/20/13 16:25:46:516](main) DEBUG - DefaultListableBeanFactory - Creating shared instance of singleton bean 'feedbackJobDetails'
                [05/20/13 16:25:46:516](main) DEBUG - DefaultListableBeanFactory - Creating instance of bean 'feedbackJobDetails'
                [05/20/13 16:25:46:516](main) DEBUG - DefaultListableBeanFactory - Eagerly caching bean 'feedbackJobDetails' to allow for resolving potential circular references
                [05/20/13 16:25:46:516](main) DEBUG - DefaultListableBeanFactory - Creating shared instance of singleton bean 'pushHelper'
                [05/20/13 16:25:46:516](main) DEBUG - DefaultListableBeanFactory - Creating instance of bean 'pushHelper'
                [05/20/13 16:25:46:517](main) DEBUG - DefaultListableBeanFactory - Eagerly caching bean 'pushHelper' to allow for resolving potential circular references
                [05/20/13 16:25:46:519](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'registrationLauncher'
                [05/20/13 16:25:46:519](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'badgeLauncher'
                [05/20/13 16:25:46:519](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'feedbackLauncher'
                [05/20/13 16:25:46:519](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'inquiryLauncher'
                [05/20/13 16:25:46:520](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'registerDevice'
                [05/20/13 16:25:46:520](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'updateBadgeCounts'
                [05/20/13 16:25:46:520](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'processUAFeedback'
                [05/20/13 16:25:46:520](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'sendInquiryNotification'
                [05/20/13 16:25:46:520](main) DEBUG - DefaultListableBeanFactory - Finished creating instance of bean 'pushHelper'
                [05/20/13 16:25:46:520](main) DEBUG - DefaultListableBeanFactory - Invoking afterPropertiesSet() on bean with name 'feedbackJobDetails'
                [05/20/13 16:25:46:520](main) DEBUG - DefaultListableBeanFactory - Finished creating instance of bean 'feedbackJobDetails'
                [05/20/13 16:25:46:520](main) DEBUG - DefaultListableBeanFactory - Creating shared instance of singleton bean 'feedbackTrigger'
                [05/20/13 16:25:46:520](main) DEBUG - DefaultListableBeanFactory - Creating instance of bean 'feedbackTrigger'
                [05/20/13 16:25:46:521](main) DEBUG - DefaultListableBeanFactory - Eagerly caching bean 'feedbackTrigger' to allow for resolving potential circular references
                [05/20/13 16:25:46:521](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'feedbackJobDetails'
                [05/20/13 16:25:46:521](main) DEBUG - DefaultListableBeanFactory - Invoking afterPropertiesSet() on bean with name 'feedbackTrigger'
                [05/20/13 16:25:46:521](main) DEBUG - DefaultListableBeanFactory - Finished creating instance of bean 'feedbackTrigger'
                [05/20/13 16:25:46:521](main) DEBUG - DefaultListableBeanFactory - Creating shared instance of singleton bean 'feedbackScheduler'
                [05/20/13 16:25:46:521](main) DEBUG - DefaultListableBeanFactory - Creating instance of bean 'feedbackScheduler'
                [05/20/13 16:25:46:521](main) DEBUG - DefaultListableBeanFactory - Eagerly caching bean 'feedbackScheduler' to allow for resolving potential circular references
                [05/20/13 16:25:46:521](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'feedbackJobDetails'
                [05/20/13 16:25:46:522](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'feedbackTrigger'
                [05/20/13 16:25:46:522](main) DEBUG - DefaultListableBeanFactory - Returning cached instance of singleton bean 'taskExecutor'
                [05/20/13 16:25:46:522](main) DEBUG - DefaultListableBeanFactory - Invoking afterPropertiesSet() on bean with name 'feedbackScheduler'
                [05/20/13 16:25:46:523](main) DEBUG - DefaultListableBeanFactory - Finished creating instance of bean 'feedbackScheduler'
                Thanks again for the time you've given me so far.

                Comment


                • #9
                  Using an AutomaticJobRegistrar has solved some of my problems. Thanks for pointing that out! Its been about a month since a gave the documentation a thorough read-through, seems to have slipped my mind.

                  Now, I can invoke the job's that I expose via REST (that is, all of my jobs that are launched "manually" seem to be working properly).

                  The Quartz jobs also run, however, I'm sure there is an issue with my configuration as I see the following error message whenever Quartz triggers a job:

                  Code:
                  org.springframework.dao.OptimisticLockingFailureException: Attempt to update step execution id=1 with wrong version (1), where current version is 2
                  ...
                  Caused by: org.springframework.dao.OptimisticLockingFailureException: Attempt to update step execution id=4 with wrong version (3), where current version is 1
                  I only configured a simplistic, proof-of-concept AutomaticJobRegistrar, so I presume it has something to do with that. My bean is configured as follows:

                  Code:
                      <bean class="org.springframework.batch.core.configuration.support.AutomaticJobRegistrar">
                          <property name="applicationContextFactories">
                              <bean class="org.springframework.batch.core.configuration.support.ClasspathXmlApplicationContextsFactoryBean">
                                  <property name="resources" value="classpath:/com/redacted/project/spring/batch/applicationContext.xml" />
                              </bean>
                          </property>
                          <property name="jobLoader">
                              <bean class="org.springframework.batch.core.configuration.support.DefaultJobLoader">
                                  <property name="jobRegistry">
                                      <bean class="org.springframework.batch.core.configuration.support.MapJobRegistry"/>
                                  </property>
                              </bean>
                          </property>
                      </bean>

                  Comment


                  • #10
                    We also have a sample job that uses Quartz you should check out: http://static.springsource.org/sprin...amples/#quartz. It addresses the background job runner and creating the job registry.

                    Comment


                    • #11
                      What jobRepository are you using?

                      Comment


                      • #12
                        I'm checking out the Quartz sample now.

                        Since this is in test we've just been using a MapJobRepositoryFactoryBean. So the repository producing that error is a SimpleJobRepository.

                        As an aside, we are still unsure what jobRepository we will want to use in production. Our use case is slightly unorthodox, but we came to the conclusion that spring batch provided a lot of nice constructs to support a push notification service for an enterprise app with a sizeable user base. However, from what I can tell, the SimpleJobRepository isn't really suited for production purposes. On the other hand, we don't necessarily need an audit trail of every push job we've run from the beginning of time.

                        Comment


                        • #13
                          Following up on this, I'm still seeing this error. It seems to be that the step id's are getting muddled, I have little clue as to why. I'll note that each step invokes a custom Tasklet, and these Tasklet's are each configured with @scope=step.

                          Code:
                          [05/21/13 09:45:18:766](SimpleAsyncTaskExecutor-3) ERROR - TaskletStep                - JobRepository failure forcing exit with unknown status
                          org.springframework.dao.OptimisticLockingFailureException: Attempt to update step execution id=1 with wrong version (2), where current version is 1
                          	at org.springframework.batch.core.repository.dao.MapStepExecutionDao.updateStepExecution(MapStepExecutionDao.java:98)
                          	at org.springframework.batch.core.repository.support.SimpleJobRepository.update(SimpleJobRepository.java:171)
                          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          	at java.lang.reflect.Method.invoke(Method.java:597)
                          	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
                          ....
                          ....
                          [05/21/13 09:45:18:772](SimpleAsyncTaskExecutor-3) ERROR - AbstractStep               - Encountered an error executing the step
                          org.springframework.batch.core.step.FatalStepExecutionException: JobRepository failure forcing exit with unknown status
                          	at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:441)
                          	...
                          Caused by: org.springframework.dao.OptimisticLockingFailureException: Attempt to update step execution id=1 with wrong version (2), where current version is 1
                          	at org.springframework.batch.core.repository.dao.MapStepExecutionDao.updateStepExecution(MapStepExecutionDao.java:98)
                          	
                          	... 17 more
                          [05/21/13 09:45:18:773](SimpleAsyncTaskExecutor-3) ERROR - AbstractStep               - Encountered an error saving batch meta data. This job is now in an unknown state and should not be restarted.
                          org.springframework.dao.OptimisticLockingFailureException: Attempt to update step execution id=1 with wrong version (1), where current version is 2
                          	at org.springframework.batch.core.repository.dao.MapStepExecutionDao.updateStepExecution(MapStepExecutionDao.java:98)
                          
                          ....
                          
                          [05/21/13 09:45:18:780](SimpleAsyncTaskExecutor-3) ERROR - AbstractJob                - Encountered fatal error executing job
                          org.springframework.batch.core.JobExecutionException: Flow execution ended unexpectedly
                          	at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:141)
                          	at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293)
                          	at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
                          	at java.lang.Thread.run(Thread.java:680)
                          Caused by: org.springframework.batch.core.job.flow.FlowExecutionException: Ended flow=processUAFeedback at state=processUAFeedback.syncMongo with exception
                          	at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:152)
                          	at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
                          	at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
                          	... 3 more
                          Caused by: org.springframework.dao.OptimisticLockingFailureException: Attempt to update step execution id=1 with wrong version (1), where current version is 2
                          	at org.springframework.batch.core.repository.dao.MapStepExecutionDao.updateStepExecution(MapStepExecutionDao.java:98)
                          	....

                          Comment


                          • #14
                            Just following up to provide some resolution:

                            The resolution to this is to use the AutomaticJobRegistrar, which - as the documentation notes - helps manage jobs imported from child contexts.

                            As always, a detailed reading and understanding of the documentation proves itself worthwhile.

                            Comment

                            Working...
                            X