Announcement Announcement Module
Collapse
No announcement yet.
How to relaunch Batch jobs using AOP! Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • How to relaunch Batch jobs using AOP!

    Problem:
    --------
    Environment has multiple threads, each running a Spring Integration context which polls a directory for a file and, once found, executes a Spring Batch job using
    the Spring Batch Integration functionality (described here:http://blog.springsource.org/2010/02...g-integration/ See: Message Trigger section).

    The problem was that every so often, I'd encounter a deadlock scenario where 2 threads would attempt to kick off a Spring Batch job simultaneously, resulting in a
    deadlock exception when dealing with the persisted jobRepository.

    Exception in log files
    ----------------------
    [CODE]2012-09-18 12:39:18,404 ERROR [task-scheduler-1] handler.LoggingHandler (LoggingHandler.java:126) - org.springframework.integration.MessageHandlingExc eption: org.springframework.dao.DataAccessResourceFailureE xception: Could not increment identity; nested exception is com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 172) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
    at org.springframework.integration.handler.MethodInvo kingMessageProcessor.processMessage(MethodInvoking MessageProcessor.java:76)
    at org.springframework.integration.handler.ServiceAct ivatingHandler.handleRequestMessage(ServiceActivat ingHandler.java:64)
    at org.springframework.integration.handler.AbstractRe plyProducingMessageHandler.handleMessageInternal(A bstractReplyProducingMessageHandler.java:97)
    at org.springframework.integration.handler.AbstractMe ssageHandler.handleMessage(AbstractMessageHandler. java:73)_
    at org.springframework.integration.dispatcher.Unicast ingDispatcher.doDispatch(UnicastingDispatcher.java :114)
    at org.springframework.integration.dispatcher.Unicast ingDispatcher.dispatch(UnicastingDispatcher.java:1 01)
    at org.springframework.integration.channel.AbstractSu bscribableChannel.doSend(AbstractSubscribableChann el.java:61)
    at org.springframework.integration.channel.AbstractMe ssageChannel.send(AbstractMessageChannel.java:157)
    at org.springframework.integration.channel.AbstractMe ssageChannel.send(AbstractMessageChannel.java:128)
    at org.springframework.integration.core.MessagingTemp late.doSend(MessagingTemplate.java:288)
    at org.springframework.integration.core.MessagingTemp late.send(MessagingTemplate.java:149)
    ....
    at java.lang.Thread.run(Thread.java:662)
    Caused by: org.springframework.dao.DataAccessResourceFailureE xception: Could not increment identity; nested exception is com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 172) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
    at org.springframework.jdbc.support.incrementer.SqlSe rverMaxValueIncrementer.getNextKey(SqlServerMaxVal ueIncrementer.java:108)
    at org.springframework.jdbc.support.incrementer.Abstr actDataFieldMaxValueIncrementer.nextLongValue(Abst ractDataFieldMaxValueIncrementer.java:125)
    at org.springframework.batch.core.repository.dao.Jdbc JobInstanceDao.createJobInstance(JdbcJobInstanceDa o.java:110)
    at org.springframework.batch.core.repository.support. SimpleJobRepository.createJobExecution(SimpleJobRe pository.java:131)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.springframework.aop.support.AopUtils.invokeJoi npointUsingReflection(AopUtils.java:318)
    at org.springframework.aop.framework.ReflectiveMethod Invocation.invokeJoinpoint(ReflectiveMethodInvocat ion.java:183)
    at org.springframework.aop.framework.ReflectiveMethod Invocation.proceed(ReflectiveMethodInvocation.java :150)
    at org.springframework.transaction.interceptor.Transa ctionInterceptor.invoke(TransactionInterceptor.jav a:110)
    at org.springframework.aop.framework.ReflectiveMethod Invocation.proceed(ReflectiveMethodInvocation.java :172)
    at org.springframework.batch.core.repository.support. AbstractJobRepositoryFactoryBean$1.invoke(Abstract JobRepositoryFactoryBean.java:168)
    at org.springframework.aop.framework.ReflectiveMethod Invocation.proceed(ReflectiveMethodInvocation.java :172)
    at org.springframework.aop.framework.JdkDynamicAopPro xy.invoke(JdkDynamicAopProxy.java:202)
    at $Proxy12.createJobExecution(Unknown Source)
    at org.springframework.batch.core.launch.support.Simp leJobLauncher.run(SimpleJobLauncher.java:111)
    at org.springframework.batch.integration.launch.JobLa unchingMessageHandler.launch(JobLaunchingMessageHa ndler.java:49)
    ...


    Initially, I tried configuring some retriable exceptions in my Spring Batch configuration and was still receiving the exceptions. I noticed the exception was actually thrown **before** my
    Batch functionality was invoked, which is why retriable exceptions wasn't working for me.

    Solution:
    --------
    Use AOP to use a RetryTemplate on the Batch Integration job launcher!! The existing Spring Batch documentation mentions how to do this, but I wanted to provide an example of how I got it to work using
    Batch Integration's JobLauncher.

    Basically, the Spring Batch Integration JobLaunchingMessageHandler invokes the Batch SimpleJobLauncher. I configured AOP to basically "listen" for exceptions on my SimpleJobLauncher and
    retry the job launch if they occur (for a max: 7 times).

    This setup will also log scenarios when this deadlock occurs!!! So far, the following configuration has worked for me flawlessly and now retries deadlock exceptions.
    Code:
        <!--Begin Batch/AOP retry config.  This is used to retry on potential deadlocks on SimpleJobRepository.createJobExecution()-->
        <bean id="batchRetryPolicy" class="org.springframework.batch.retry.policy.SimpleRetryPolicy" >
            <property name="maxAttempts" value="7"></property>
            <property name="retryableExceptions">
                <util:map>
                    <entry key="org.springframework.dao.DeadlockLoserDataAccessException" value="true"/>
                    <entry key="org.springframework.dao.DataAccessResourceFailureException" value="true"/>
                </util:map>
            </property>
        </bean>
    
        <bean id="batchRetryTemplate" class="org.springframework.batch.retry.support.RetryTemplate" >
            <property name="retryPolicy" ref="batchRetryPolicy"></property>
            <property name="listeners">
                <bean id="retryLogger" class="com.wf.fx.tradefeed.batch.aop.LoggingRetryListener"/>
            </property>
        </bean>
    
        <bean id="batchRetryAdvice"
              class="org.springframework.batch.retry.interceptor.RetryOperationsInterceptor" >
            <property name="retryOperations" ref="batchRetryTemplate"></property>
        </bean>
    
        <aop:config>
            <aop:pointcut id="launching"
                          expression="execution(* org.springframework.batch.core.launch.support.SimpleJobLauncher.*(..))"></aop:pointcut>
            <aop:advisor pointcut-ref="launching" advice-ref="batchRetryAdvice" order="-1"></aop:advisor>
        </aop:config>
        <!--End Batch/AOP retry config-->
    I wanted to share my solution in case anyone else encountered this issue. Good luck!!!

    -Chris
Working...
X