Announcement Announcement Module
Collapse
No announcement yet.
My Steps are hanging waiting on a semphore Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • My Steps are hanging waiting on a semphore

    I am trying to use Spring Batch 2.1.1 with Spring 3.0.2, but am seeing my step hang when it tries to process the second chuck of data.

    My step hangs in TaskStep$ChunkTransactionCallback while trying to acquire the semaphore, which has not been released from the first chunk.

    I have attached the logs and my spring config files.

    Does anyone know what's going on?

    This the last bit of the log.


    2010-05-01 11:25:20,786 TRACE [org.springframework.transaction.interceptor.Transa ctionInterceptor] - Getting transaction for [org.springframework.batch.core.repository.JobRepos itory.update]
    2010-05-01 11:25:20,789 TRACE [org.springframework.transaction.interceptor.Transa ctionInterceptor] - Completing transaction for [org.springframework.batch.core.repository.JobRepos itory.update]
    2010-05-01 11:25:20,789 DEBUG [org.springframework.batch.support.transaction.Reso urcelessTransactionManager] - Initiating transaction commit
    2010-05-01 11:25:20,789 TRACE [org.springframework.transaction.support.Transactio nSynchronizationManager] - Clearing transaction synchronization
    2010-05-01 11:25:20,789 DEBUG [org.springframework.batch.support.transaction.Reso urcelessTransactionManager] - Resuming suspended transaction after completion of inner transaction
    2010-05-01 11:25:20,789 TRACE [org.springframework.transaction.support.Transactio nSynchronizationManager] - Initializing transaction synchronization
    2010-05-01 11:25:20,791 DEBUG [org.springframework.batch.repeat.support.RepeatTem plate] - Repeat operation about to start at count=2
    2010-05-01 11:25:20,791 DEBUG [org.springframework.batch.core.scope.context.StepC ontextRepeatCallback] - Preparing chunk execution for StepContext: org.springframework.batch.core.scope.context.StepC ontext@20dbd794
    2010-05-01 11:25:20,791 DEBUG [org.springframework.batch.core.scope.context.StepC ontextRepeatCallback] - Chunk execution starting: queue size=0
    2010-05-01 11:25:20,791 TRACE [org.springframework.transaction.support.Transactio nSynchronizationManager] - Retrieved value [org.springframework.orm.hibernate3.SessionHolder@1 90c0fa7] for key [org.hibernate.impl.SessionFactoryImpl@3f78e13f] bound to thread [main]
    2010-05-01 11:25:20,809 TRACE [org.springframework.transaction.support.Transactio nSynchronizationManager] - Retrieved value [org.springframework.jdbc.datasource.ConnectionHold er@717757ad] for key [com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 1hge0w588nhwmtw1erus3g|6c63a721, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1hge0w588nhwmtw1erus3g|6c63a721, idleConnectionTestPeriod -> 60, initialPoolSize -> 1, jdbcUrl -> jdbc:mysql://localhost:3306/test, lastAcquisitionFailureDefaultUser -> null, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 15, maxStatements -> 180, maxStatementsPerConnection -> 0, minPoolSize -> 3, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> true, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ]] bound to thread [main]
    2010-05-01 11:25:20,809 DEBUG [org.springframework.batch.repeat.support.RepeatTem plate] - Starting repeat context.
    2010-05-01 11:25:20,809 DEBUG [org.springframework.batch.repeat.support.RepeatTem plate] - Repeat operation about to start at count=1
    2010-05-01 11:25:20,809 DEBUG [org.springframework.batch.repeat.support.RepeatTem plate] - Repeat operation about to start at count=2
    2010-05-01 11:25:20,810 DEBUG [org.springframework.batch.repeat.support.RepeatTem plate] - Repeat operation about to start at count=3
    2010-05-01 11:25:20,810 DEBUG [org.springframework.batch.repeat.support.RepeatTem plate] - Repeat operation about to start at count=4
    2010-05-01 11:25:20,810 DEBUG [org.springframework.batch.repeat.support.RepeatTem plate] - Repeat operation about to start at count=5
    2010-05-01 11:25:20,810 DEBUG [org.springframework.batch.repeat.support.RepeatTem plate] - Repeat is complete according to policy and result value.
    2010-05-01 11:25:20,810 DEBUG [org.springframework.batch.core.step.item.ChunkOrie ntedTasklet] - Inputs not busy, ended: false

  • #2
    It looks like you are using ResourcelessTransactoinManager with Hibernate. That's probably not a good idea and not what you intended - try HibernateTransactionManager? It's not immediately obvious why that would fix a deadlock, but it rings a bell somewhere, so try it.

    Comment


    • #3
      ChunkTransactionCallback.afterCompletion is never called

      I think the problem is that the semaphore is acquired in ChunkTransactionCallback.doInTransaction(), but it is never released.

      See the following code in TaskletStep.java.

      Code:
      RepeatStatus result;
      				try {
      					result = (RepeatStatus) new TransactionTemplate(transactionManager, transactionAttribute)
      							.execute(new ChunkTransactionCallback(chunkContext));
      				}
      				catch (TransactionException e) {
      					// Allow checked exceptions to be thrown inside callback
      					throw (Exception) e.getCause();
      				}
      ChunkTransactionCallback acquires the semaphore in doInTransaction(), and releases it in afterCompletion(). There is no way to call afterCompletion on the ChunkTransactionCallback because the reference to it is never saved. TransactionTemplate only knows about doInTransaction because that is the only method interface TransactionCallback exposes.

      Seems like afterCompletion is a dangling method.

      Comment


      • #4
        The afterCompletion() method is a callback registered explicitly in doInTransaction(), so if it is not being called in your case I suspect that you have some iffy transaction manager configuration (as I already said). Make sure if you are using a single transaction manager and that it is consistent with the resources you are using (e.g. Hibernate in this case apparently).

        Comment


        • #5
          My Steps are still hanging

          I looked at the log again and the ResourcelessTransactionManager was being used because MapJobRepositoryBean uses it, I switched to using JobRepositoryFactoryBean and my hibernate transaction manager is now being used throughout.

          But still my job hangs when it tries to process the second chunk.

          I have attached the newest log, please take a look.

          The batch runs successfully if I set my commit-interval to something high like 30,000.

          Comment


          • #6
            It looks like your transactions are never committed (so the semaphore is never released). How did you launch the job? Make sure you don't do it synchronously from a @Transactional service method. If you can't spot the problem post your config and the code that calls the JobLauncher.

            Comment


            • #7
              I took all for the transaction annotations away from my JUnit test and the batch started to work. Unfortunately this problem took a long time to figure out :-(

              Thanks for the help!

              Comment


              • #8
                I've just lost a fair few hours of my time struggling with the exact same problem, funnily enough in my case I had to add a ResourceLessTransactionManager to solve the issue.

                I'm hoping this post will point people with the same setup in the right direction because the error isn't very obvious to spot IMVHO. So anyway, here's my use case:

                In my writer I am using the ApplicationContextJobFactory to triggering a nested job. As the outer job and the inner job were using the same TransactionManager I was facing the same semaphore problem described in this thread on execution of the second chunk of the inner job. Changing the TransactionManager of the outer job to a ResourceLessTransactionManager solved the issue.

                Comment


                • #9
                  ResourcelessTransactionManager shouldn't be used in production. It would be better to simply use Propagation.NOT_SUPPORTED or a background thread to launch the sub-job.

                  Comment


                  • #10
                    Unit testing and rollback?

                    Sorry to revive this thread, but has it then become impossible to rollback batch jobs when running in a integration/unit test?

                    Until Spring Batch 2.1 I've used this quite a lot to test jobs end-to-end on a separate database without doing any cleaning up afterwards. Is this still possible with a rollback somehow or should I start cleaning up manually from now on? If there's a better solution built-in 2.1 and later, feel free to let me know.

                    Comment


                    • #11
                      I'm actually surprised it ever worked. I always use JobRepositoryTestUtils and JdbcTemplate to clean up after a Job integration test.

                      If you'd like to raise a JIRA ticket it might be a decent target for 3.0 to support a single transaction for testing jobs. It's not really a great idea for production use because jobs tend to run too long for a single transaction.

                      Comment


                      • #12
                        Originally posted by Dave Syer View Post
                        I'm actually surprised it ever worked. I always use JobRepositoryTestUtils and JdbcTemplate to clean up after a Job integration test.

                        If you'd like to raise a JIRA ticket it might be a decent target for 3.0 to support a single transaction for testing jobs. It's not really a great idea for production use because jobs tend to run too long for a single transaction.
                        Well, it might have been more luck than anything else since I assume that having more data than the commit size should blow up my test anyway.
                        I do agree with the statement about production but I'll think about it if it would be a good idea for integration/unit testing. If I can't find any (real) downsides I'll raise a JIRA ticket for this. I might actually develop a quick 'n dirty solution for unit testing myself.

                        Comment

                        Working...
                        X