Announcement Announcement Module
Collapse
No announcement yet.
Transaction manager with In-Memory Derby JobRepository + MySQL DB for data Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Transaction manager with In-Memory Derby JobRepository + MySQL DB for data

    Hi,

    What are the implications/caveats of using non-JTA TransactionManager, while having JobRepository as an In-Memory DB (H2 or Derby) and having business data in MySQL DB, being read by JpaPagingItemRead and updated by JpaItemWriter ?

    Thanks

    Jakub

  • #2
    Spring Batch can't guarantee the synchronization between business data and batch metadata in case of failure (e.g. the number of skipped items could be wrong). This could happen for corner cases, not on a "sunny day".

    Comment


    • #3
      And how about having actually two transaction managers , one for the Spring Batch (in-mem) DB and the other for the business data DB ?

      Comment


      • #4
        They should bind to different threads. You can have multiple transaction managers tied into different datasources. Does this clarify your question?

        Comment


        • #5
          They for sure will be tied to different datasources, one into the Spring Batch in-mem Derby DB (for job-repository) and the other into the business data DB. But as regards them being bound to different threads, I'm not sure about this one. I don't think I can tell Spring Batch to somehow manage this. Or do they really have to be each in a separate thread ?
          Last edited by Jakub.Kahovec; Jul 4th, 2011, 03:23 PM.

          Comment


          • #6
            There's a rudimentary example of synchronizing 2 database commits in my article in Javaworld (http://www.javaworld.com/javaworld/j...nsactions.html) - it could do with some tweaks but it probably works for simple use cases. Spring doesn't provide a transaction manager that works this way out of the box (mainly because it is buyer beware - you have to be confident you know what you are doing), but all the APIs are there to make your own. As explained in the article, it doesn't give you the full protection of XA, and you need to be prepared for partial failures.

            Comment


            • #7
              Originally posted by Dave Syer View Post
              There's a rudimentary example of synchronizing 2 database commits in my article in Javaworld (http://www.javaworld.com/javaworld/j...nsactions.html) - it could do with some tweaks but it probably works for simple use cases. Spring doesn't provide a transaction manager that works this way out of the box (mainly because it is buyer beware - you have to be confident you know what you are doing), but all the APIs are there to make your own. As explained in the article, it doesn't give you the full protection of XA, and you need to be prepared for partial failures.
              Dave is he trying to synchronize the db commits or is he trying to just use two different datasources and transaction managers? I think it is the latter, but I could be wrong

              Comment


              • #8
                It doesn't really make much sense to use two transaction managers, especially with a Batch application. I was just hinting that it might be possible to use a synchronization approach (best efforts 1 phase commit) if you understand the risks.

                Comment


                • #9
                  Well, the reason I was asking about using two transaction manager in the first place is that in my scenario I'm using embedded in-mem DB for Spring Batch data (originally used MapRepository but was suggested (also by Dave I think) to use an in-mem DB instead due to its transactional nature). The thing is, that for business data I'm using another DB and I can't use a JTA transaction manager. So I was using a JPA transaction manager defined for my business data DB also for the embedded DB used by Spring Batch (which I'm not sure its a correct way) and occasionally I'm getting ab OptimimistLockingException on Spring Batch saying that there was an incorrect version of step while updating it. I'm not sure if this can by caused by using one transaction manager for both DBs, so that's why I was asking about using two, each for its own DB.

                  Comment


                  • #10
                    Originally posted by Jakub.Kahovec View Post
                    Well, the reason I was asking about using two transaction manager in the first place is that in my scenario I'm using embedded in-mem DB for Spring Batch data (originally used MapRepository but was suggested (also by Dave I think) to use an in-mem DB instead due to its transactional nature). The thing is, that for business data I'm using another DB and I can't use a JTA transaction manager. So I was using a JPA transaction manager defined for my business data DB also for the embedded DB used by Spring Batch (which I'm not sure its a correct way) and occasionally I'm getting ab OptimimistLockingException on Spring Batch saying that there was an incorrect version of step while updating it. I'm not sure if this can by caused by using one transaction manager for both DBs, so that's why I was asking about using two, each for its own DB.
                    Jakub

                    First you do not want to use the embedded in-mem DB for the Spring Batch tables (it is used automatically when you do not configure batch to use the db you want it to use). I would recommend using the mysql db. The h2 database is primarily used for testing, and I would not recommend to use it in production. Let me wrap up what Dave and I where talking about. First I mentioned that you can use two different transaction manager, but Dave pointed out that he would not recommend it. Dave's reason (please correct me if I am wrong) is that all of the transactions of the batch must rollback together. The batch table writes and the writes into mysql need to be handled with-in the same transaction. This allows batch to have such functionality as, stop, restart and other recovery components.

                    Does this help?

                    Comment


                    • #11
                      Originally posted by cnmconsulting View Post
                      Jakub

                      First you do not want to use the embedded in-mem DB for the Spring Batch tables (it is used automatically when you do not configure batch to use the db you want it to use). I would recommend using the mysql db. The h2 database is primarily used for testing, and I would not recommend to use it in production. Let me wrap up what Dave and I where talking about. First I mentioned that you can use two different transaction manager, but Dave pointed out that he would not recommend it. Dave's reason (please correct me if I am wrong) is that all of the transactions of the batch must rollback together. The batch table writes and the writes into mysql need to be handled with-in the same transaction. This allows batch to have such functionality as, stop, restart and other recovery components.

                      Does this help?
                      As I mentioned earlier, my main concern is to avoid the OptimisticLockingException being thrown by Spring Batch saying that there was an incorrect version of step while updating it. As I understand from the posts having two transaction managers wouldn't sort this out, right ? So what else I can do to not to get this exception ? Couldn't this be a bug in Spring Batch ? I'm using version 2.1.7.

                      Comment


                      • #12
                        I would recommend using 2.1.8 over 2.1.7, but that won't necessarily make locking errors go away. There could be a bug, I suppose, but we haven't got enough information to see what you do - this is the first I heard about your OptimisticLockingException. All I can say for sure is that using a second database and transaction manager is definitely not the way to fix it.

                        Comment


                        • #13
                          I actually tried 2.1.8 as well but it didn't help.

                          The OptimisticLockingException usually occurs when something fails during the step execution.

                          Here is a stacktrace:
                          Caused by: Exception [EclipseLink-5006] (Eclipse Persistence Services - 2.2.0.v20110202-r8913): org.eclipse.persistence.exceptions.OptimisticLockE xception
                          Exception Description: The object table.View@2302 cannot be updated because it has changed or been deleted since it was last read.
                          Class> table.View Primary Key> 754
                          at org.eclipse.persistence.exceptions.OptimisticLockE xception.objectChangedSinceLastReadWhenUpdating(Op timisticLockException.java:137)
                          at org.eclipse.persistence.descriptors.VersionLocking Policy.validateUpdate(VersionLockingPolicy.java:76 6)
                          at org.eclipse.persistence.internal.queries.DatabaseQ ueryMechanism.updateObjectForWriteWithChangeSet(Da tabaseQueryMechanism.java:1167)
                          at org.eclipse.persistence.queries.UpdateObjectQuery. executeCommitWithChangeSet(UpdateObjectQuery.java: 84)
                          at org.eclipse.persistence.internal.queries.DatabaseQ ueryMechanism.executeWriteWithChangeSet(DatabaseQu eryMechanism.java:291)
                          at org.eclipse.persistence.queries.WriteObjectQuery.e xecuteDatabaseQuery(WriteObjectQuery.java:58)
                          at org.eclipse.persistence.queries.DatabaseQuery.exec ute(DatabaseQuery.java:808)
                          at org.eclipse.persistence.queries.DatabaseQuery.exec uteInUnitOfWork(DatabaseQuery.java:711)
                          at org.eclipse.persistence.queries.ObjectLevelModifyQ uery.executeInUnitOfWorkObjectLevelModifyQuery(Obj ectLevelModifyQuery.java:108)
                          at org.eclipse.persistence.queries.ObjectLevelModifyQ uery.executeInUnitOfWork(ObjectLevelModifyQuery.ja va:85)
                          at org.eclipse.persistence.internal.sessions.UnitOfWo rkImpl.internalExecuteQuery(UnitOfWorkImpl.java:28 42)
                          at org.eclipse.persistence.internal.sessions.Abstract Session.executeQuery(AbstractSession.java:1521)
                          at org.eclipse.persistence.internal.sessions.Abstract Session.executeQuery(AbstractSession.java:1503)
                          at org.eclipse.persistence.internal.sessions.Abstract Session.executeQuery(AbstractSession.java:1463)
                          at org.eclipse.persistence.internal.sessions.CommitMa nager.commitChangedObjectsForClassWithChangeSet(Co mmitManager.java:265)
                          at org.eclipse.persistence.internal.sessions.CommitMa nager.commitAllObjectsWithChangeSet(CommitManager. java:128)
                          at org.eclipse.persistence.internal.sessions.Abstract Session.writeAllObjectsWithChangeSet(AbstractSessi on.java:3766)
                          at org.eclipse.persistence.internal.sessions.UnitOfWo rkImpl.commitToDatabase(UnitOfWorkImpl.java:1404)
                          at org.eclipse.persistence.internal.sessions.Repeatab leWriteUnitOfWork.commitToDatabase(RepeatableWrite UnitOfWork.java:616)
                          ... 38 more
                          @collector <collectorScheduler-2> [] ERROR #AbstractStep # Encountered an error saving batch meta data. This job is now in an unknown state and should not be restarted.
                          org.springframework.dao.OptimisticLockingFailureEx ception: Attempt to update step execution id=4 with wrong version (1), where current version is 2
                          at org.springframework.batch.core.repository.dao.Jdbc StepExecutionDao.updateStepExecution(JdbcStepExecu tionDao.java:185)
                          at org.springframework.batch.core.repository.support. SimpleJobRepository.update(SimpleJobRepository.jav a:171)
                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
                          at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:597)
                          at org.springframework.aop.support.AopUtils.invokeJoi npointUsingReflection(AopUtils.java:309)
                          at org.springframework.aop.framework.ReflectiveMethod Invocation.invokeJoinpoint(ReflectiveMethodInvocat ion.java:183)
                          at org.springframework.aop.framework.ReflectiveMethod Invocation.proceed(ReflectiveMethodInvocation.java :150)
                          at org.springframework.transaction.interceptor.Transa ctionInterceptor.invoke(TransactionInterceptor.jav a:110)
                          at org.springframework.aop.framework.ReflectiveMethod Invocation.proceed(ReflectiveMethodInvocation.java :172)
                          at org.springframework.aop.framework.JdkDynamicAopPro xy.invoke(JdkDynamicAopProxy.java:202)
                          at $Proxy36.update(Unknown Source)
                          at org.springframework.batch.core.step.AbstractStep.e xecute(AbstractStep.java:244)
                          at org.springframework.batch.core.job.SimpleStepHandl er.handleStep(SimpleStepHandler.java:135)
                          at org.springframework.batch.core.job.flow.JobFlowExe cutor.executeStep(JobFlowExecutor.java:61)
                          at org.springframework.batch.core.job.flow.support.st ate.StepState.handle(StepState.java:60)
                          at org.springframework.batch.core.job.flow.support.Si mpleFlow.resume(SimpleFlow.java:144)
                          at org.springframework.batch.core.job.flow.support.Si mpleFlow.start(SimpleFlow.java:124)
                          at org.springframework.batch.core.job.flow.FlowJob.do Execute(FlowJob.java:135)
                          at org.springframework.batch.core.job.AbstractJob.exe cute(AbstractJob.java:281)
                          at org.springframework.batch.core.launch.support.Simp leJobLauncher$1.run(SimpleJobLauncher.java:120)
                          at org.springframework.core.task.SyncTaskExecutor.exe cute(SyncTaskExecutor.java:48)
                          at org.springframework.batch.core.launch.support.Simp leJobLauncher.run(SimpleJobLauncher.java:114)
                          ..............
                          at org.springframework.scheduling.support.DelegatingE rrorHandlingRunnable.run(DelegatingErrorHandlingRu nnable.java:51)
                          at java.util.concurrent.Executors$RunnableAdapter.cal l(Executors.java:441)
                          at java.util.concurrent.FutureTask$Sync.innerRunAndRe set(FutureTask.java:317)
                          at java.util.concurrent.FutureTask.runAndReset(Future Task.java:150)
                          at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.access$101(ScheduledThreadPoolE xecutor.java:98)
                          at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.runPeriodic(ScheduledThreadPool Executor.java:181)
                          at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.run(ScheduledThreadPoolExecutor .java:205)
                          at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:886)
                          at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:908)
                          at java.lang.Thread.run(Thread.java:619)

                          Comment


                          • #14
                            I can't tell for sure, but it looks like the error in the Batch meta data was triggered by a business exception in eclipselink (optimistic lock failing for table.View). If you use JPA in your Batch components be sure to flush the changes manually in an ItemWriter (or similar), so that those errors don't show up during the transaction commit, which Batch relies on heavily to succeed as sign that all is well.

                            Comment


                            • #15
                              In my ItemWriter I'm delegating all calls to DAO components, so I'm not sure where I shall call the flush method.

                              Here is a pseudo code of my writer

                              public MyWriter implements ItemWriter<Item> {

                              MyDAO myDAO;

                              AnotherDao anotherDAO;

                              write(List<? extends Item> items) {

                              for(Item item : items) {
                              myDAO.persist(item)
                              }

                              anotherDao.doSomethingWithItems(items);
                              }
                              The OptimisticLockException usually happens inside of anotherDao.doSomethingWithItems(items).

                              So where flush should be called ?

                              Comment

                              Working...
                              X