Announcement Announcement Module
Collapse
No announcement yet.
How to make transactional import with spring batch Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • How to make transactional import with spring batch

    Hello,
    I am trying to implement the import module of our project using spring batch with the samples provided. I have 5 text files carrying the data to be written to the 5 tables of the database. If any error occurs during one of the import processes of the 5 files i have to rollback the data in all the tables.

    For this purpose, i defined a job consisting of 5 simple steps(single step for each file import) I managed to make all the insertions transactional within a step by setting "commitInterval" property of simple step to a large value. However i need to make it transactional within the whole job. Is there a way of doing that in spring batch?

    I tried to define a job consisting of a single step which has a CompositeItemProcessor in it.However, it didnt work either.

    Any help or idea will be appreciated.
    Thanks in advance,
    Firis

  • #2
    Just to make sure I understand your issue correctly, you have 5 files that need to be loaded, and if say, line 100 fails in the third file, then all the data loaded for the previous 2 files should be removed from the database as well.

    I'm assuming you can't just skip the record and process it later? If not, is there any reason why, before the job ended, the data loaded by the steps couldn't be deleted from their respective tables? If that was the case, you could easily add an onError interceptor to the stepExecutor that would just blow out the data if there was an error. This could potentially be done by using temp tables and truncing them if there was an exception.

    If this isn't possible, you could also pass a dummy transaction Manager to the StepExecutor, then wrap the JobExecutor.run() call in a transaction using AOP. You would get a start transaction when the job was first run, and it wouldn't commit until after run() returned, rolling back on any exceptions. My only concern with this approach would if the database could handle that much data in one transaction (5 files could be a lot of data).

    Comment


    • #3
      Thank you for your comments. My issue is as you mentioned.

      I have written a class implementing the RepeatInterceptor interface in order to rollback the data when there is an error in one of the input files. Rollback operation is done in the onError method before the job completes. My implementation works fine when there is an error in the inputs.

      In the steps of the job, i am inserting the data to temp tables as you recommended. But after running 5 steps successfully, i need to insert the data from the temp tables to the original ones. I thought this could be done in some kind of onJobComplete interceptor. However, I could not find setInterceptor etc method in JobExecuter or JobFacade interfaces. Do you have any suggestions to accomplish this?Can CompletionPolicy interface and its implementations be related with this issue?
      Thanks in advance

      Comment


      • #4
        Unfortunately, we don't have any interceptors on the JobExecutor (since a RepeatTemplate isn't used). We've only run into this issue with one other project, and AOP was used to wrap the JobExecutor. If you're unfamiliar with how to use AOP in spring, please check the spring reference documentation. (we have an example of wrapping a few things with AOP in our samples jobs as well)

        If it looks like there is a large demand for Interceptors on the job executor, they can be added, unless it is a one-off that can be handled when it comes up using AoP.

        Comment

        Working...
        X