Announcement Announcement Module
Collapse
No announcement yet.
Passing an object between Listener and Writer Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Passing an object between Listener and Writer

    Hi,

    Requirement-1
    To create a unique id (load_id) before execution of the step begins.
    Solution-1
    Have used CustomListener class (implementing StepExecutionListener), beforeStep method to create load_id. Then, inserted this load_id and and job_instance_id in the table A. Also, stored load_id in the ExecutionContext.

    Requirement-2
    To get this load_id in the step, possibly in the writer class. (without accessing the db again... probably from the running ExecutionContext)

    How do i get the load_id in the writer class? Can i access ExecutionContext for this step execution in the writer class?

    Thanks in advance!
    SpringForever

  • #2
    1. Have your writer implement ItemStream (or extend AbstractItemStreamItemWriter if you are not already extending another class) if it doesn't already do so.

    2. Add a private ExecutionContext member variable

    3. Implement/Override/Add To the "open(ExecutionContext)" method to set the member execution context to the passed execution context. (If you are not extending the Abstract implementation, you may also need to add a no-op implementation of update(ExecutionContext) and close(ExecutionContext).

    4. Add to / retrieve from the execution context whenever you want during the write method.

    //QED

    This is currently the best practice for making ItemReader/Writer collaborators "ExecutionContext aware"

    Hope this helps

    Comment


    • #3
      Thank you! It works.

      Comment


      • #4
        Another Question (Related to Writer and Listener):

        Writer_1 writes to the table A. What would be the best way to write to the table B, if there is a CommaDelimited error? (i think i would catch the error in "onErrorInStep" method in the listener and write to the table B)

        Do i have to create writer_2 to write into the table B?

        Thank you.

        Comment


        • #5
          No, you don't have to create a separate ItemWriter unless you really feel like it, since you will be manually invoking the logic yourself -- using the ItemWriter only buys you functionality when it's surrounded by framework code.

          What you'll most likely want to do is to create a transaction inside your onErrorInStep method using spring-tx (this will require you to inject your transaction manager into your listener) that performs your write.

          Take a look at how transactions are handled in ItemOrientedStep for more information.

          You might also want to wrap it in a RetryTemplate callback, to give it several chances to write to table B.

          The real concern is: what do you do when the write to table B fails?

          Comment


          • #6
            Thanks dkaminsky!

            Is there a way to implement the following scenario using Spring batch? (with respect to the table A and B above)

            File has 4 records:
            Rec-1 -> good
            Rec-2 -> CommaDelimited error
            Rec-3 -> good
            Rec-4 -> CommaDelimited error

            Output:
            table A -> empty
            table B -> Rec-2 and Rec-4 (all error records should be logged in table B)

            Flow explanation:
            -Job reads all the records.
            -If all records are valid (no CommaDelimited error) then file data goes to table A.
            -If one or more record is invalid then invalid records should be logged in table B, while table A will be left empty. (So, i think transaction will need to be applied to the entire file in one chunk)...In this case, transaction will be rolled back, job fails and job will be re-processed if a new file is provided.

            Questions:
            -Is there a way to implement this using Spring batch?
            -What if i create a step, which does these validations and if all records are valid then the job will go to the next step, which will insert file data in table A? Can TaskletStep as the first step solve this problem?

            Thank you for your help!
            SpringForever
            Last edited by springforever; Apr 7th, 2008, 01:26 PM.

            Comment


            • #7
              Basically, you want your task of writing the file to the database to be all or nothing... You could tackle this a number of ways depending on your preferences - here are a couple of ideas

              Option 1: "Simple" Approach - One Step with Listener
              • item reader reads from file (FlatFileItemReader)
              • no-op item writer (write an implementation with blank methods)
              • add a step listener with
                • afterStep method defined to write whole file to database
                • onErrorInStep method defined to write bad records to database
              Your listener would also have to count errors or keep a boolean flag as to whether or not any exceptions were encountered, since your exception handling method will not cause your job to halt. Your afterStep method will need to check to see if any errors were encountered, and only perform the write if none were.

              Option 2: Modified "Simple" Approach - Two Steps with Listener
              Exactly the same as option 1 except instead of defining your logic to write the whole file in an afterStep method you will create a 2nd step with the same item reader but an actual database-related (e.g. JDBC) item writer. The afterStep method would only contain the logic as to whether or not the job should terminate before moving on to the next step.

              Option 2 is my preferred option of the two because although it duplicates the read, it puts more of the potentially failed operations (i.e. database writes) in framework-controlled code, allowing for more easily configurable repeat behavior, error handling, etc.

              Option 3: Data Modeling Approach - One Step with Listener
              Instead of making this a purely Spring Batch concern, model your database to stage the data as it is read and then just have Spring Batch "commit" it at the end of the step (or job). Most databases support the creation of session-scoped temporary tables, so you can leverage this to achieve your goal.
              • run a query to create a temporary table in your beforeStep
              • use the item reader (FlatFileItemReader) to read the file
              • write error records in your onErrorInStep method in the listener
              • use a database item writer (e.g. JDBC) to write good records to the temporary table
              • run a query to copy the data from the temporary table to the real table in your afterStep (and clean up the temp table if your database doesn't do it for you) if no errors were encountered (i.e. onErrorInStep was never called)

              All said and done, I'd say 3 is the way to go if you can swing it, 2 otherwise.

              Comment


              • #8
                Option 2 works best for me.

                However, onErrorInStep (1st step of 2 steps) throws an exception at Record-2 and terminates the step. How do i continue processing to the next records?

                Comment


                • #9
                  You could set the skip limit really high?

                  What I would do is set up a pre-processing step that is less strict with the validation, and weed out the bad records there. Write bad records to your second database table, and good ones to another input file for the second step, so it only has good records to crunch.

                  Comment


                  • #10
                    I did try with the skip limit. In that case, onErrorInStep never gets called. Do i need to add SkipListner?
                    When i say CommaDelimited error it's that the file is not well formed.

                    How to use pre-processing? Is there any document where i can find "how to implement different things of Spring Batch"?
                    Last edited by springforever; Apr 8th, 2008, 08:28 AM.

                    Comment


                    • #11
                      Setting a skip limit would mean that the errors were ignored, unless you add a SkipListener, so that might be the way to go for you.

                      There are quite a few examples of using the various callback interfaces in the user guide. We could add some more, I am sure. Maybe this is a good one (if I understood what it was you were doing a bit better I would know)?

                      Comment


                      • #12
                        Thanks Dave!

                        Please look at #6 above, for the requirement. If records (Rec-2 and Rec-4) are not well-formed then they should be logged to table B and rest of the good records (Rec-1 and Rec-3) should not be inserted in the table A. Basically, these bad records will be shown to the users. User will correct these records in the file and then reload a file to be processed by the Spring Batch.

                        load_id (#1 above), needs to be inserted in table B along with these bad records. If i use SkipListener then not sure how would i get these bad records. (As onSkipInRead has no parameter for getting the running record)

                        If all records are good then they will be inserted in the table A along with load_id. (That's why needed ExecutionContext in the Writer)
                        Last edited by springforever; Apr 8th, 2008, 09:29 AM.

                        Comment


                        • #13
                          Sorry, I should have realized that you'd need a SkipListener to manage that as well. There's currently an open issue about getting the records to the skip listener on a read error - http://jira.springframework.org/browse/BATCH-540. What you could do, (this is kind of kludgy, but it should work,) is to have your item reader keep track of the last item read and add a method to your item reader that does the write to the database -- then, your onSkipInRead method could just call this method and let the item reader handle being aware of the item.

                          Comment


                          • #14
                            That helps!

                            One more question.... how do i call Reader (ABCMapper implements FieldSetMapper) from onSkipInRead...? (It has to be the running Mapper, i guess)... Do i need to create a dependency injection object ABCMapper within my Listener?

                            Comment


                            • #15
                              I'm confused as to what you are trying to do - is this in relation to the same issue that I just responded to? (if it is, you lost me)

                              Comment

                              Working...
                              X