Announcement Announcement Module
Collapse
No announcement yet.
ItemProcessor called multiple times Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • ItemProcessor called multiple times

    Here is what I'm trying to do for handling invalid/erroneous records: I would like to skip an item during processing, and log it to database for further investigation. For this, in the Processor, currently I throw an exception registered as skippable in the batch config file, expecting that SkipListener's onSkipInProcess will be called just before commit, and I would log the item inside that method.

    Code:
    <batch:job id="loadMyTables" job-repository="jobRepository"
    		restartable="true">
    		<batch:step id="loadTableX">
    			<batch:tasklet transaction-manager="MyTransactionManager">
    				<batch:chunk  reader="XReader" processor="XProcessor"
    					writer="XWriter" commit-interval="250" skip-limit="250" >
    					<batch:skippable-exception-classes>java.lang.Exception</batch:skippable-exception-classes>
    				</batch:chunk>
    				<batch:transaction-attributes
    					isolation="READ_COMMITTED" />
    				<batch:listeners>
    					<batch:listener ref="MyTablesSkipListener" />
    				</batch:listeners>
    			</batch:tasklet>
    		</batch:step>
    </batch:job>
    But I observe the following behavior:
    The processor is called multiple times when an exception is throw from it, thereby registering it as multiple 'Process skips', and hence giving me an incorrect 'process skip' count (way more than expected). I fail to understand why it is calling the processor multiple times. Could you explain why it would be so? Also, the filter count is negative. I'm not returning null anywhere within the processor, so theoretically there shouldn't be any filter count. Any clue on this?

    More precisely, I would like to understand the 'basic' sequential flow of execution of a step. eg. where does the control go when a null is returned from the processor vs when an exception is thrown? Of course, http://static.springsource.org/sprin...nsactions.html did help a lot, but its not very clear though. Is there some kind of sequence diagram or flow chart depicting the execution paths?

    Waiting for a response!

    Thanks.

  • #2
    I investigated the problem further and found that the processor is calling multiple times when an exception is thrown because internally FaultTolerantChunkProcessor is being invoked, which is actually meant to deal with skipping and retry of items. But i would still like to know two things:

    1. What is the default chunk processor?
    2. Is it possible to configure a chunk processor of choice in the job configuration?

    Awaiting a response. Thanks!

    Comment


    • #3
      Hi,

      I've the same problem, did you found a solution for the problem that the processor is called twice when an (skippable) exception is thrown?

      Thanks
      Last edited by Reijnemans; Jul 28th, 2011, 06:30 AM.

      Comment


      • #4
        i could not recreate your problem with a real simple example, it worked as expected

        do you have a transactional reader ?

        Comment


        • #5
          Originally posted by michael.lange View Post
          i could not recreate your problem with a real simple example, it worked as expected

          do you have a transactional reader ?
          Hi Mike,

          Thanks for your response. Yes, my reader is transactional. Which version of Spring batch did you use? I used version: 2.0.4.RELEASE as this is the latest version in maintenance.

          Hi Reijnemans,

          No, I am still waiting for an answer.

          Comment


          • #6
            i tried it(*) with both 2.1.8.RELEASE and a downgraded version with 2.0.4.RELEASE, both times the skip happened only once

            Can you show the source of both reader and processor ?

            (*) both times non-transactional reader, but you did not turn off the buffering so i guess that's comparable

            Comment


            • #7
              HI thanks for the responses,

              Unfortunely I''m not able to post my code till after the weekend, but I also think this is not realy necessary because my reader is a staxEventReader. And my processor is only increasing a property in the executionContext, (some kind of step count, (for an object within an object)). My writer is a POJO which trough a checked exception (defined by my self). It also seems that the writer is not executed twice (At least I didn't see the error twice in the logging)

              My step is configured as default:
              Code:
              <step id="step1">
                  <tasklet>
                      <chunk reader="itemReader" processor="itemProcessor" writer="itemWriter" commit-interval="2"/>
                  </tasklet>
              </step>
              The spring batch version is 2.1.7.RELEASE and the version of Spring is 3.0.5.RELEASE

              Thanks

              Comment


              • #8
                Originally posted by Reijnemans View Post
                My writer is a POJO which trough a checked exception (defined by my self). It also seems that the writer is not executed twice (At least I didn't see the error twice in the logging)
                your problem looks different

                if your writer throws an exception, already processed items will be processed again to find the bad one, see this log output

                Code:
                --> 5 items read, processing starts
                <processor called:15>
                <processor called:16>
                <processor called:17> will throw an error on write
                <processor called:18>
                <processor called:19>
                <before write:[15, 16, 17, 18, 19]>
                <on write error>
                --> error on item 17, but it was in the list, lets find it
                --> rollback
                <before chunk>
                --> spring batch goes through all items of the chunk again to find the bad item
                --> basically it runs now with commit-rate="1" (only for this chunk)
                <processor called:15>
                <after write:[15]>
                <after chunk>
                <before chunk>
                <processor called:16>
                <after write:[16]>
                <after chunk>
                <before chunk>
                <processor called:17> called again for the bad item, because it's still unknown to spring batch, that this is the bad one
                --> no write, because itemWriter.write() was called with the bad item only and did throw an exception (again)
                --> but now spring batch knows the bad item
                <before chunk>

                Comment


                • #9
                  Hi micheale,

                  Thanks for the explanation, it's clear for me.

                  @shiralkarprashant, sorry for messing up your post with my question

                  Comment

                  Working...
                  X