Announcement Announcement Module
Collapse
No announcement yet.
Double processing trouble in ItemWriter (regression from 2.0.0 ?) Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Double processing trouble in ItemWriter (regression from 2.0.0 ?)

    Hi all,

    I've recently done a migration of SpringBatch on my project, from 2.0.0 to 2.0.4 (also tried the latest 2.1.0) and I have encountered a problem of double processing of items in my "ItemWriter".

    I've not done any modification in my code, so it's sure it comes from the new jars. I cannot go back on the 2.0.0 versions cause newer versions solve some of my other problems.

    Here is the description of my problem. I've executing a step that reads in a flat file. When an exception "MyException" happens, I want to skip it.
    Actually, it works, but the ItemWriter is called twice. And this is problem for me...
    Maybe the way the skip of exception works has changed ? Could you help me please ?
    Thanks a lot.


    Here is the configuration of the step.
    Code:
    		<step id="myStep">		
    			<tasklet>
    				<chunk 
    					reader="myReader" 
    					writer="myWriter" 					
    					commit-interval="1"
    					skip-limit="999999">
    					<skippable-exception-classes>
    						MyException
    					</skippable-exception-classes>
    				</chunk>
    			</tasklet>
    		</step>

  • #2
    If the exception is in the writer we have to assume that the transaction is dead and roll it back. Then we have no choice but to re-process the chunk to ty and identify the bad item (if there is one) so that it can be skipped. I have no idea what was different in 2.0 - it should have worked the same way, and if it didn't that probably causes other problems.

    If the exception is in a reader or a processor you have more flexibility (e.g. in 2.1 you can declare that the processor is not transactional) and the framework can identify the failed item without re-processing. There are also options with no-rollback declaration hints. So the best advice is to make sure that writers do only transactional work that you know will be rolled back on failure and/or make sure only unrecoverable errors happen there so the only outcome from an exception is a failure (not a skip). Put business processing in an ItemProcessor if you can.

    What is your writer doing and why is it not rolled back after a failed transaction?

    Comment


    • #3
      Thanks for your reply

      Here is the process of my writer:
      Code:
      @Transactional(readOnly = false, propagation = Propagation.NOT_SUPPORTED)
      public void write(List<?extends MyObject> pList){
      
         // Call few webservices (in a mode so that SpringBatch doesn't manage transaction in this case)
      }
      
      @Transactional(readOnly = false, propagation = Propagation.NOT_SUPPORTED)
      @OnWriteError
          public void onWriteError(Exception pException, List<? extends DonneesContrat> pList) {
         // Call a service in charge of logging excepotion in DB
      }
      Here is what it happens:

      ItemReader (in a flat file):
      workfine (tokenizer and mapper ok)

      ItemWriter:
      1) Execute the write() method: Exception is happening (but no rollback, transaction managed by EJB, it's ok)
      2) Execute the onWriteError(): call the logger service (ok too)
      3) Execute one more time the write() method (but not the BeforeWrite or OnWriteError). So my Webservice process is done again.

      I want to avoid the point (3), is there a way for that ?

      Thanks for helping me and I apologize if my english is not good enought.

      Comment


      • #4
        Originally posted by Zico View Post
        I want to avoid the point (3), is there a way for that ?
        I assume you put the web service call in an ItemWriter so you can batch the items together into one call? If not just put them in an ItemProcessor.

        If so, then is the exception really skippable? If it is then your best option is to catch it in your writer and deal with the recovery in a catch clause.

        The only way we could deal with a skip on write without re-processing is to skip *all* items in the chunk. If that would be useful to you raise a JIRA and we can make that option available at least via the beans namespace. A workaround in the meantime would be to extend FaultTolerantStepfactoryBean and override the configureChunkProcessor() method to return your own implementation that skips all items on write failure.

        Comment


        • #5
          Originally posted by Dave Syer View Post
          I assume you put the web service call in an ItemWriter so you can batch the items together into one call? If not just put them in an ItemProcessor.
          It's almost that. Actually I process my items one by one.
          The process consists in a successive calls to web services.
          This is done because the item is quite complex. It is composed in 3 distincts "parts". It means that I insert main information on my item in a first call (web service 1). Then other calls (webservice 2 and 3) are done in order to register other inforamtion about my item.
          I wrote it like that (with this transaction mode) because I don't want to cancel the task done in WS1, even if WS2 or WS3 fails.

          I've done this in the ItemWriter because my services are writing in DB. That's not the aim of ItemProcessor which is rather in charge of transforming objects, am I right ?

          As you said, I could try to encapsulate whole process in a "try/catch" block.
          But I would have prefer to let go this specific exception until SpringBatch, SpringBatch would have "ignored" the exception (using 'skippable' feature) and then call the OnWriteError() method in order to log my exception.

          If I want to skip this exception, it is because, I'm processing a flat file (containing over 10000lines) and I don't want that an error on a line stops the job. I want it to continue, ignoring the line in error and not trying again to process it.


          You spoke about the "FaultTolerantStepfactoryBean" class.
          How does it exactly work ? I did not manage to find any information/example of it. Is it an interface that I have to implement ? If yes, for which element ?

          Thanks.

          Comment


          • #6
            If the processing is per item, just put it in an ItemProcessor and set item-processor-transactional="false" in your chunk configuration (Spring Batch 2.1). That's much easier than trying to implement a custom step factory (it's not an interface, it's extendable, but I wouldn't recommend that unless there is no alternative).

            Comment


            • #7
              Ok, I'm going to try that then.
              But, last question (maybe a bit silly), if all my process is done in itemProcessor, what will be the role of the itemWriter ?
              Will it have to be empty ? Must I declare one ?

              Comment


              • #8
                Good point. Yes an ItemWriter is mandatory, so yours will be a no-op. Maybe it shouldn't be mandatory (I'm neutral - raise a JIRA and see if anyone agrees).

                Comment

                Working...
                X