Announcement Announcement Module
Collapse
No announcement yet.
Skipping advice on web service write failure Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Skipping advice on web service write failure

    I have a job that reads from a folder of .xml files (each file is a reader item), does some transformation, and submits them to a vender webservice. Then an archive step moves the .xml files to an archive folder.

    On occasion there is missing data and I get a SOAP exception. Right now the job fails on any error. This leaves all the .xml files in the processing folder (the successful ones, the failed one, and the unprocessed ones).
    I have to manually archive the successful, remove the failed one for research, and then run the job again to process the rest of the files.
    Obviously I'd like this to be automated by the job.

    I have tried to use the skip functionality, however since I am writing to a webservice, rollback is not possible in a failed transaction. This results in multiple submits of the same data to the vender as the chunk is (re)processing to find the failing item. Even with a commit limit of 1, the failed record is submitted twice.

    I have thought of two possible solutions.
    One, keep using skips, but keep up with each filename written to the webservice and check against this list in the processor/writer to prevent multiple webservice writes. Move the failed file out to a research folder with a SkipListener.onSkipInWrite().
    Two, don't use the skip feature and just trap the exceptions in the writer and move the failed file to a research folder and continue on processing.

    Anyone have any thoughts on how to best accomplish this?

  • #2
    I played with this for a while and ended up creating a NoOp Writer and moved my web service call to the Processor. I configured the step with processor-transactional="false" so a skip does not cause a rollback (and the corresponding re-processing of items that happens in the chunk-oriented Writer). My SkipExecution count properties look right and skip-limits work as expected. A skip listener moves failed .xml files to a research folder, so there is no manual clean up now.

    After a lot of testing I am satisfied this will meet my needs, however it still seems a bit odd to avoid using a Writer in my Step.

    Comment

    Working...
    X