Announcement Announcement Module
Collapse
No announcement yet.
Threadsafe ItemReader Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Threadsafe ItemReader

    Hi,

    I have configured a job to download a bunch of files. There is a job listener configured for this download job, which is launched after this job is completed.

    In the afterjob method of this listener, I launch a number of upload jobs (same type of job) in a loop for each one of the file that is downloaded to load the data into the database. This listener has a dependency on the SimpleJobLauncher with AsyncTaskExecutor.

    The jobs are getting launched and executed in parallel and the data is getting loaded into the database. But it seems that occasionally after repeated runs, there are synchronization issues. The launched jobs ItemReader and ItemWriter properties holding the results are getting corrupted.

    1. Any better way to handle the above case. Presently this issue has been mitigated by using the result values in a ThreadLocal in the ItemReader. Since this is intermittent issue, not sure whether this would resolve the issue.

    2. I know ItemReaders are not thread-safe. How is it possible to make these thread safe?

    thnks and rgds,
    Basav

  • #2
    Originally posted by bnagur View Post
    2. I know ItemReaders are not thread-safe. How is it possible to make these thread safe?
    Firstly, use step scope - this provides each step with its own instance of the reader. If the step is single threaded (which it seems like it is in your case) that should be enough. Otherwise you might want to additionally write a wrapper that synchronizes its read() method. The file readers are empirically thread safe (presumabky because of synchronization in the IO libraries), but weren't designed to be, so the "empirically" is just a random observation and might turn out to be untrue.

    Comment


    • #3
      The readers are thread unsafe in step context. As Dave said you can synchronize read() method but from my experience it's not hi performance solution. Better way is to find method how to partition input data (e.g. ranges of objects ids or ranges of rows) and use a splitter to serve them. I hope I'm right
      Cheers

      Comment


      • #4
        Thanks Jul. Actually I have resolved this by launching each job in their separate applicationContext. This ensures there is no conflict between beans and the data is loaded correctly.

        thnks and rgds,
        Basav

        Comment

        Working...
        X