Announcement Announcement Module
No announcement yet.
Pass data between steps Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Pass data between steps

    I have a job that reads from a database and writes to a flat file. In one step I read the database data, process the data and write registers to the file. While processing I need to collect some totals, group information and cache some data to avoid re-read that from database, that I'll use in further steps to append other kind of registers in the file. So I need to pass some complex objets between steps.

    Another requirement is that my job has to allow several threads executing with different parameters. So my problem is where can I store this interstep data? If I use a holder, if singleton I'll have to take care of concurrency and I'll have to clean data after use it to avoid a out-of-memory, with prototype or step scope I can share data. ExecutionContext is place for simple and small data and parameters becase is serialized to the database I think.

    Anyone knows a pattern/idea to solve this problem?

    Thank you for your help.

  • #2
    Is the put(String, Object) method in ExecutionContext not good enough for your use case?
    The only requirement to use this method is that the object must be serializable.


    • #3
      What I did is put my data into a threadLocal, but this is only possible with a single threaded job


      • #4
        Thanks for your replies. Yes, I think ThreadLocal with prototype scoped beans its the solucion to my problem as I have multiple threads executing multiple jobs, but each job is single-threaded. I think It could be useful some kind of interstep, not-persisted context to share this kind of cache structures in batch-processing.


        • #5
          Agree. Interstep non-persisted context would be useful. We are redesigning a job due to hitting size limits on temporary context data which we don't need to be persisted. Getting MySQL errors on exceeding size on the batch context table(s).