Announcement Announcement Module
Collapse
No announcement yet.
limitations of ExecutionContext Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • limitations of ExecutionContext

    Hi guys,

    I am busy with a POC to see if Spring Batch is a suitable option. We are implementing a Siebel ebilling system that requires two input files, one for masterData and one for clientData. To get these files in the required format there is an input data file that has to be pre-processed.
    This pre-processing function firstly reads in a XML file that contains transaction numbers (these transaction numbers are mapped to the Siebel ebilling system and the actual transaction categories are specified by the client). Currently our pre-processor reads in the xml once for each batch run and then reads in an input file (csv) that contains the actual transaction data.
    Each line in the input file is mapped to a specific transaction number and then the transaction data and the transaction number is written to an output file.
    My question is: can the data in the xml file be saved on the ExecutionContext and be used in subsequent steps? There is a good possibility that the xml data could be larger than 2500 characters and I have read in some forums that large objects should not be written to the ExecutionContext.
    Could someone please clarify this for me.

    Regards
    Robin

  • #2
    Originally posted by robinmurphy View Post
    There is a good possibility that the xml data could be larger than 2500 characters and I have read in some forums that large objects should not be written to the ExecutionContext.
    That limit doesn't apply to large objects. In fact it doesn't apply to anything in the ExecutionContext (at least with Spring Batch 2.x). Where did you read that?

    You can store large objects in ExecutionContext, but be aware that they will be stored in the JobRepository, so it might not be particularly efficient. It might be better to store the data as individual records in your own business schema if you need them to be persistent (e.g. if you need the steps to be restartable). If not then you can just transfer them between steps in memory (don't use ExecutionContext).

    Comment


    • #3
      Thanks for the reply Dave,

      this was in response to a question on stackoverflow.com:

      "From a step, you can put data into the StepExecutionContext. Then, with a listener, you can promote data rom StepExecutionContext to JobExecutionContext.

      This JobExecutionContext is available in all the following steps.

      Becareful : data must be short. These contexts are saved in the JobRepository by serialization and the length is limited (2500 chars if I remember well).

      So these contexts are good to share strings or simple values, but not for sharing collections or huge amounts of data.

      SHaring huge amounts of data is not the philosophy of Spring Batch."

      Comment


      • #4
        Well, that shows how useful stackoverflow can be, I suppose. He's right about the warning to be careful, just not about the limits (which are only set by the database schema anyway, and you can always chage that).

        Comment


        • #5
          Saving data across steps in memory

          Originally posted by Dave Syer View Post
          If not then you can just transfer them between steps in memory (don't use ExecutionContext).
          Hi Dave,
          Can you pls elaborate on how to transfer the data across steps "in memory" without using the executionContext. Code snippet would help. Should we put it in another singleton bean ?

          My use case is to consolidate data from 2 tables (each of them huge) and write an output file in the end, with several steps in b/n. Would it mean these 2 tables have to be fully loaded into memory at once?

          Thanks,
          jaa

          Comment


          • #6
            If you have large amounts of data, there's no point trying to cache it all in memory is there? You would need some form of persistent storage. If you do it in memory, then the pattern is the same - a custom repository, which could well be a singleton.

            Comment

            Working...
            X