Announcement Announcement Module
No announcement yet.
Continuous Processing Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Continuous Processing

    At its core, Spring Batch is focused on Job processing in general and the specifics of batch jobs in particular. Jobs are parametrized and have a formal beginning and end.

    Are there suggested patterns or examples of using the rest of the Spring Batch domain model (Steps, InputReaders, Processors, OutputWriters) to process data continuously?

    My initial thoughts are to simply use a no-op JobFactory like the ones typically used for unit testing.

    Obviously I do give up Job based functionality: metrics, contexts, restartability, etc, and that's ok.

    Are there future plans to bring job like metrics to a more SOA use case where metrics over a sliding time window are more relevant than the notion of the beginning and end of a parametrized job?

    I suppose I could have a custom job executor manage starting and stopping this continuous job at given time intervals, a daily instance of a continuous job if you will.

    I think that's certainly possible, but feels hackish. Any thoughts on leveraging the bulk of the Spring Batch processing model for continuous work?

  • #2
    All of Spring batch Infrastructure is independent of the "batch domain" (jobs, steps, status etc.) so it is ideally suited to a class of continuous processes (and indeed is used in that form in many places). Combining that with Spring Integration for triggering and data routing makes quite a flexible framework with many nice features.

    Sliding windows are an interesting idea. I have no idea how it would be implemented, but if you'd like to create a JIRA, that would be a great way to find out if anything occurs to anyone later.


    • #3
      Thanks Dave. I see in the documentation that there are examples of the infinite loop scenario. However in looking over the 2.1.0RC1 codebase I'm only able to find an unreferenced infiniteLoopJob.xml definition. The definition has a commented out processing tasklet at a location that no longer exists. I wonder if some refactoring happened and this sample didn't fully make it.

      I'm in the process now of getting this job to execute. Simply getting this to run continuously will allow me to answer questions about how the JobRepository deals with these kinds of unending metrics and what kind of performance over time I see.

      The real point of the documented infinite loop sample was to demonstrate eventing/messaging to control the flow of an infinite task. This is outside my area of Spring Batch expertise atm. Could you point me to a test case that demonstrates this functionality, or to a JIRA that will be updated once a test case is written?


      • #4
        I don't think the infiniteLoopSample.xml has much overlap with your use case. It might be an interesting starting point, as would the loopFlowSample.xml for a different kind of loop (controlled by the exit status of a step). Both are used and tested in the Spring Batch Samples (where they live), so I'm not sure what you meant about them being "unreferenced". Maybe you are looking at a different project?

        As far as events/messages controlling the flow goes: it's an interesting idea, and easy to implement by injecting a reference to a @MessageEndpoint (assuming you are using Spring Integration) into a component of your step (tasklet, listener, whatever...). There's no sample that does this right now though. If you explain your use case a bit I might add one to Spring Batch Integration (which lives with Spring Batch Admin).


        • #5
          SO If I have something like

          <job id="myJob" job-repository="jobRepository" xmlns="">
          <step id="myFirstStep" next="continue" xmlns="">
          <listener ref="skipListener" />
          <listener ref="itemListener" />
          <chunk reader="itemReader" processor="itemProcessor" writer="itemWriter" commit-interval="${batch.chunk.commit-interval}" skip-limit="${batch.chunk.skip-limit}">
          <include class="SomeException"/>
          <include class="SomeException"/>
          <decision id="continue" decider="executionDecider">
          <next on="LOOP" to="myJob" />
          <end on="STOPPED" />

          Then as long as the decider returns LOOP, the job would keep running. Correct?

          Is this the right way to achieve a kind of polling job which would keep running for ever? Can I use this for a scenario when I need to process say credit card swipes of customers and send them sms alerts?

          Specifically, will the step executions be stored in memory and as the job is running for a long time will the Job Execution occupy more and more memory?