Announcement Announcement Module
No announcement yet.
Spring Batch: Too much overhead for very short lived in memory Jobs? Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Spring Batch: Too much overhead for very short lived in memory Jobs?

    I'm working on implementing a document processing pipeline. We have currently have a proof of concept implementation. But it lacks flexibility. There is a requirement to be able to easily wire up several different versions of the pipeline in terms of what steps are preformed (and their order). We get a steady stream of documents. ~200 a minute. They will come from different sources such as FTP, filesystem, JMS, etc. I'm in the process of migrating our POC to Spring integration to take advantage of its flexibility of handling diverse incoming channels.

    The processing is primarily in memory. At least a Job is in memory from start to finish. There are a couple of service calls, but they are quick (100 -200ms) and calls to them simply block waiting for the response. Overall processing time for a document is ~2-5 seconds depending on the steps and the document size. We run a couple dozen threads simultaneously.

    While Spring Integration is helping with certain aspects, it seems like it might be a bit more cumbersome to wire up different sequence implementations. The Spring Batch config <job> element with its <step> elements seems more straight forward and easier for people to groc what the pipeline flow is. Being able to use Spring Batch Admin for monitoring and its retry ability has some appeal as well. But before I go digging into learning Spring Batch and using it, I'm curious if Spring Batch would add too much overhead to such a short lived job. (FYI we would likely use a file based H2 Database for SB.) Any thoughts on the potential overhead cost?

    Also, since I've only done some preliminary reading on Spring Batch, are there any concerns with multiple threads running the same job definition? I would assume not, but it never hurts to be sure.