Announcement Announcement Module
Collapse
No announcement yet.
Newbie to SpringBatch - Design Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Newbie to SpringBatch - Design

    We are in the process of designing a batch application for a large investment bank, being a newbie to spring batch I would like consult batch gurus.

    Apologies for the long post...

    Here's the requirement -

    1. We get files from upstream during business hours in moderate volume (intra-day activities) and high volume** during nightly loads.
    2. A set of files that are uniquely identified constitute an ENTITY(~8 flat files) and there could be several ENTITIES at any point of time (ENTITIES are pushed as and when they are available from the upstream system)
    3. For an ENTITY to be successfully loaded all the process in the ENTITY should complete successfully (assuming one process/file, we have 8 processes).
    4. Copy of one of the files(reference data) needs to be pushed to another system and it needs to be processed completed.
    5. After the step4 is successful the same file is run against our db.
    6. Once step 5 is complete rest of processes(7 files) can run in parallel to complete the jobs.
    7. Individual failed records from process should be captured and notified which will result in a refeed of the ENTITY from upstream.

    **Volume/SLA
    ------
    During nightly feeds there will be 200 ENTITIES(100k records/entity ~2m records in total) that needs to be in 2 hours.
    Numbers could grow in future,,

    Comments
    -----------
    The application should launched one time and run continiously processing entities(set of files) as they become available in a directory - so there is no scheduling.
    We can't use any sort of messaging at this point for many internal reasons.
    We would like to use spring core, hibernate, hbm tx mgr with AOP and spring batch??

    Your comments/suggestions are higly appreciated !!

    Thank You,
    Nitty

  • #2
    Sounds like a plan to me. I would add Spring Integration to your shopping list to handle the directory polling and triggering. Spring Batch Integration (now part of Spring Batch Admin) has some features that play in that space.

    Comment

    Working...
    X