Announcement Announcement Module
Collapse
No announcement yet.
Partitioning based on even break up of record set Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Partitioning based on even break up of record set

    Hi Spring Batchers,

    I had a look at the code base and I can not see any functionality yet for doing partitioning based on even break up of a record set.

    When is this functionality planned for? Will it be part of the 21 March RC?

    Thanks!

  • #2
    It definitely won't be included in 1.0, but partitioning should be one of the main topics addressed in future releases (no scheduled date for that though). Meanwhile the sample parallelJob gives an example how to do parallel processing.
    Last edited by robert.kasanicky; Mar 7th, 2008, 08:27 AM.

    Comment


    • #3
      Parrallel processing is a bit different. I think that the basics needed for partitioning are there, but require a little bit of extra wrapping to pull it off. If I was going to implement it with the latest code, I would simply create a wrapper for the JobLauncher, and partition the data there, creating entries in the JobParameters for each the beginning and end of the partitioning range, kicking off a separate job with it's own unique range as the partition. Then, you would only need an ItemReader that pulls those ranges from the job parameters. You could even skip the wrapper and put it in your scheduler, which is also common. Something like kicking off three jobs, one for north america, one for europe and one for asia. You wold then simply need a sql statement with something like: select * from DATA_TO_PROCESS where region =${region} You should be able to kick these three jobs off at the same time without issues. There is a little work that needs to be done to the various jdbc item readers to ensure they can pull values from JobParameters and put them into preparedstatements (as it has been mentioned in the forums) but it still works now.

      I suppose what I'm saying is that it's still very possible to partition your jobs up now, but the framework isn't doing it for you. Post release 1.0 we need to take a long look at this and decide if we want to try and do this within the framework (again, probably by wrapping the launcher) and if so how we might accomplish that. We'll start immediately looking at these types of issues once release 1 comes out and we've had time to sort through any pains that are inevitable for the first release of a framework.

      Comment


      • #4
        Thanks for the reply - it is giving me some direction.

        Comment

        Working...
        X