Announcement Announcement Module
Collapse
No announcement yet.
MultiResourcePartitioner Partioning issue Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • MultiResourcePartitioner Partioning issue

    Hi,

    I have large feed file coming in which I need to partition and process.

    Initially we were having a strategy where we will split the large file using unix splitter command and the provide this to a spring batch job and then parallel execution. For this we were planning to use MultiResourcePartioner.

    When I dove a little deep it looks like the grid-size we mention in this partioner does not take any effect as it will partion only according to the number of resources provided.
    My problem is that I want the partions to happen as per the grid size and not by the resources.

    Can some one suggest me any other implementation for partioner provided by spring batch which will serve my requirement.

    Thanks in advance!

    Ninad

  • #2
    How do you plan on associating a resource to a partition if they are based on the grid size? Say the grid size is 4 and you have 10 files...how does the partitioner know which file goes to what partition? I think the example you're describing would need a custom partitioner...

    Comment


    • #3
      I have the same problem

      hi,
      I know this is an old post but I'd like to know if you find a solution to this problem?

      Comment


      • #4
        I've seen this in the MultiResourcePartitioner javadoc :
        HTML Code:
        Implementation of {@link Partitioner} that locates multiple resources and
         * associates their file names with execution context keys. Creates an
         * {@link ExecutionContext} per resource, and labels them as
         * <code>{partition0, partition1, ..., partitionN}</code>. The grid size is
         * ignored.

        Comment

        Working...
        X