Announcement Announcement Module
Collapse
No announcement yet.
Spring Batch : MultiResourcePartitioner: Job ending with smallest file write complete Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Spring Batch : MultiResourcePartitioner: Job ending with smallest file write complete

    Hi,
    We have a batch job configured to read data from csv and load it into oracle db. We have configured a partition step which reads from multiple files and writes it to database parallel y, the issue is that the job is ending with a success status as soon as the smallest file with fewer records are done transferring to oracle.

    Ex: Say we have 3 files and the commit point is 500
    file1: 1000 records
    file2: 1000 records
    file3: 500 records
    In the very first iteration as soon as the file3 is transferred, the entire job successfully completes successfully. With the other 2 files still having 500 more records to transfer.
    Following is the configuration I am using, please let me know what as to what is wrong in the configuration.
    It will of great help if you can help me with this.
    ALSO I WOULD LIKE TO KNOW THE LINK BETWEEN COMMIT-POINT, GRID-SIZE AND NUMBER OF FILES.

    <job job-repository="jobRepository" id="SampleJob"
    xmlns="http://www.springframework.org/schema/batch" restartable="true">

    <step id="UpdatePartition">
    <partition step="UpdateStep" partitioner="Partitioner" >
    <handler grid-size="${chunk.size}" task-executor="taskExecutor"/>
    </partition>
    </step>

    </job>
    <step id="UpdateStep" xmlns="http://www.springframework.org/schema/batch">
    <tasklet transaction-manager="transactionManager" throttle-limit="${spring.batch.throttlelimit}">
    <chunk reader="CSVFileReaderUpdate" processor="Transformer"
    writer="JDBCItemWriterUpdate" commit-interval="${spring.batch.commit.interval}">
    </chunk>
    </tasklet>
    </step>


    <bean id="Partitioner"
    class="org.springframework.batch.core.partition.su pport.MultiResourcePartitioner">
    <property name="resources" value="file://${filepath}/${batchFile}" />
    </bean>

    <bean id="CSVFileReaderUpdate" class="org.springframework.batch.item.file.MultiRe sourceItemReader">
    <property name="resources"
    value="file://${filepath}/${batchFile}" />
    <property name="delegate">
    <bean class="org.springframework.batch.item.file.FlatFil eItemReader">
    <property name="lineMapper">
    <bean class="org.springframework.batch.item.file.mapping .DefaultLineMapper">
    <!-- Rest of the config removed for brevity -->
    </bean>

    <bean id="JDBCItemWriterUpdate"
    class="org.springframework.batch.item.database.Jdb cBatchItemWriter"
    p:assertUpdates="true" p:dataSource-ref="datasource">
    <property name="sql">
    <!-- Rest of the config removed for brevity -->
    </bean>


    Thanks

  • #2
    your partitioner misses a wildcard to select multiple files (it's hard to tell given the token you use). And you should use the partitioner to create one partition for file and then a regular reader to read one file per partition.

    Have a look to:
    http://static.springsource.org/sprin...cutionSplitter

    Comment


    • #3
      @snicoll: Oh! Of course, sorry about that.
      We are using 'fileName*' with splits with names 'fileName1', 'fileName2', etc..
      I have already checked the link that you have provided here, I have similar configuration, but kindly let me know if there is a flaw in this configuration.
      Also please let me know, how does the chunk-size, commit-point, no.of files and no.of records in the file affect the write count?

      Comment

      Working...
      X