Announcement Announcement Module
Collapse
No announcement yet.
commitInterval not working as per spec. Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • commitInterval not working as per spec.

    I am working on a project where I have to handle 100,000 records in couple of hours.

    I am able to process first 6000 records in 21 mins(285 around record per min). But over the period of time, performance is getting degraded to 100 records per min. My initial thought was might be heap issue. Later I found my heap and GC graph looks clean. I have given 1GB max heap size but application is using max of 150 MB and GC bring it back to 65 MB. My lower level not increasing after several hours of job run.

    Next possible candidate is database. Spring is configured with commitInterval as 2. As per spec I assume , transaction would be commited for every two record it process. I thought this may put more stress on database and tried to increase the commitInterval to 250. I am not seeing any difference in database commit.

    Time taken for 6000 record is exactly 21 minutes when the commitInterval is 2 and 250 as well. So I don't know how this parameter really works.

    Anybody has an idea on how to make it work please do share.
    I am using batch 1.0.0-FINAl as developement started more than a year ago.

    <bean id="simpleStep" class="org.springframework.batch.core.step.item.Si mpleStepFactoryBean" abstract="true">
    <property name="transactionManager" ref="transactionManager" />
    <property name="jobRepository" ref="jobRepository" />
    <property name="startLimit" value="100" />
    <property name="commitInterval" value="250" />
    </bean>


    <bean id="aplusAutoFileToGeo" parent="simpleJob">
    <property name="steps">
    <list>

    <bean id="step1" parent="simpleStep">
    <property name="streams">
    <list>
    <ref bean="inputReader" />
    </list>
    </property>
    <property name="itemReader" ref="fileReader" />
    <property name="itemWriter" ref="fileWriterTarget" />
    <property name="commitInterval" value="250" />
    <property name="listeners">
    <list>
    <bean parent="stepExecutionListener"/>
    <bean parent="itemExecutionListener"/>
    </list>
    </property>
    </bean>
    </list>
    </property>
    </bean>



    Please kindly throw some light.

    -Lakshmanan

  • #2
    Please use code tags...

    What does your DB schema look like? What indices do you have?

    Does the performance start out fast with table(s) that already have several thousand rows or are you measuring the performance on empty table(s)?

    Comment


    • #3
      I am checking against the table that has several thousand records already. But it starting really fast but degrading over time.

      Leaving that issue aside for a minute, i could not have this commitinterval parameter working in spring batch.

      I appreciate your help.

      -Lakshmanan

      Comment


      • #4
        It doesn't sound like the commit interval isn't working, it sounds like it's not having the effect you want, which could be for a number of reasons. I agree that increasing the commit interval should equate to an increase in speed, but without more details it's hard to say. Can you post the reader and writer, etc that you're using?

        Comment


        • #5
          You are right that i am not seeing the effect of that property.

          I am using FlatFileItemReader to read the data from the text file. This is not multiline record.

          I am using the Itemwriter which intracts with the sqlserver to persist the rows into database.

          Initially good, but degrade badly over time.

          Thanks for looking into it. Let me know if you need more information.

          -Lakshmanan

          Comment


          • #6
            Originally posted by lvenkatachalam View Post
            Thanks for looking into it. Let me know if you need more information.

            -Lakshmanan
            Several of us have now asked you for more information and you haven't really provided it. As far as I'm concerned, the ball is in your court until you are more forthcoming with your configuration...

            Comment

            Working...
            X