Announcement Announcement Module
No announcement yet.
Spring Batch: Problem with chunk and commit-interval Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Spring Batch: Problem with chunk and commit-interval


    I'm using Spring Batch to read from a file and then to write into a data Base. And there is a file structure example (not real):

    01565466516597JackLor 874854984984

    the tow first lines are data lines, and the last is for control:
    cccccc means control line
    2 the number of lines in the file
    93 the sum of values after the name (index 24): 06+87

    After reading all lines from the file we have to read the control line and do verification. If OK we commit else we stop (BATCH FAIL).
    the problem is that if I have a commit-interval 100 and I have 1000 line in the file. The job will read the 100 first line and it will not find the control line. So the job will stop.
    If I keep the verification in the end, the job will commit every 100 line, so if the control is not OK we can't roll back.

    I use :
    ItemReader: org.springframework.batch.item.file.FlatFileItemRe ader
    lineMapper: org.springframework.batch.item.file.mapping.Defaul tLineMapper
    lineTokenizer: org.springframework.batch.item.file.transform.Fixe dLengthTokenizer
    itemWriter: org.springframework.batch.item.database.JdbcBatchI temWriter

    and step config:

    <batch:step id="step1">
    <batch:tasklet transaction-manager="transactionManager" >
    <batch:chunk reader="myItemReader" processor="myProcessor"
    writer="myItemWriter" commit-interval="100">

    can I make a tasklet without a chunk and without commit interval ?

    Or can I read by chunk but do commit in the end (if OK).

    Thank you in advance.
    Last edited by gitos; Aug 5th, 2012, 08:55 AM.

  • #2
    I have very similar problem, have you found a solution for this ?


    • #3
      I think i found a solution of the problem, I will set commit-interval quite big, and write a custom chunk-completion-policy. This will probably work ... i hope so.


      • #4
        Just make it 2 seperate steps: the first step does the control check, the second step writes the records to the DB.


        • #5
          You either set commit interval or have a chunk completion policy and in your case, you need to have the chunk completion policy and not the commit interval


          • #6
            Originally posted by jpraet View Post
            Just make it 2 seperate steps: the first step does the control check, the second step writes the records to the DB.
            I do the first step well but the second step i can't write the records? What's happened?


            • #7
              To me this is similar to the question I asked before, how do we do cross item data validation for the input file as a whole?


              • #8
                DefaultResultCompletionPolicy for 1 chunk for entire file

                Just learning Spring Batch but I had the same problem. Our transactional requirements are to process the entire file or nothing at all, so I needed a commit-interval of #lines. I used the completion policy idea but found that I could use the provided org.springframework.batch.repeat.policy.DefaultRes ultCompletionPolicy in the chunk attribute chunk-completion-policy. Saved me from writing something custom. I haven't tested exhaustively but seems to be doing what I want.

                <batch:job id="myJob"><batch:step id="processAFile">
                [INDENT=2]<batch:chunk reader="myReader" writer="myFileWriter" [/INDENT][INDENT=4] chunk-completion-policy="wholeFileCompletionPolicy"/> [/INDENT][INDENT=2] </batch:tasklet> [/INDENT]
                </batch:job> <bean id="wholeFileCompletionPolicy" class="org.springframework.batch.repeat.policy.DefaultResultCompletionPolicy"/>