Announcement Announcement Module
Collapse
No announcement yet.
Problem with JpaPagingItemReader and pageSize Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Problem with JpaPagingItemReader and pageSize

    We have a job configured like this :

    Batch configuration :
    Code:
        <batch:job id="minimal" job-repository="jobRepository"
            restartable="true">
    
            <batch:step id="step1">
                <batch:tasklet>
                    <batch:chunk reader="jpaPersonReader" writer="addAddressToPersonWriter" commit-interval="4" />
                </batch:tasklet>
            </batch:step>
        </batch:job>
    
        <bean id="jpaPersonReader"
            class="org.springframework.batch.item.database.JpaPagingItemReader">
            <property name="entityManagerFactory" ref="entityManagerFactory" />
            <property name="queryString" value="select p from Person p order by p.id" />
            <property name="pageSize" value="4" />
        </bean>
        
        <bean id="addAddressToPersonWriter" class="org.jp.spring_batch_labs.AddAddressToPersonWriter" />
    AddAddressToPersonWriter class :
    Code:
    public class AddAddressToPersonWriter implements ItemWriter<Person> {
    
        private static final Logger LOG = Logger.getLogger(AddAddressToPersonWriter.class);
    
        @Override
        public void write(List<? extends Person> items) throws Exception {
            LOG.info("updating " + items.size() + " persons.");
            for (Person person : items) {
                Address address = new Address();
                address.setCp(Long.toString(System.currentTimeMillis()));
                person.addAddress(address);
            }
        }
    }
    We launch the batch with a test which initialize 15 persons in the @BeforeClass.
    15 persons are read by JpaPagingItemReader and modified by our AddAddressToPersonWriter but at the end, we can see 12 addresses in the database instead of 15.

    Our commit-interval is 4, so 12 commited modifications = 3 pages * 4 items per page, and we have the 3 last records not modified.

    Output logs :
    Code:
    --  INFO org.springframework.batch.core.launch.support.SimpleJobLauncher - Job: [FlowJob: [name=minimal]] launched with the following parameters: [{param1=1}]
    --  INFO org.springframework.batch.core.job.SimpleStepHandler - Executing step: [TaskletStep: [name=step1]]
    -- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 0
    --  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 4 persons.
    -- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 1
    --  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 4 persons.
    -- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 2
    --  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 4 persons.
    -- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 3
    --  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 3 persons.
    --  INFO org.springframework.batch.core.launch.support.SimpleJobLauncher - Job: [FlowJob: [name=minimal]] completed with the following parameters: [{param1=1}] and the following status: [COMPLETED]
    --  INFO org.jp.spring_batch_labs.SimpleTestJpa - 12 adresses in the db

    We noticed that the JpaPagingItemReader has been modified since version 1.1.3 : the flush has been moved from the end of the method doReadPage to the beginning of this method. If we write our own JpaPagingItemReader with a flush at the end of the doReadPage method, we solve our problem.

    So :
    - why was the flush moved from the end to the beginning of the method ?
    - is there any risk in replacing it at the end ?
    - is there a Jira that references this pb and will it be soon fixed ?

    Thank you for your answers,

    JP

  • #2
    Just curious... Could you run your test with a commit-interval="1" to see if the results differ?

    Thanks,

    Jeff

    Comment


    • #3
      If you do change the commit-interval for a test... It nees to matche the page size used by JpaPagingItemReader.

      Comment


      • #4
        Wow, did I have a brain fart.

        It nees to matche the page size used by JpaPagingItemReader.

        Should be:

        It needs to match the page size used by JpaPagingItemReader.

        Anyway, I suspect you already know this.

        Jeff

        Comment


        • #5
          The behaviour is the same : the items of the last page are not commited ==> with pageSize = 1, the last record will not have is modifications persisted.

          Comment


          • #6
            Commit interval is related to writer, and in the log you provided it's receiving the 15 records.
            Code:
            -- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 0
            --  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 4 persons.
            -- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 1
            --  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 4 persons.
            -- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 2
            --  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 4 persons.
            -- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 3
            --  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 3 persons.
            ItemReader is not meant to update any data, just retrieve them, AFAIK.
            And why not using JdbcPagingItemReader? Better performance.

            visualjeff you could just edit your post instead of flooding.
            Last edited by traduz; Jul 26th, 2012, 10:30 AM.

            Comment


            • #7
              I agree with you : the itemreader just reads. But the AddAddressToPersonWriter do modifies.
              About JdbcPagingItemReader, we use Jpa in our app to leverage the orm layer and we don't want to drop it.

              Comment


              • #8
                So the writer should update the data in db as well, don't you think? That's what the doc says.

                Comment


                • #9
                  Yes, that's what happens... except for the items of the last page.

                  Comment


                  • #10
                    No I mean, don't rely in the Reader to update your data, this isn't the right way to do it. The writer should use an update method.
                    That's what the documentation says: http://static.springsource.org/sprin...ndWriters.html
                    6.2. ItemWriter

                    Comment


                    • #11
                      Well, what I get in the writer is an object attached to an entitymanager (because it is read by the em). So if I modify this object, the modification should be synchronized in the db on a flush, right (and it is for the first pages) ?

                      Comment


                      • #12
                        Yes you are right, but don't you agree the writer should have the responsability to update and save those changes?
                        So flush it in the writer.

                        Comment


                        • #13
                          Humm may be. I'm not sure. But, anyway, to do this, I have to have access to the em instanciated by Spring (from the emf) for this transaction (i.e. this page) and I don't know how I could do this (AFAIK there is no way to get the "current em" with JPA).

                          Comment


                          • #14
                            Use JpaItemWriter.

                            Comment


                            • #15
                              Yep, thanx. I'll try tomorrow (processor + JpaItemWriter) and I'll let you know.

                              Comment

                              Working...
                              X