Problem with JpaPagingItemReader and pageSize

[jprio]

We have a job configured like this :

Batch configuration :

    <batch:job id="minimal" job-repository="jobRepository"
        restartable="true">

        <batch:step id="step1">
            <batch:tasklet>
                <batch:chunk reader="jpaPersonReader" writer="addAddressToPersonWriter" commit-interval="4" />
            </batch:tasklet>
        </batch:step>
    </batch:job>

    <bean id="jpaPersonReader"
        class="org.springframework.batch.item.database.JpaPagingItemReader">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="queryString" value="select p from Person p order by p.id" />
        <property name="pageSize" value="4" />
    </bean>
    
    <bean id="addAddressToPersonWriter" class="org.jp.spring_batch_labs.AddAddressToPersonWriter" />

AddAddressToPersonWriter class :

public class AddAddressToPersonWriter implements ItemWriter<Person> {

    private static final Logger LOG = Logger.getLogger(AddAddressToPersonWriter.class);

    @Override
    public void write(List<? extends Person> items) throws Exception {
        LOG.info("updating " + items.size() + " persons.");
        for (Person person : items) {
            Address address = new Address();
            address.setCp(Long.toString(System.currentTimeMillis()));
            person.addAddress(address);
        }
    }
}

We launch the batch with a test which initialize 15 persons in the @BeforeClass.
15 persons are read by JpaPagingItemReader and modified by our AddAddressToPersonWriter but at the end, we can see 12 addresses in the database instead of 15.

Our commit-interval is 4, so 12 commited modifications = 3 pages * 4 items per page, and we have the 3 last records not modified.

Output logs :

--  INFO org.springframework.batch.core.launch.support.SimpleJobLauncher - Job: [FlowJob: [name=minimal]] launched with the following parameters: [{param1=1}]
--  INFO org.springframework.batch.core.job.SimpleStepHandler - Executing step: [TaskletStep: [name=step1]]
-- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 0
--  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 4 persons.
-- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 1
--  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 4 persons.
-- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 2
--  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 4 persons.
-- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 3
--  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 3 persons.
--  INFO org.springframework.batch.core.launch.support.SimpleJobLauncher - Job: [FlowJob: [name=minimal]] completed with the following parameters: [{param1=1}] and the following status: [COMPLETED]
--  INFO org.jp.spring_batch_labs.SimpleTestJpa - 12 adresses in the db

We noticed that the JpaPagingItemReader has been modified since version 1.1.3 : the flush has been moved from the end of the method doReadPage to the beginning of this method. If we write our own JpaPagingItemReader with a flush at the end of the doReadPage method, we solve our problem.

So :
- why was the flush moved from the end to the beginning of the method ?
- is there any risk in replacing it at the end ?
- is there a Jira that references this pb and will it be soon fixed ?

Thank you for your answers,

JP

[visualjeff]

Just curious... Could you run your test with a commit-interval="1" to see if the results differ?

Thanks,

Jeff

[visualjeff]

If you do change the commit-interval for a test... It nees to matche the page size used by JpaPagingItemReader.

[visualjeff]

Wow, did I have a brain fart.

It nees to matche the page size used by JpaPagingItemReader.

Should be:

It needs to match the page size used by JpaPagingItemReader.

Anyway, I suspect you already know this.

Jeff

[jprio]

The behaviour is the same : the items of the last page are not commited ==> with pageSize = 1, the last record will not have is modifications persisted.

[traduz]

Commit interval is related to writer, and in the log you provided it's receiving the 15 records.

-- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 0
--  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 4 persons.
-- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 1
--  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 4 persons.
-- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 2
--  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 4 persons.
-- DEBUG org.springframework.batch.item.database.JpaPagingItemReader - Reading page 3
--  INFO org.jp.spring_batch_labs.AddAddressToPersonWriter - updating 3 persons.

ItemReader is not meant to update any data, just retrieve them, AFAIK.
And why not using JdbcPagingItemReader? Better performance.

visualjeff you could just edit your post instead of flooding.

[jprio]

I agree with you : the itemreader just reads. But the AddAddressToPersonWriter do modifies.
About JdbcPagingItemReader, we use Jpa in our app to leverage the orm layer and we don't want to drop it.

[traduz]

So the writer should update the data in db as well, don't you think? That's what the doc says.

[jprio]

Yes, that's what happens... except for the items of the last page.

[traduz]

No I mean, don't rely in the Reader to update your data, this isn't the right way to do it. The writer should use an update method.
That's what the documentation says: http://static.springsource.org/sprin...ndWriters.html
6.2. ItemWriter

[jprio]

Well, what I get in the writer is an object attached to an entitymanager (because it is read by the em). So if I modify this object, the modification should be synchronized in the db on a flush, right (and it is for the first pages) ?

[traduz]

Yes you are right, but don't you agree the writer should have the responsability to update and save those changes?
So flush it in the writer.

[jprio]

Humm may be. I'm not sure. But, anyway, to do this, I have to have access to the em instanciated by Spring (from the emf) for this transaction (i.e. this page) and I don't know how I could do this (AFAIK there is no way to get the "current em" with JPA).

[traduz]

Use JpaItemWriter.

[jprio]

Yep, thanx. I'll try tomorrow (processor + JpaItemWriter) and I'll let you know.