Announcement Announcement Module
Collapse
No announcement yet.
Problem with JpaPagingItemReader and pageSize Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    Couldn't wait tomorrow
    I have my JpaItemReader (pageSize = 8), 15 persons in my db. I have a processor which adds 2 addresses per person. I use a JpaItemWriter and I have a commit interval = 8. At the end of my test, I have ... 46 adresses (???) in my db.
    If I change the commit interval to 4 and pageSize to 4, I have 54 adresses in my db.
    I works only if pageSize > total number of record in the db.

    Comment


    • #17
      Do you mind posting the code and config you used for JpaItemWriter please?
      There's a Jira related to it, why flush/clear was moved, damn old: https://jira.springsource.org/browse/BATCH-1110
      And are you sure about those numbers? Are you clearing the address before testing? So we'll know how many updates we had.
      Last edited by traduz; Jul 26th, 2012, 10:45 PM.

      Comment


      • #18
        The relevant parts :

        <batch:job id="minimal" job-repository="jobRepository"
        restartable="true">

        <batch:step id="step1">
        <batch:tasklet>
        <batch:chunk reader="jpaPersonReader" processor="addAdressToPersonProcessor"
        writer="jpaPersonWriter" commit-interval="4" />
        </batch:tasklet>
        </batch:step>
        </batch:job>

        <bean id="addAdressToPersonProcessor" class="org.jp.spring_batch_labs.AddAddressToPerson Processor" />

        <bean id="jpaPersonReader"
        class="org.springframework.batch.item.database.Jpa PagingItemReader">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="queryString" value="select p from Person p" />
        <property name="pageSize" value="4" />
        </bean>

        <bean id="jpaPersonWriter" class="org.springframework.batch.item.database.Jpa ItemWriter">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        </bean>

        The processor :

        public class AddAddressToPersonProcessor implements ItemProcessor<Person, Person> {
        private static final Logger LOG = Logger
        .getLogger(AddAddressToPersonProcessor.class);

        public Person process(Person person) throws Exception {
        LOG.info("processing : " + person);
        Address address = new Address();
        address.setCp("92160");
        address.setStreet("street1");
        person.addAddress(address);

        Address address2 = new Address();
        address2.setCp("92160");
        address2.setStreet("street2");
        person.addAddress(address2);
        return person;
        }
        }


        My u.t. :

        public class SimpleTestJpa {

        private static final Logger LOG = Logger.getLogger(SimpleTestJpa.class);
        static ApplicationContext ac = new ClassPathXmlApplicationContext(
        new String[] { "spring-test-jpa.xml" });

        @BeforeClass
        public static void beforeClass() {
        // Création du schéma pour spring batch
        SimpleJdbcTemplate jdbcTemplate = (SimpleJdbcTemplate) ac
        .getBean("jdbcTemplate");
        Resource resource = new ClassPathResource(
        "/create_drop_spring_batch.sql");
        SimpleJdbcTestUtils.executeSqlScript(jdbcTemplate, resource, true);

        // dbunit : population des tables
        DataSource dataSource = (DataSource) ac.getBean("dataSource");
        Connection con = DataSourceUtils.getConnection(dataSource);
        IDatabaseConnection dbUnitCon = new DatabaseConnection(con);

        try {
        IDataSet dataSet = new FlatXmlDataSet(new FileInputStream(
        "./src/test/resources/persons_dbunit.xml"));
        DatabaseOperation.REFRESH.execute(dbUnitCon, dataSet);
        } catch (Exception e) {
        e.printStackTrace();
        } finally {
        DataSourceUtils.releaseConnection(con, dataSource);
        }
        }

        @Test
        public void simpleTest() throws Exception {
        Job job = (Job) ac.getBean("minimal");
        JobLauncher jobLauncher = (JobLauncher) ac.getBean("jobLauncher");
        // Lancement effectif
        JobExecution je = jobLauncher.run(job, new JobParametersBuilder()
        .addString("param1", "1").toJobParameters());
        }

        @AfterClass
        public static void afterClass() {
        DataSource dataSource = (DataSource) ac.getBean("dataSource");
        int count = 0;
        try {
        ResultSet rs = dataSource.getConnection().createStatement()
        .executeQuery("select count(*) from Address");
        rs.next();
        count = rs.getInt(1);
        rs.close();
        } catch (SQLException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
        }
        LOG.info("Nbre d'enregistrements : " + count);

        }
        }


        My dataset :
        <?xml version='1.0' encoding='UTF-8'?>
        <dataset>
        <PERSON ID='1' NAME="name1"/>
        <PERSON ID='2' NAME="name2"/>
        <PERSON ID='3' NAME="name3"/>
        <PERSON ID='4' NAME="name3"/>
        <PERSON ID='5' NAME="name3"/>
        <PERSON ID='6' NAME="name3"/>
        <PERSON ID='7' NAME="name3"/>
        <PERSON ID='8' NAME="name3"/>
        <PERSON ID='9' NAME="name3"/>
        <PERSON ID='10' NAME="name3"/>
        <PERSON ID='11' NAME="name3"/>
        <PERSON ID='12' NAME="name3"/>
        <PERSON ID='13' NAME="name3"/>
        <PERSON ID='14' NAME="name3"/>
        <PERSON ID='15' NAME="name3"/>
        </dataset>

        Person :
        @Entity
        public class Person {
        @Id
        @GeneratedValue(strategy = GenerationType.AUTO)
        private Long id;
        private String name;
        @OneToMany(cascade = CascadeType.ALL)
        @JoinTable(name = "PERSON_ADDRESS", joinColumns = { @JoinColumn(name = "PERSON_ID") }, inverseJoinColumns = { @JoinColumn(name = "ADDRESS_ID") })
        private Set<Address> addresses = new HashSet<Address>();
        ...+get/set


        Address :
        @Entity
        public class Address {
        @Id
        @GeneratedValue(strategy = GenerationType.AUTO)
        private Long id;
        private String cp;
        private String street;
        ...+get/set

        Comment


        • #19
          Thanks, next time use the code tag so will be better to read it my friend. I'll take a look at it.
          Also you didn't answered, are you clearing the address before testing? So we'll know how many updates we had.

          Comment


          • #20
            Sorry about the 'code tag'
            I forgot to answer for the address : yes. The db is empty and just filled with the dataset above (only 'persons' wo address).

            Comment


            • #21
              No problem.
              Originally posted by jprio View Post
              Couldn't wait tomorrow
              I have my JpaItemReader (pageSize = 8), 15 persons in my db. I have a processor which adds 2 addresses per person. I use a JpaItemWriter and I have a commit interval = 8. At the end of my test, I have ... 46 adresses (???) in my db.
              Assuming everything you said is correct, looks like the first page (8 person, 16 address) are saved in the JpaItemWriter (correct) but when returning to ItemReader they are being saved again, so first page makes 32 records. Next page (7 persons, 14 address), ItemWriter saves it, giving us 46 records.
              JpaItemWriter already does a merge + flush so the entities should be sync and Reader wouldn't update them. So weird.

              Comment

              Working...
              X