Announcement Announcement Module
Collapse
No announcement yet.
Performances problems : Very long treatment OR Java Heap Space ... Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Performances problems : Very long treatment OR Java Heap Space ...

    Hello,

    I'm new to Spring batch and have to do a simple job with it.
    I got to request 9.000.000 rows and to work on them to get extra information from the database.

    I'm working on spring-batch "2.1.9.RELEASE" and my database is "Postgresql 9.0".

    I tried different config :

    1) Use a JdbcCursorItemReader to read my db.
    the fetsize is set to 1000, but if my request is a select *, then i have a Java Heap Space.
    if my request is like this "select a from T", i don't have any Java Heap Space, but I don't have all the information i need ...

    2) Use a JdbcPagingItemReader
    the pageSize is set to 1000. The treatment works fine but I need 321 sec to read and write 1000 rows !
    -> 5 days for 9.000.000 rows

    3) Use a JpaPagingItemReader
    the pageSize is set to 100. The treatment works fine but I need 15 sec to read and write 1000 rows !
    -> 1.5 days for 9.000.000 rows

    I'm a little bit confused...

    I must have done something wrong don't you think ?

    Here's my xml config file :

    Code:
    <bean id="jobLauncher"
    		class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
    		<property name="jobRepository" ref="jobRepository" />
    	</bean>
    	
    	<bean id="jobRepository"
    		class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
    		<property name="transactionManager" ref="transactionManager" />
    		<property name="isolationLevelForCreate" value="ISOLATION_DEFAULT" />
    	</bean>
    
    <batch:job id="batchName" xmlns="http://www.springframework.org/schema/batch">
    		<batch:step id="stepName">
    			<batch:tasklet>
    				<batch:chunk reader="inputReader" processor="inputProcessor" writer="inputWriter" processor-transactional="false" commit-interval="1000" skip-limit="50">
                	</batch:chunk>
    			</batch:tasklet>
    		</batch:step>
    	</batch:job>
    
    <bean id="inputWriter" class="org.springframework.batch.item.xml.StaxEventItemWriter" scope="step">		
    		<property name="rootTagName" value="inputElements" />
    		<property name="overwriteOutput" value="true" />
    		<property name="resource"> 
    			<value>file:#{jobParameters["OUTPUT_FILE"]}</value>
    		</property>
    		<property name="marshaller" ref="marshallerJaxb" />
    	</bean>
    	
    	<bean id="marshallerJaxb" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
    		<property name="classesToBeBound">
    			<list>
    				<value>ClassA</value>
    				<value>ClassB</value>
    			</list>
    		</property>
    		<property name="schema" value="classpath:schema.xsd" />
    	</bean>
    Here's my readers :
    1) JdbcCursorItemReader
    Code:
    <bean id="inputReader" class="org.springframework.batch.item.database.JdbcCursorItemReader" scope="step">
    		<property name="dataSource" ref="dataSource"/>
        	<property name="sql"> 
        		<value>SELECT a FROM T WHERE b IS NULL OR b = ''</value> 
    		</property>
    		<property name="fetchSize" value="100" />	
        	<property name="rowMapper">
            	<bean class="RowMapper"/>
       		</property>
    	</bean>
    	<bean id="inputProcessor" class="InputProcessor" />
    2) JdbcPagingItemReader
    Code:
    <bean id="inputReader" class="org.springframework.batch.item.database.JdbcPagingItemReader" scope="step">	    
    	    <property name="dataSource" ref="dataSource" />
    		<property name="queryProvider">
    			<bean id=" queryProvider" class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
    				<property name="dataSource" ref="dataSource" />
    				<property name="selectClause" value="SELECT a" />
    				<property name="fromClause" value="T" />
    				<property name="whereClause" value="b IS NULL OR b = ''" />
    				<property name="sortKey" value="a" />
    			</bean>
    		</property>
    		<property name="pageSize" value="1000" />
        	<property name="rowMapper">
            	<bean class="RowMapper"/>
       		</property>
    	</bean>
    	<bean id="inputProcessor" class="InputProcessor" />
    3) JpaPagingItemReader
    Code:
    <bean id="inputReader" class="org.springframework.batch.item.database.JpaPagingItemReader" scope="step">
    		<property name="entityManagerFactory" ref="entityManagerFactory"/>
    		<property name="queryString" value="select e from TEntity e WHERE e.b IS NULL OR e.b = ''"/>
    		<property name="pageSize" value="1000"/>
    	</bean>
    	<bean id="inputProcessor" class="JpaInputProcessor" />
    Thx for your help !
Working...
X