Announcement Announcement Module
Collapse
No announcement yet.
configure mutithread and parallel step for a batch load Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • configure mutithread and parallel step for a batch load

    I am evaluating the technology for a batch load. Our input volume is 6 million record. I am doing springbatch vs ETL performance analysis. Here are the information for the project.

    1. There are six pipe delimited files. Each file has 1 million records.
    2. One file has to be processed first. After that, other five files can be processed in parallel. So after the first file is processed, I want to use
    parallel steps to process other five files.
    3. Each record is independent within each file. So I want to use multithread in each step. The batch load will read each record, process it and then load it to database.

    I am pretty new to Springbath. I would like to know whether use case 3 design is valid and how to configure use case 3 and 2. The main goal is to test the performance of Springbatch. Any help is greatly appreciated.

    Wayne

  • #2
    I try this configuration for use case 3. The code is running. But I wonder whether I need to move the itemreader after the taskexecutor.

    Code:
    <bean id="loadDemographicInformation" parent="abstractSimpleStep"
    		p:itemReader-ref="demographicInformationReader"
    		p:itemKeyGenerator-ref="demographicInformationReader">
    		<property name="taskExecutor">
    		     <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor" >
    		    </bean> 
    		</property>
    		<property name="itemWriter">
    			<bean class="whi.prescriber.load.writer.SummaryReportItemWriter"
    			p:delegate-ref="demographicItemTransformerItemWriter">
    			</bean>
    		</property>
    		<property name="listeners">
    			<list>
    				<ref bean="demographicInformationFile"/>
    				<ref bean="reportListener"/>
    			</list>
    		</property>
    	</bean>
    Last edited by Dave Syer; Aug 20th, 2009, 06:50 AM. Reason: added code tags

    Comment


    • #3
      I'm not sure what you mean by "move the itemreader after the taskexecutor". Is something not working?

      It looks like you are using an old version of Spring Batch (1.1.x?). In that versions option 3 was easy and works out of the box the way you have done it. If you upgrade there are more options and your other use cases are better supported (http://static.springsource.org/sprin...alability.html).

      Comment

      Working...
      X