Announcement Announcement Module
Collapse
No announcement yet.
CommandLineJobRunner and MultiResourcePartitionner Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • CommandLineJobRunner and MultiResourcePartitionner

    Hi !
    I built a job to process multiple huge xml gzipped files in parallel and write each input file to a distinct tsv file. For that, I use MultiResourcePartitionner.
    Everything works well when I launch my job from a junit test using a SimpleJobLauncher. I test with 3 huge input files, and I can actually see 3 output tsv files being created in parallel.

    When I launch that same job from the commandLine, I only get 1 file processed at a time...
    Is there any specific TaskExecutor type I have to use for the CommandLineJobRunner to behave as the JobRunner in my Junit test ?

    Below is my spring config file :

    HTML Code:
    <?xml version="1.0" encoding="UTF-8"?>
    <beans xmlns="http://www.springframework.org/schema/beans"
    	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:batch="http://www.springframework.org/schema/batch"
    	xmlns:oxm="http://www.springframework.org/schema/oxm" xmlns:int="http://www.springframework.org/schema/integration"
    	xmlns:int-stream="http://www.springframework.org/schema/integration/stream"
    	xmlns:context="http://www.springframework.org/schema/context"
    	xmlns:task="http://www.springframework.org/schema/task"
    	xsi:schemaLocation="http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration-2.0.xsd
    		http://www.springframework.org/schema/oxm http://www.springframework.org/schema/oxm/spring-oxm-3.0.xsd
    		http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd
    		http://www.springframework.org/schema/integration/stream http://www.springframework.org/schema/integration/stream/spring-integration-stream-2.0.xsd
    		http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
    		http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
    		http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd">
    
    	<context:component-scan base-package="com.mycompany"/>	
    	<task:annotation-driven/>
    	
    	<oxm:jaxb2-marshaller id="myMarshaller"
    		contextPath="com.mycompany.domain">
    	</oxm:jaxb2-marshaller>
    		        	
    	<bean id="myReader" class="org.springframework.batch.item.xml.StaxEventItemReader"
    		scope="step">
    		<property name="fragmentRootElementName" value="RootItem" />
    		<property name="saveState" value="false"/>
    		<property name="resource">
    			<bean class="org.springframework.core.io.InputStreamResource">
    				<constructor-arg name="inputStream">
    					<bean class="java.util.zip.GZIPInputStream">
    						<constructor-arg>
    							<bean class="com.mycompany.utils.io.ResourceInputFile" factory-method="getInputStream">
    								<constructor-arg value="#{stepExecutionContext[fileName]}" />
    							</bean>
    						</constructor-arg>
    					</bean>
    				</constructor-arg>
    			</bean>
    		</property>
    		<property name="unmarshaller" ref="myMarshaller" />
    	</bean>
    
            <bean id="myProcessor" class="my.company.batch.MyItemProcessor" />
    
    	<bean id="myFileWriter" class="org.springframework.batch.item.file.FlatFileItemWriter"
    		scope="step">
    		<property name="resource" value="file:#{stepExecutionContext[outputDbName]}.tsv" />
    		<property name="shouldDeleteIfExists" value="true"/>
    		<property name="transactional" value="false"/>
    		<property name="lineAggregator">
    			<bean class="my.company.batch.MyLineAggregator" />
    		</property>
    	</bean>
    
            <bean id="fileNameListener" class="my.company.batch.MyFileNameListener"/>
        
    	<batch:step id="xml_to_tsv_step">
    		<batch:tasklet transaction-manager="transactionManager">
    			<batch:chunk reader="myReader" processor="myProcessor" writer="myFileWriter" commit-interval="100" />
    			<batch:listeners>
    				<batch:listener ref="fileNameListener"/>
    			</batch:listeners>
    		</batch:tasklet>
    	</batch:step>
    	
    	
    	<bean id="partitioner" class="org.springframework.batch.core.partition.support.MultiResourcePartitioner" scope="step">
    		<property name="resources" value="file:#{jobParameters['input.dir.name']}"/>
    	</bean>
    			
    	<batch:job id="partitionJob">
    		<batch:step id="step1">
    			<batch:partition step="xml_to_tsv_step" partitioner="partitioner">		
    				<batch:handler grid-size="10" task-executor="taskExecutor"/>		
    			</batch:partition>
    		</batch:step>
    	</batch:job>
    
    	<task:executor id="taskExecutor" queue-capacity="100" pool-size="10"/>
    
    </beans>
    Thanks in advance for your help.

    Best regards,

    Philippe
    Last edited by psoares; Sep 30th, 2011, 08:18 AM.

  • #2
    I found out what my problem was.
    I was calling my job in a script this way :
    Code:
    #!/bin/bash
    
    typeset inputFile=$1
    [...]
    java org.springframework.batch.core.launch.support.CommandLineJobRunner process-multiple-files-context.xml partitionJob input.file.name=$inputFile
    Then I launched my script with :
    > ./myscript.sh /path/to/my/files*.xml.gz

    When I do this instead, it works :
    > ./myscript.sh "/path/to/my/files*.xml.gz"

    I guess the first version was doing shell expansion and got the first file matching my file glob.

    Comment

    Working...
    X