Announcement Announcement Module
Collapse
No announcement yet.
Using JobParameters in hadoop namespace Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Using JobParameters in hadoop namespace

    I can't get the jobParamenters to be late loaded:

    here is my config:

    Code:
    <?xml version="1.0" encoding="UTF-8"?>
    <beans:beans xmlns="http://www.springframework.org/schema/hadoop"
    	xmlns:beans="http://www.springframework.org/schema/beans"
    	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    	xmlns:batch="http://www.springframework.org/schema/batch"
    	xmlns:context="http://www.springframework.org/schema/context"
    	xsi:schemaLocation="http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
    		http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
    		http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
    		http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">
    
    
    	<beans:bean class="org.springframework.batch.core.scope.StepScope">
    		<beans:property name="proxyTargetClass" value="true"/>
    	</beans:bean>
    
    	
    	<batch:job id="csv-to-cube-BATCH-job">
    		<batch:step id="csv-import" next="csv-to-cube-step">
    			<batch:tasklet ref="csv-import-tasklet"/>
    		</batch:step>
    		<batch:step id="csv-to-cube-step" >
    			<batch:tasklet ref="csv-to-cube-tasklet" />
    		</batch:step>
    	</batch:job>
    
        <script-tasklet id="csv-import-tasklet"  >
            <script location="cp-data.js">
                <property name="inputPath" value="cube-csv-inputs/tmp" />
                <property name="output" value="cubes/adwork" />
                <property name="localResource" value="cube-csv-inputs/adworks.csv" />
            </script>
        </script-tasklet>
    	
    
    	<tasklet id="csv-to-cube-tasklet" job-ref="csv-to-cube-MR-job" scope="step"/>
    
    	<job id="csv-to-cube-MR-job" scope="step"
    		properties-location="#{jobParameters['properties-file']}"
    		configuration-ref="hbaseConfiguration"
    		input-path="${input}"
    		output-path="${output}"
    		mapper="haruspex.etl.csv.CsvToCubeMapper"
    		reducer="haruspex.etl.csv.CsvToCubeReducer"
    		validate-paths="false" 
    		/>
    		
    	
    </beans:beans>
    this is the error:
    Code:
    ERROR 2012-06-13 10:04:24,634 [org.springframework.batch.core.step.AbstractStep(AbstractStep.java)]: Encountered an error executing the step
    org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.csv-to-cube-MR-job': Cannot create inner bean '(inner bean)' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] while setting bean property 'properties'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)': Invocation of init method failed; nested exception is java.io.FileNotFoundException: class path resource [#{jobParameters['properties-file']}] cannot be opened because it does not exist
    	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:281)
    	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:125)
    .
    .
    .
    Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)': Invocation of init method failed; nested exception is java.io.FileNotFoundException: class path resource [#{jobParameters['properties-file']}] cannot be opened because it does not exist
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1455)
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
    	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:270)
    	... 78 more
    Caused by: java.io.FileNotFoundException: class path resource [#{jobParameters['properties-file']}] cannot be opened because it does not exist
    	at org.springframework.core.io.ClassPathResource.getInputStream(ClassPathResource.java:158)
    	at org.springframework.core.io.support.PropertiesLoaderSupport.loadProperties(PropertiesLoaderSupport.java:181)
    	at org.springframework.core.io.support.PropertiesLoaderSupport.mergeProperties(PropertiesLoaderSupport.java:150)
    	at org.springframework.beans.factory.config.PropertiesFactoryBean.createInstance(PropertiesFactoryBean.java:113)
    	at org.springframework.beans.factory.config.PropertiesFactoryBean.createProperties(PropertiesFactoryBean.java:98)
    	at org.springframework.beans.factory.config.PropertiesFactoryBean.afterPropertiesSet(PropertiesFactoryBean.java:69)
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1514)
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1452)
    	... 81 more
    I have read similar posts on the batch forum but they aren't helping...

  • #2
    Hmm... I have to try this myself. If you specify the job parameters on a different property do they get properly replaced?

    Comment


    • #3
      Yes - after some fiddling around I can get both the script-tasklet and the hdp:job to find #{jobParameter['monkey']} except for the properties-ref. I think this is a life cycle issue - but it sure would be convenient to be able to load the hadoopConfiguration from a passed in file.

      Comment


      • #4
        Can you please raise a bug mentioning the fields where jobParameter works and where it doesn't? thanks!

        Comment


        • #5
          Could you show us how you accomplished this, please?

          Comment

          Working...
          X