Announcement Announcement Module
Collapse
No announcement yet.
Passing Attributes/Parameters Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Passing Attributes/Parameters

    This may have been answered before, but I did not see it.

    How do you pass attributes between Steps? As an example, let's say you have a step that has a Tasklet that FTP's a compressed file from a remote location and places it on the local file system. The next step has a Tasklet that decompress' the file. How would one pass a String that is the full path to the local file from the FTP Tasklet to the decompress Tasklet?

    To extend this question, how would one pass the file path to the decompressed file to a Step that contains an ItemReader to read the file?

  • #2
    I have done some more digging and it seems the ExecutionContext within an ItemReader will save the information to the DB. HOWEVER, when you do:

    ChunkContext.getStepContext().getJobExecutionConte xt().put("key","value")

    You get an Exception because the ExecutionContext is immutable. This is starting to sound like a bug. Why couldn't a Tasklet insert information into the ExecutionContext?

    Comment


    • #3
      I also tried a test of Saving the JobExecutionContext via the ExeuctionContextDao. The JobExecutionContext does not seem to save to the DB. I see it in the BATCH_STEP_EXECUTION_CONTEXT table, but not in the BATCH_JOB_EXECUTION_CONTEXT table.

      Comment


      • #4
        The StepContext has a read only version of the JobExecution context for convenience in late binding to step scoped beans. If you want to write to it you would need to grab the StepExecution directly and locate the JobExecution and its context. E.g. a late binding expression #{stepExecution.jobExecution.executionContext}, or in a StepExecutionListener.

        Comment


        • #5
          I think I have found a bug. I have been digging into this and have come across the fact that the JdbcExecutionContextDao is not saving information to the BATCH_JOB_EXECUTION_CONTEXT table. The only way I could find to save information to that table is to use the JdbcExecutionContextDao directly.

          Code:
          String jobName = chunkContext.getStepContext().getJobName();
                  JobExecution jobExecution = getJobExecutionContext(jobName, chunkContext.getStepContext().getJobParameters());
                  
                  ExecutionContext ec = jobExecution.getExecutionContext();
                  ec.put("TEST_ATTRIBUTE", "Test Attribute");
                  
                  LOGGER.debug("Is Dirty = " + ec.isDirty());
                  LOGGER.debug("Is Empty = " + ec.isEmpty());
                  
                  ExecutionContext ec2 = new ExecutionContext(ec);
                  jobExecution.setExecutionContext(ec2);
                  getEcd().updateExecutionContext(jobExecution);
          What gets saved in the Table is this:
          Code:
          JOB_EXECUTION_ID	SHORT_CONTEXT	SERIALIZED_CONTEXT
          199	{"map":""}	(null)
          What should have been saved is this:
          Code:
          UPDATE BATCH_JOB_EXECUTION_CONTEXT SET SHORT_CONTEXT = '{"map":{"entry":{"string":["TEST_ATTRIBUTE","Test Attribute"]}}}', SERIALIZED_CONTEXT = null WHERE JOB_EXECUTION_ID = 199
          What the Logs are saying is this:
          Code:
          09:52:10,426 DEBUG main JdbcTemplate:790 - Executing prepared SQL update
          09:52:10,426 DEBUG main JdbcTemplate:574 - Executing prepared SQL statement [UPDATE BATCH_JOB_EXECUTION_CONTEXT SET SHORT_CONTEXT = ?, SERIALIZED_CONTEXT = ? WHERE JOB_EXECUTION_ID = ?]
          09:52:10,426 TRACE main TransactionSynchronizationManager:139 - Retrieved value [org.springframework.jdbc.datasource.ConnectionHolder@6d4c4d61] for key [org.apache.commons.dbcp.BasicDataSource@5bf0cf51] bound to thread [main]
          09:52:10,427 TRACE main TransactionSynchronizationManager:139 - Retrieved value [org.springframework.jdbc.datasource.ConnectionHolder@6d4c4d61] for key [org.apache.commons.dbcp.BasicDataSource@5bf0cf51] bound to thread [main]
          09:52:10,427 DEBUG main JdbcExecutionContextDao:182 - PreparedStatement Class = org.apache.commons.dbcp.DelegatingPreparedStatement
          09:52:10,427 DEBUG main JdbcExecutionContextDao:183 - Short Context '{"map":{"entry":{"string":["TEST_ATTRIBUTE","Test Attribute"]}}}'
          09:52:10,427 DEBUG main JdbcExecutionContextDao:184 - Long Context 'null'
          09:52:10,427 DEBUG main JdbcExecutionContextDao:185 - Execution ID '199'
          09:52:10,428 DEBUG main JdbcExecutionContextDao:197 - This is the SQL statement that should be ran:
          09:52:10,428 DEBUG main JdbcExecutionContextDao:198 - com.mysql.jdbc.JDBC4PreparedStatement@6bf51e5c: UPDATE BATCH_JOB_EXECUTION_CONTEXT SET SHORT_CONTEXT = '{"map":{"entry":{"string":["TEST_ATTRIBUTE","Test Attribute"]}}}', SERIALIZED_CONTEXT = null WHERE JOB_EXECUTION_ID = 199
          09:52:10,429 DEBUG main JdbcTemplate:800 - SQL update affected 1 rows
          It looks like the Prepared Statement is creating the correct SQL, however, the data is not being saved. There seems to be a mismatch between what SQL should be executed and what is actually executed. Either that, or my configs are wrong.

          data-source-context.xml
          HTML Code:
          <beans ....>
          	
          	<!-- <bean id="placeholderProperties" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer" depends-on="environment">
          		<property name="location" value="classpath:batch-mysql.properties" />
          		<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
          		<property name="ignoreUnresolvablePlaceholders" value="true" />
          		<property name="order" value="1" />
          	</bean> -->
          	
          	<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
          		<property name="driverClassName" value="com.mysql.jdbc.Driver" />
          		<property name="url" value="jdbc:mysql://localhost:3306/feed_sandbox" />
          		<property name="username" value="spring_batch" />
          		<property name="password" value="spring_batch" />
          	</bean>
          	
          	<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager" lazy-init="true">
          		<property name="dataSource" ref="dataSource" />
          	</bean>
          	
          	<bean id="lobHandler" class="org.springframework.jdbc.support.lob.DefaultLobHandler" />
          	
          	<bean id="incrementerParent" class="org.springframework.jdbc.support.incrementer.MySQLMaxValueIncrementer">
          		<property name="dataSource" ref="dataSource" />
          		<property name="incrementerName" value="ID" />
          		<property name="columnName" value="ID"/>
          	</bean>
          
          	<bean id="jdbcTemplate" class="org.springframework.jdbc.core.simple.SimpleJdbcTemplate">
          		<constructor-arg index="0" ref="dataSource"/>
          	</bean>
          	
          	<bean id="jobInstanceDao" class="org.springframework.batch.core.repository.dao.JdbcJobInstanceDao">
          		<property name="jdbcTemplate" ref="jdbcTemplate"/>
          		<property name="jobIncrementer" ref="incrementerParent" />
          	</bean>
          	
          	<bean id="jobExecutionDao" class="org.springframework.batch.core.repository.dao.JdbcJobExecutionDao">
          		<property name="jdbcTemplate" ref="jdbcTemplate"/>
          		<property name="jobExecutionIncrementer" ref="incrementerParent"/>
          	</bean>
          	
          	<bean id="stepExecutionDao" class="org.springframework.batch.core.repository.dao.JdbcStepExecutionDao">
          		<property name="jdbcTemplate" ref="jdbcTemplate"/>
          		<property name="stepExecutionIncrementer" ref="incrementerParent"/>
          	</bean>
          	
          	<bean id="executionContextDao" class="org.springframework.batch.core.repository.dao.JdbcExecutionContextDao">
          		<property name="jdbcTemplate" ref="jdbcTemplate"/>
          	</bean>
          
          </beans>
          simple-job-launcher-context.xml
          HTML Code:
          <beans ....>
          
          	<import resource="data-source-context.xml" />
          
          	<bean id="jobRepository" 
          		class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean"
          		p:databaseType="mysql" 
          		p:dataSource-ref="dataSource" 
          		p:transactionManager-ref="transactionManager"/>
          		
          	<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
          		<property name="jobRepository" ref="jobRepository" />
          	</bean>
          
          	<bean id="mapJobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" 
          		p:transactionManager-ref="transactionManager" lazy-init="true" autowire-candidate="false" />
          
          	<bean id="jobExplorer" class="org.springframework.batch.core.explore.support.SimpleJobExplorer">
          		<constructor-arg index="0" ref="jobInstanceDao"/>
          		<constructor-arg index="1" ref="jobExecutionDao"/>
          		<constructor-arg index="2" ref="stepExecutionDao"/>
          		<constructor-arg index="3" ref="executionContextDao"/>
          	</bean>
          	
          	<bean id="simpleJob" class="org.springframework.batch.core.job.SimpleJob" abstract="true">
          		<property name="jobRepository" ref="jobRepository" />
          		<property name="restartable" value="true" />
          	</bean>
          
          	<bean id="taskletStep" class="org.springframework.batch.core.step.tasklet.TaskletStep" abstract="true">
          		<property name="jobRepository" ref="jobRepository" />
          		<property name="allowStartIfComplete" value="true" />
          	</bean>
          
          	<bean id="simpleStep" class="org.springframework.batch.core.step.item.SimpleStepFactoryBean"
          		abstract="true">
          		<property name="transactionManager" ref="transactionManager" />
          		<property name="jobRepository" ref="jobRepository" />
          		<property name="startLimit" value="100" />
          		<property name="commitInterval" value="1" />
          	</bean>
          	
          	<bean id="skipLimitStep" class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"
          		parent="simpleStep" abstract="true">
          		<property name="skipLimit" value="0" />
          	</bean>
          
          	<bean id="customEditorConfigurer" class="org.springframework.beans.factory.config.CustomEditorConfigurer">
          		<property name="customEditors">
          			<map>
          				<entry key="int[]">
          					<bean class="org.springframework.batch.support.IntArrayPropertyEditor" />
          				</entry>
          				<entry key="org.springframework.batch.item.file.transform.Range[]">
          					<bean class="org.springframework.batch.item.file.transform.RangeArrayPropertyEditor" />
          				</entry>
          				<entry key="java.util.Date">
          					<bean class="org.springframework.beans.propertyeditors.CustomDateEditor">
          						<constructor-arg>
          							<bean class="java.text.SimpleDateFormat">
          								<constructor-arg value="yyyyMMdd" />
          							</bean>
          						</constructor-arg>
          						<constructor-arg value="false" />
          					</bean>
          				</entry>
          			</map>
          		</property>
          	</bean>
          
          	<bean id="jobRegistry" class="org.springframework.batch.core.configuration.support.MapJobRegistry" />
          
          	<bean class="org.springframework.batch.core.configuration.support.AutomaticJobRegistrar">
          		<property name="applicationContextFactories">
          			<bean class="org.springframework.batch.core.configuration.support.ClassPathXmlApplicationContextFactory">
          				<property name="resource" value="classpath:/JobContext.xml" />
          			</bean>
          		</property>
          		<property name="jobLoader">
          			<bean class="org.springframework.batch.core.configuration.support.DefaultJobLoader">
          				<property name="jobRegistry" ref="jobRegistry" />
          			</bean>
          		</property>
          	</bean>
          	
          </beans>

          Comment


          • #6
            I forgot to mention I am using MySQL as the DB.

            Comment


            • #7
              I assume you added those log statements to your own version of JdbcExecutionContextDao (they aren't in the framework)? Since it works in the unit tests, I assume there is something else wrong with your set up or your dao implementation. What are you using exactly.

              P.S. there's a factory bean for the JobExplorer so you don't have to create all the daos manually.

              Comment


              • #8
                Correct, I copied and pasted JdbcExecutionContextDao into my own module. It is the exact same code with just a few extra logging statements.

                The configs I use are in the previous post. This is a very simple test. I have a "PrintTasklet" which prints out the Job Parameters and the Context Attributes to the log files. The Tasklet will place an attribute on the Job Context. There isn't much to this.

                I would agree there may be something with my setup. The config files i use are the ones in my previous post (except for the Job Config). Does anything look out of place there?

                Comment


                • #9
                  I looked at the Unit Tests and Configs used for the Unit Tests and the only difference I see is the Unit Test uses HSQLDB and I am using MySQL. Will download the source code and do a build. See if there is a problem with using MySQL.

                  Comment


                  • #10
                    Looks like there is an issue using MySQL as the DB with the Unit Tests. I downloaded the source code and did a 'mvn install'. I imported the project into Eclipse and ran the JdbcJobExecutionDaoTests (with HSQLDB) and everything works fine. I then changed over the datasource to be MySQL and the Tests now fail. Here are the changes I made:

                    I added the following to the dependencies of spring-batch-core/pom.xml
                    HTML Code:
                    <dependency>
                    			<groupId>mysql</groupId>
                    			<artifactId>mysql-connector-java</artifactId>
                    			<version>5.1.14</version>
                    		</dependency>
                    Since, I already have the DB in place, I changed the datasource-context.xml file to look like this:

                    HTML Code:
                    <?xml version="1.0" encoding="UTF-8"?>
                    <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                    	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">
                    	<!-- bean class="test.jdbc.datasource.DataSourceInitializer">
                    		<property name="dataSource" ref="dataSource" />
                    		<property name="initScripts">
                    			<list>
                    				<value>org/springframework/batch/core/schema-drop-hsqldb.sql</value>
                    				<value>org/springframework/batch/core/schema-hsqldb.sql</value>
                    				<value>foo.sql</value>
                    			</list>
                    		</property>
                    	</bean -->
                    	
                    	<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
                    		<property name="driverClassName" value="com.mysql.jdbc.Driver" />
                    		<property name="url" value="jdbc:mysql://localhost:3306/sandbox" />
                    		<property name="username" value="spring_batch" />
                    		<property name="password" value="spring_batch" />
                    	</bean>
                    	<!-- bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
                    		<property name="driverClassName" value="org.hsqldb.jdbcDriver" />
                    		<property name="url" value="jdbc:hsqldb:mem:testdb;sql.enforce_strict_size=true" />
                    		<property name="username" value="sa" />
                    		<property name="password" value="" />
                    	</bean -->
                    	<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
                    		<property name="dataSource" ref="dataSource" />
                    	</bean>
                    	
                    	<bean id="incrementerParent" class="org.springframework.jdbc.support.incrementer.MySQLMaxValueIncrementer">
                    		<property name="dataSource" ref="dataSource" />
                    		<property name="incrementerName" value="ID" />
                    		<property name="columnName" value="ID"/>
                    	</bean>
                    	
                    	<!-- bean id="incrementerParent" class="org.springframework.jdbc.support.incrementer.HsqlMaxValueIncrementer"
                    		abstract="true">
                    		<property name="dataSource" ref="dataSource" />
                    		<property name="columnName" value="ID" />
                    	</bean -->
                    	
                    	<bean id="jobIncrementer" parent="incrementerParent">
                    		<property name="incrementerName" value="BATCH_JOB_SEQ" />
                    	</bean>
                    	<bean id="jobExecutionIncrementer" parent="incrementerParent">
                    		<property name="incrementerName" value="BATCH_JOB_EXECUTION_SEQ" />
                    	</bean>
                    	<bean id="stepIncrementer" parent="incrementerParent">
                    		<property name="incrementerName" value="BATCH_STEP_SEQ" />
                    	</bean>
                    	<bean id="stepExecutionIncrementer" parent="incrementerParent">
                    		<property name="incrementerName" value="BATCH_STEP_EXECUTION_SEQ" />
                    	</bean>
                    </beans>
                    I get the following errors:
                    HTML Code:
                    java.lang.AssertionError: expected:<5> but was:<0>
                    ....
                    	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.testFindRunningExecutions(AbstractJobExecutionDaoTests.java:202)
                    ....
                    
                    java.lang.IllegalStateException: There must be at most one latest job execution
                    	at org.springframework.util.Assert.state(Assert.java:384)
                    	at org.springframework.batch.core.repository.dao.JdbcJobExecutionDao.getLastJobExecution(JdbcJobExecutionDao.java:226)
                    	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.testGetLastExecution(AbstractJobExecutionDaoTests.java:158)
                    ....
                    
                    java.lang.AssertionError: expected:<Wed Jan 26 12:49:40 PST 2011> but was:<2011-01-26 12:49:40.0>
                    ....
                    	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.assertExecutionsAreEqual(AbstractJobExecutionDaoTests.java:342)
                    	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.testSaveAndFind(AbstractJobExecutionDaoTests.java:71)
                    ....
                    
                    java.lang.AssertionError: expected:<Wed Dec 31 16:00:00 PST 1969> but was:<1969-12-31 16:00:00.0>
                    ....
                    	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.assertExecutionsAreEqual(AbstractJobExecutionDaoTests.java:345)
                    	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.testFindExecutionsOrdering(AbstractJobExecutionDaoTests.java:94)
                    ....
                    
                    java.lang.AssertionError: expected:<Wed Jan 26 12:49:40 PST 2011> but was:<2011-01-26 12:49:40.0>
                    ....
                    	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.assertExecutionsAreEqual(AbstractJobExecutionDaoTests.java:345)
                    	at org.springframework.batch.core.repository.dao.AbstractJobExecutionDaoTests.testUpdateExecution(AbstractJobExecutionDaoTests.java:140)
                    ....
                    The timestamp warnings are one thing, not a big deal. But the first 2 errors listed are a little disconcerting. Also, I noticed there is no test for saving information for the JobExecutionContext. So I updated the testUpdateExecution to look like:

                    Code:
                    ublic void testUpdateExecution() {
                    		execution.setStatus(BatchStatus.STARTED);
                    		dao.saveJobExecution(execution);
                    
                    		execution.setLastUpdated(new Date(0));
                    		execution.setStatus(BatchStatus.COMPLETED);
                    		execution.getExecutionContext().put("TEST_ATTR", "Test Attr");
                    		dao.updateJobExecution(execution);
                    
                    		JobExecution updated = dao.findJobExecutions(jobInstance).get(0);
                    		assertEquals(execution, updated);
                    		assertEquals(BatchStatus.COMPLETED, updated.getStatus());
                    		assertExecutionsAreEqual(execution, updated);
                    	}
                    I saw nothing in the DB saved.
                    Last edited by pminearo; Jan 26th, 2011, 03:16 PM.

                    Comment


                    • #11
                      You need to add
                      Code:
                      jobRepository.updateExecutionContext(execution)
                      to get your test to pass.

                      The core tests are not supposed to pass with MySQL. If you try the integration tests or the samples they should work (but we don't test it very often). I just tried the core integration tests (spring-batch-core-tests project) and it worked with a couple of config tweaks (which I have now pushed to github, so if you try it yourself get the latest version).

                      So I'm still not sure what is the root cause of your original problem. What data are you trying to persist in the context? Maybe there is a serialization error that we haven't seen yet?

                      Comment

                      Working...
                      X