Announcement Announcement Module
Collapse
No announcement yet.
Overriding write(List<? extends T>) method of StaxEventItemWriter Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Overriding write(List<? extends T>) method of StaxEventItemWriter

    Hi everybody,

    I'm using Spring Batch 2.1.8 and I've a question... Maybe a silly (very silly) question... :-)

    Is it advisable to override the "write(List<? extends T>)" method of a StaxEventItemWriter in order to apply some business logic in composing an XML instance?

    I'll try to explain...

    This is my scenario (it's a simplification):
    I have a DB with some records (let's say "record_A", "record_B" and "record_C").
    FOR EACH of these records I have to write THREE XML nodes; the node values are based on the values of record_A, record_B and record_C.
    Each record is mapped to a domain object (using a row mapper): "MyRecordObject".
    MyRecordObject contains JAXB annotation so that it can represent a single XML node.

    The final XML instance has to be:

    <data>
    <!-- from A -->
    <record value="A.0">
    <record value="A.1">
    <record value="A.2">
    <!-- from B -->
    <record value="B.0">
    <record value="B.1">
    <record value="B.2">
    <!-- from C -->
    <record value="C.0">
    <record value="C.1">
    <record value="C.2">
    </data>

    and not only

    <data>
    <record value="A">
    <record value="B">
    <record value="C">
    </data>

    The ItemReader only knows A, B and C values and the result is a List<MyRecordObject> with 3 items only.

    Currently I override the "write(List<? extends T>)" method of a StaxEventItemWriter<MyRecordObject> in order to replace the original List<MyRecordObject> received by the write method with a new extended List<MyRecordObject>

    This is a simplificated snippet of my overriding method:

    Code:
    @Override
    public void write(List<? extends MyRecordObject> oldRecords) throws XmlMappingException, Exception {
    
    	String[] suffixes = {".0", ".1", ".2"}; // i put here a public final property
    	
    	ArrayList<MyRecordObject> newRecords = new ArrayList<MyRecordObject>(); // a new List<MyRecordObject>
    	
    	for(MyRecordObject oldRecord : oldRecords) {
    		
    		for(String suffix : suffixes) {
    		
    			MyRecordObject newRecord = new MyRecordObject();
    			
    			newRecord.setValue(oldRecord.getValue() + suffix);
    			
    			newRecords.add(newRecord);
    		}
    	}
    	
    	super.write(newRecords); // write to XML the new item list
    }
    This way works... But it seems to me this is the wrong place to apply business logic...

    I've tried to use an ItemProcessor<MyRecordObject, MyRecordObjectList> in order to apply elsewhere the business logic and to extend the number of items for each original item; but the problem is that the stax writer is <MyRecordObject> and not <MyRecordObjectList>. And I cannot add a global def for MyRecordObjectList in the XML schema ( it's not mine! ;-) )...

    Is there a way to "intercept" the original List<MyRecordObject> before it's received by the write method of the stax writer?
    Can I use the ItemWriteListener's beforeWrite(List<? extends S> items) as a kind of "chunk-processor"?

    I know this is not a "tech" question about Spring Batch Framework, but I'm not a Spring Batch guru and any suggestion/opinion will be absolutely appreciated. :-)

    Thank you!

    Emilio

  • #2
    Overriding write(List&lt;? extends T&gt method of StaxEventItemWriter

    Maybie I misunderstood your point, but why don't you create a custom ItemProcessor like the one below and leave your itemwriter as it is ?

    HTML Code:
    import java.util.ArrayList;
    import java.util.List;
    
    import org.springframework.batch.item.ItemProcessor;
    
    public class MyRecordItemProcessor implements ItemProcessor<MyRecord, List<MyRecord>> {
    
    	@Override
    	public List<MyRecord> process(MyRecord item) throws Exception {
    		String[] suffixes = {".0", ".1", ".2"}; // i put here a public final property
    		
    		ArrayList<MyRecord> newRecords = new ArrayList<MyRecord>(); // a new List<MyRecord>
    		
    		for(String suffix : suffixes) {
    			
    			MyRecord newRecord = new MyRecord();
    				
    			newRecord.setValue(item.getValue() + suffix);
    				
    			newRecords.add(newRecord);
    		}
    		return newRecords;
    	}
    
    }

    Comment


    • #3
      Hi, psoares.

      Thank you for your reply.

      I already tried what you suggest and I got an error. :-(

      Code:
      SEVERE: Encountered an error executing the step
      java.lang.IllegalStateException: Marshaller must support the class of the marshalled object
      	
      	
      	**** at org.springframework.util.Assert.state(Assert.java:384) ****
      	
      	
      	at org.springframework.batch.item.xml.StaxEventItemWriter.write(StaxEventItemWriter.java:571)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      	at $Proxy11.write(Unknown Source)
      	at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:171)
      	at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:150)
      	at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:269)
      	at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:194)
      	at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:74)
      	at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386)
      	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
      	at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264)
      	at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76)
      	at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367)
      	at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214)
      	at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
      	at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250)
      	at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195)
      	at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
      	at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
      	at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
      	at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
      	at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
      	at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:91)
      	at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:90)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      I've tried to figure out why I get that error and I suppose the marshaller simply cannot marshal the list that the processor returns for each read item...

      Currently only MyRecord class is bound to the marshaller.

      If you take a glance at the source code of the stax writer, you'll see:

      Code:
      /**
      * Write the value objects and flush them to the file.
      * 
      * @param items the value object
      * @throws IOException
      * @throws XmlMappingException
      */
      public void write(List<? extends T> items) throws XmlMappingException, Exception {
      
      	currentRecordCount += items.size();
      
      	for (Object object : items) {
      		Assert.state(marshaller.supports(object.getClass()),
      				"Marshaller must support the class of the marshalled object");
      		Result result = StaxUtils.getResult(eventWriter);
      		marshaller.marshal(object, result );
      	}
      	try {
      		eventWriter.flush();
      	}
      	catch (XMLStreamException e) {
      		throw new WriteFailedException("Failed to flush the events", e);
      	}
      
      }
      The supports method of the marshaller is something like this:

      Code:
      public boolean supports(Class clazz) {
      	
      	return clazz.getAnnotation(XmlRootElement.class) != null || JAXBElement.class.isAssignableFrom(clazz);
      }
      I tried to introduce a class MyRecordList to include the list as a field ( with @XmlElement(name="record") ), but I only added an extra level to handle...



      Thank you again.

      Emilio

      Comment


      • #4
        Try this

        Hi Emilio,
        It can be achieved with some work with a 2-step flow.
        First, check this part of the documentation.

        So, following this, I took a concrete example with a simple object called "Author", having 2 properties (first and last name).
        I define an XSD schema for it, assuming you already have one for your object, but this is just for the demo :
        HTML Code:
        <?xml version="1.0" encoding="UTF-8"?>
        <xsd:schema targetNamespace="http://com.authors/batch"
        	xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://com.authors/batch"
        	elementFormDefault="qualified" attributeFormDefault="unqualified">
        	<xsd:element name="Authors">
        		<xsd:complexType>
        			<xsd:sequence>
        				<xsd:element ref="Author" maxOccurs="unbounded"/>
        			</xsd:sequence>
        		</xsd:complexType>
        	</xsd:element>
        	<xsd:element name="Author">
        		<xsd:complexType>
        			<xsd:sequence>
        				<xsd:element name="FirstName" type="xsd:string" />
        				<xsd:element name="LastName" type="xsd:string" />
        			</xsd:sequence>
        		</xsd:complexType>
        	</xsd:element>
        
        </xsd:schema>
        Also, I will take a sample input file, like this :
        HTML Code:
        <?xml version="1.0" encoding="UTF-8"?>
        <Authors xmlns="http://com.authors/batch">
        	<Author>
        		<FirstName>Chopin</FirstName>
        		<LastName>Kate</LastName>
        	</Author>
        	<Author>
        		<FirstName>London</FirstName>
        		<LastName>Jack</LastName>
        	</Author>
        </Authors>
        You then generate the classes from the XSD. They will be in package com.authors.batch.domain in my example.
        Assuming that is done, the beans that will read my xml file is defined like this :
        HTML Code:
        <oxm:jaxb2-marshaller id="authorMarshaller"
        		contextPath="com.authors.batch.domain">
        </oxm:jaxb2-marshaller>
        
        <bean id="authorReader" class="org.springframework.batch.item.xml.StaxEventItemReader">
        		<property name="fragmentRootElementName" value="Author" />
        		<property name="resource" value="file:src/main/resources/authors.xml"/>
        		<property name="unmarshaller" ref="authorMarshaller" />
        </bean>
        Then I need a processor to add the suffixes and create new xml records :
        AuthorItemProcessor.java
        Code:
        package com.authors.batch;
        
        import java.util.ArrayList;
        import java.util.List;
        
        import org.springframework.batch.item.ItemProcessor;
        
        import com.authors.batch.domain.Author;
        
        public class AuthorItemProcessor implements ItemProcessor<Author, List<Author>> {
        
        	@Override
        	public List<Author> process(Author author) throws Exception {
        		String[] suffixes = new String[]{".1", ".2", ".3"};
        		List<Author> output = new ArrayList<Author>();
        		
        		for (String suffix : suffixes) {
        			Author newAuthor = new Author();
        			newAuthor.setFirstName(author.getFirstName() + suffix);
        			newAuthor.setLastName(author.getLastName());
        			output.add(newAuthor);
        		}
        		return output;
        	}
        
        }
        Just declare this as a bean on your spring config file :
        HTML Code:
        <bean id="authorProcessor" class="com.authors.batch.AuthorItemProcessor"/>
        Now, once we generate the lists for each record, we're going to merge the authors they contain in a single list, and save them for the next step of the batch flow.
        This is done through this class :

        SavingItemWriter.java
        Code:
        package com.authors.batch;
        
        import java.util.ArrayList;
        import java.util.List;
        
        import org.springframework.batch.core.StepExecution;
        import org.springframework.batch.core.annotation.BeforeStep;
        import org.springframework.batch.item.ExecutionContext;
        import org.springframework.batch.item.ItemWriter;
        
        import com.authors.batch.domain.Author;
        
        public class SavingItemWriter implements ItemWriter<List<Author>> {
        	
            private StepExecution stepExecution;
        
            @BeforeStep
            public void saveStepExecution(StepExecution stepExecution) {
                this.stepExecution = stepExecution;
            }
        
        	@Override
        	public void write(List<? extends List<Author>> items) throws Exception {
        		ExecutionContext stepContext = this.stepExecution.getExecutionContext();
        		@SuppressWarnings("unchecked")
        		List<Author> currentItems = (List<Author>)stepContext.get("itemList");
        		if (currentItems != null) {
        			List<Author> newItems = new ArrayList<Author>();
        			newItems.addAll(currentItems);
        			for (List<Author> list : items) {
        				for (Author author : list) {
        					newItems.add(author);
        				}
        			}
        			stepContext.put("itemList", newItems);
        		} else {
        			currentItems = new ArrayList<Author>();
        			for (List<Author> list : items) {
        				for (Author author : list) {
        					currentItems.add(author);
        				}
        			}
        			stepContext.put("itemList", currentItems);
        		}
        	}
        
        }
        Add this as a bean in your spring config file :
        HTML Code:
        <bean id="authorsWriter" class="com.authors.batch.SavingItemWriter"/>
        It's time for STEP 2 now !
        The class in charge of retrieving the list of authors previously saved is as follows (I just mixed the code of org.springframework.batch.item.support.ListItemRea der<T> and the code from chapter 11.8 of the documentation (link above):

        RetrievingItemReader.java
        Code:
        package com.authors.batch;
        
        import java.util.ArrayList;
        import java.util.List;
        
        import org.springframework.aop.support.AopUtils;
        import org.springframework.batch.core.JobExecution;
        import org.springframework.batch.core.StepExecution;
        import org.springframework.batch.core.annotation.BeforeStep;
        import org.springframework.batch.item.ExecutionContext;
        import org.springframework.batch.item.ItemReader;
        
        import com.authors.batch.domain.Author;
        
        public class RetrievingItemReader<T> implements ItemReader<T>{
        	
        
        	private List<T> list;
        
        	public RetrievingItemReader(List<T> list) {
        		// If it is a proxy we assume it knows how to deal with its own state.
        		// (It's probably transaction aware.)
        		if (AopUtils.isAopProxy(list)) {
        			this.list = list;
        		}
        		else {
        			this.list = new ArrayList<T>(list);
        		}
        	}
        
        	public T read() {
        		if (!list.isEmpty()) {
        			return list.remove(0);
        		}
        		return null;
        	}
        	
        	
        	@SuppressWarnings("unchecked")
        	@BeforeStep
            public void retrieveInterstepData(StepExecution stepExecution) {
                JobExecution jobExecution = stepExecution.getJobExecution();
                ExecutionContext jobContext = jobExecution.getExecutionContext();
                this.list = (List<T>)jobContext.get("itemList");
            }
        	
        	
        }
        I wire this to the spring configuration like this :
        HTML Code:
        <bean id="authorsReader" class="com.authors.batch.RetrievingItemReader" scope="step">
        	<constructor-arg name="list" value="#{jobExecutionContext['itemList']}"/>
        </bean>
        Then I am able to write that to xml using a StaxEventWriter, configured this way :
        HTML Code:
        <bean id="authorWriter" class="org.springframework.batch.item.xml.StaxEventItemWriter">
        	<property name="rootTagName" value="{http://com.authors/batch}:Authors"/>
        	<property name="overwriteOutput" value="true"/>
        	<property name="resource" value="file:./output-authors.xml"/>
        	<property name="marshaller" ref="authorMarshaller"/>
        </bean>
        The rest of the spring configuration is like this :
        HTML Code:
                <bean id="promotionListener" class="org.springframework.batch.core.listener.ExecutionContextPromotionListener">
        		<property name="keys" value="itemList"/>
        	</bean>
        	
        	<batch:job id="job1">
        		<batch:step id="step1" next="step2">
        			<batch:tasklet transaction-manager="transactionManager"	start-limit="1">
        				<batch:chunk reader="authorReader" processor="authorProcessor" writer="authorsWriter" commit-interval="1" />
        			</batch:tasklet>
        			<batch:listeners>
        				<batch:listener ref="promotionListener"/>
        			</batch:listeners>
        		</batch:step>
        		<batch:step id="step2">
        			<batch:tasklet>
        				<batch:chunk reader="authorsReader" writer="authorWriter" commit-interval="1"/>
        			</batch:tasklet>
        		</batch:step>
        	</batch:job>
        And when I run this, I get an xml file containing this :
        HTML Code:
        <?xml version="1.0" encoding="UTF-8"?>
        <Authors xmlns="http://com.authors/batch">
        	<Author>
        		<FirstName>Chopin.1</FirstName>
        		<LastName>Kate</LastName>
        	</Author>
        	<Author>
        		<FirstName>Chopin.2</FirstName>
        		<LastName>Kate</LastName>
        	</Author>
        	<Author>
        		<FirstName>Chopin.3</FirstName>
        		<LastName>Kate</LastName>
        	</Author>
        	<Author>
        		<FirstName>London.1</FirstName>
        		<LastName>Jack</LastName>
        	</Author>
        	<Author>
        		<FirstName>London.2</FirstName>
        		<LastName>Jack</LastName>
        	</Author>
        	<Author>
        		<FirstName>London.3</FirstName>
        		<LastName>Jack</LastName>
        	</Author>
        </Authors>
        That's it...
        It's probably not the simplest solution, nor the cleaner code (I'm not a jaxb pro but I guess the same can be achieved using some xsd tricks), but at least I didn't touch the initial xsd.

        I hope it helps.

        P.S.: Spring batch should definetely make things like this easier. If there is an easy solution, I'll be very interested in seeing what it looks like.
        Last edited by psoares; Sep 30th, 2011, 02:25 AM. Reason: Added namespace to the staxItemWriter root tag

        Comment


        • #5
          Hi, psoares!

          Thank you very much for your detailed reply! I think you gave me a very helpful tip. You're UBUNTU!

          I missed the section about Passing Data to Future Steps...

          I will let you know as soon as I have done.

          Thank you again. Bye!

          Emilio

          Comment


          • #6
            Simpler solution

            I knew this couldn't be that difficult . I know groovy is great for building markup, so I decided to give it a try, and I managed to achieve the same results in a single step using this spring config file :
            HTML Code:
            <?xml version="1.0" encoding="UTF-8"?>
            <beans xmlns="http://www.springframework.org/schema/beans"
            	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            	xmlns:batch="http://www.springframework.org/schema/batch"
            	xmlns:oxm="http://www.springframework.org/schema/oxm"
            	xmlns:int="http://www.springframework.org/schema/integration"
            	xmlns:int-stream="http://www.springframework.org/schema/integration/stream"
            	xmlns:context="http://www.springframework.org/schema/context"
            	xmlns:task="http://www.springframework.org/schema/task"
            	xmlns:lang="http://www.springframework.org/schema/lang"
            	xsi:schemaLocation="http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang-3.0.xsd
            		http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration-2.0.xsd
            		http://www.springframework.org/schema/oxm http://www.springframework.org/schema/oxm/spring-oxm-3.0.xsd
            		http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd
            		http://www.springframework.org/schema/integration/stream http://www.springframework.org/schema/integration/stream/spring-integration-stream-2.0.xsd
            		http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
            		http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
            		http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd">
            
            	
            	<oxm:jaxb2-marshaller id="authorMarshaller"
            		contextPath="com.authors.batch.domain">
            	</oxm:jaxb2-marshaller>
            	
            	<bean id="authorReader" class="org.springframework.batch.item.xml.StaxEventItemReader">
            		<property name="fragmentRootElementName" value="Author" />
            		<property name="resource" value="file:src/main/resources/authors.xml"/>
            		<property name="unmarshaller" ref="authorMarshaller" />
            	</bean>
            	
            	
            	<lang:groovy id="authorsWriter">
            		<lang:inline-script>
            			<![CDATA[
            			import org.springframework.batch.item.ItemWriter
            			import com.authors.batch.domain.Author
            			import org.springframework.batch.core.StepExecution
            			import java.io.File
            			import groovy.xml.*
            
            
            			class MultiRecordWriter implements ItemWriter<List<Author>>{
            				StepExecution stepExecution;
            				def suffixes = ['.1','.2','.3']
            				def writer = new FileWriter("groovyOutput.xml")
            				def xml = new MarkupBuilder(writer)
            				
            				void write(List<? extends List<Author>> items) throws Exception {
            					xml.Authors(){
            						items.each { item ->
            							suffixes.each { suffix ->
            								xml.Author(){
            									FirstName(item.firstName + suffix)
            									LastName(item.lastName)
            								}
            							}
            						}
            					}
            				}
            			}
            			]]>
            			
            		</lang:inline-script>
            	</lang:groovy>
            	
            	
            	<batch:job id="job1">
            		<batch:step id="step1">
            			<batch:tasklet transaction-manager="transactionManager"	start-limit="1">
            				<batch:chunk reader="authorReader" writer="authorsWriter" commit-interval="1" />
            			</batch:tasklet>
            		</batch:step>
            	</batch:job>
            
            </beans>
            All you have to do is to add this dependency to your pom.xml (if you're using maven):
            HTML Code:
                  <dependencies>
                           ...
                           <dependency>
            			<groupId>org.codehaus.groovy</groupId>
            			<artifactId>groovy</artifactId>
            			<version>1.8.2</version>
            			<scope>runtime</scope>
            		</dependency>
                  </dependencies>
            And here is my output :
            groovyOutput.xml :
            HTML Code:
            <Authors>
              <Author>
                <FirstName>Chopin.1</FirstName>
                <LastName>Kate</LastName>
              </Author>
              <Author>
                <FirstName>Chopin.2</FirstName>
                <LastName>Kate</LastName>
              </Author>
              <Author>
                <FirstName>Chopin.3</FirstName>
                <LastName>Kate</LastName>
              </Author>
            </Authors>
            <Authors>
              <Author>
                <FirstName>London.1</FirstName>
                <LastName>Jack</LastName>
              </Author>
              <Author>
                <FirstName>London.2</FirstName>
                <LastName>Jack</LastName>
              </Author>
              <Author>
                <FirstName>London.3</FirstName>
                <LastName>Jack</LastName>
              </Author>
            </Authors>
            And... yeah... I agree this is what you wanted to avoid (having the logic in the writer)... but still :-)
            Last edited by psoares; Sep 30th, 2011, 06:48 AM.

            Comment


            • #7
              would be interesting to know if using inline Groovy has any impact on the performances.

              Comment


              • #8
                No way out...

                Hi psoares,

                I was a little busy and it took me some time to re-write my app on the basis of what you suggested.

                Well, it's strange, but the groovy writer in my app doesn't work as expected (the only difference between mine and yours is that I use, to define the writer, the script-source attribute: <lang:groovy id="idOfTheWriter" script-source="classpath:TheWriter.groovy"/> ).

                As you did show, the write method of the groovy writer writes the root element as well; but I noticed that this works only if the number of items (total number!) is less than or equal to the commit interval (chunk size). I mean: the write method is called (total items / commit interval) times. I have hundreds of read items and what happens is that for each chunk the root element is injected. The result is a non-well-formed XML.

                Ouf!

                I think the root element scope should be opened before the injection of the fragments (a fragment for each read item) and closed after the last chunk. If I'm not wrong this is what a stax writer, more or less, does.

                Well, at the end I think that, if you don't want to have any logic in the writer (e.g. overriding the write method of a stax writer), the best way is the two-step way... But you have to save a big list in memory before being able to feed the writer.

                Mmm... I have to admit that I feel a little bit stupid. :-(
                This problem is trivial outside SB and SB is putting me off a little.

                Ok, maybe I could have a better schema, but I haven't. I think that If you cannot create your domain logic from scratch, sometimes it could be difficult to tailor SB to an existing domain.

                I could write an intermediate XML and then transform it with XSL...

                @arno: my little app (~10K fragments for each file) doesn't suffer from having groovy components (I have four parallel writers). I tested inline-coded writers as well. But maybe my numbers are too trivial to say something significant about SB/Groovy performances. :-)


                Have a nice day!

                Emilio

                Comment


                • #9
                  This is what I get with the two-step way if the number of items is high...

                  Any idea?

                  I'm so sorry for all these problems...

                  I know I have a bad domain. Not SB's fault.

                  Code:
                  Caused by: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data too long for column 'SERIALIZED_CONTEXT' at row 1
                  	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3595)
                  	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
                  	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
                  	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
                  	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
                  	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
                  	at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
                  	at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
                  	at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
                  	at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:102)
                  	at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:817)
                  	at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:1)
                  	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:586)

                  Comment


                  • #10
                    An in-memory repo solved, for now.

                    Comment


                    • #11
                      Hi everybody,

                      passing data through steps is OK.

                      To avoid writing too many data in the execution context, I decided to use an holder bean.

                      The holder bean

                      Code:
                      public class AuthorListHolder {
                      	
                      	private List<Author> authorList;
                      
                      	public List<Author> getAuthorList() {
                      		
                      		return authorList;
                      	}
                      
                      	public void setAuthorList(List<Author> authorList) {
                      		
                      		this.authorList = authorList;
                      	}
                      }
                      Config snippets

                      Code:
                      <bean id="authorListHolder" class="..."/>
                      
                      <bean id="savingAuthorListItemWriter" class="...">
                      	<property name="authorListHolder" ref="authorListHolder"/>
                      </bean>
                      
                      <bean id="retrievingAuthorItemReader" class="...">
                      	<property name="authorListHolder" ref="authorListHolder"/>
                      </bean>
                      The write method of the writer

                      Code:
                      @Override
                      public void write(List<? extends List<Author>> items)throws Exception {
                      	
                      	for(List<Author> authorList : items) {
                      		
                      		// my SavingAuthorListItemWriter.setAuthorListHolder contains: this.authorListHolder.setAuthorList(new ArrayList<Author>());
                      		authorListHolder.getGroupList().addAll(authorList);
                      	}
                      }
                      The read method of the reader

                      Code:
                      @Override
                      public Author read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
                      	
                      	if(!authorListHolder.getAuthorList().isEmpty()) {
                      		
                      		return authorListHolder.getAuthorList().remove(0);
                      	}
                      	
                      	return null;
                      }
                      I think the holder bean solution is ok, because it allowed me to restore DB job repository and to have a cleaner code.

                      Thank you for your patience!

                      Comment


                      • #12
                        Nice !
                        It looks a lot cleaner than passing things to the jobContext map !
                        Thanks for the tip ;-)

                        Comment


                        • #13
                          So, following this, I took a concrete example with a simple object called "Author", having 2 properties (first and last name).
                          I define an XSD schema for it, assuming you already have one for your object, but this is just for the demo :
                          HTML Code:

                          <?xml version="1.0" encoding="UTF-8"?>
                          <xsd:schema targetNamespace="http://com.authors/batch"
                          xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://com.authors/batch"
                          elementFormDefault="qualified" attributeFormDefault="unqualified">
                          <xsd:element name="Authors">
                          <xsd:complexType>
                          <xsd:sequence>
                          <xsd:element ref="Author" maxOccurs="unbounded"/>
                          </xsd:sequence>
                          </xsd:complexType>
                          </xsd:element>
                          <xsd:element name="Author">
                          <xsd:complexType>
                          <xsd:sequence>
                          <xsd:element name="FirstName" type="xsd:string" />
                          <xsd:element name="LastName" type="xsd:string" />
                          </xsd:sequence>
                          </xsd:complexType>
                          </xsd:element>

                          </xsd:schema>

                          ________________
                          virtual assistant

                          Comment

                          Working...
                          X