Announcement Announcement Module
Collapse
No announcement yet.
Pass File name to spring bath job from servlet on demand basis Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Pass File name to spring bath job from servlet on demand basis

    How to Run a Job without hardcoding the input file name - "e:\player.csv"

    Code:
     
    <bean id="playerFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
    <!-- <property name="resource" value="classpath:input/player.csv" /> -->
    <property name="resource" value="file:e:\player.csv" /> 
    .............
    .............
    </bean>
    Regards, Rams

  • #2
    use step scope, job parameters, and SpEL (e.g. #{jobParameters['inputFile']}): http://static.springsource.org/sprin...l#late-binding

    Comment


    • #3
      Hi arno,

      I wrote the files like this....

      FileServlet.java
      ServletContext context = getServletContext();
      WebApplicationContext applicationContext = WebApplicationContextUtils.getWebApplicationContex t(context);
      JobLauncher launcher = (JobLauncher ) applicationContext.getBean("jobLauncher");
      Job job = (Job ) applicationContext.getBean("fileToFilesJob");
      //JobParameters jobParameters= new JobParametersBuilder().addDate("schedule.time", date).toJobParameters();
      JobParameters jobParameters= new JobParametersBuilder().addString("inputFile", "e:\\player.csv").addDate("schedule.time", date).toJobParameters();
      try
      {
      status = launcher.run((org.springframework.batch.core.Job)j ob,jobParameters);
      }
      catch(Exception e)
      ....................
      Filejob.xml
      <bean id="playerFileItemReader" scope="step" class="org.springframework.batch.item.file.FlatFil eItemReader">
      <property name="resource" value="#{jobParameters['inputFile']}" />
      <property name="lineMapper">
      .......................
      i am getting the following error
      ERROR - Encountered an error executing the step: class org.springframework.batch.item.ItemStreamException : Failed to initialize the reader
      org.springframework.batch.item.ItemStreamException : Failed to initialize the reader
      at org.springframework.batch.item.support.AbstractIte mCountingItemStreamItemReader.open(AbstractItemCou ntingItemStreamItemReader.java:111)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknow n Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Un known Source)
      at java.lang.reflect.Method.invoke(Unknown Source)
      ........
      Caused by: java.lang.IllegalStateException: Input resource must exist (reader is in 'strict' mode): ServletContext resource [/e:/player.csv]
      at org.springframework.batch.item.file.FlatFileItemRe ader.doOpen(FlatFileItemReader.java:245)
      at org.springframework.batch.item.support.AbstractIte mCountingItemStreamItemReader.open(AbstractItemCou ntingItemStreamItemReader.java:108)
      Last edited by Rams; May 12th, 2011, 06:52 AM. Reason: detailinfo

      Comment


      • #4
        arno can you please correct me..................

        Comment


        • #5
          first, are you sure the file does exist? second, try "file:e:/player.csv" instead of "e:\\player.csv"

          Comment


          • #6
            Hi arno,

            Very greatfull thanks to you. it is working fine now.

            i have one more doubt. If i want to pass two out put file names dynamically then, how can i define in the below filejob.xml.
            Code:
             <bean id="playerFileItemWriter1" class="org.springframework.batch.item.file.FlatFileItemWriter">
            <property name="name" value="fw1" /> <!--  This is import. a way distinguish writes -->
            <property name="resource" value="file:e:\output1.csv" /> 
            <property name="shouldDeleteIfExists" value="true"/>......
            <bean id="playerFileItemWriter2" class="org.springframework.batch.item.file.FlatFileItemWriter">
            <property name="name" value="fw2" /> <!--  This is import. a way distinguish writes -->
            <property name="resource" value="file:e:\output2.csv" /> 
            <property name="shouldDeleteIfExists" value="true"/>......
            Regards,
            Rams

            Comment


            • #7
              I don't get why you want to do, can you be more precise? You want to write in multiple files in the same step or in different steps?

              Comment


              • #8
                Hi arno,

                Yes, I want to write multiple files in the same step.
                FileJob.xml
                Code:
                 
                <bean id="dynamicJobParameters" class="DynamicJobParameters"/>    
                 
                <batch:job id="fileToFilesJob" incrementer="dynamicJobParameters" job-repository="jobRepository">
                <batch:step id="step1">
                <batch:tasklet transaction-manager="jobRepository-transactionManager" >
                <batch:chunk reader="playerFileItemReader" processor="careerProcessor" writer="compositeWriter" 
                commit-interval="1">
                <batch:streams>
                <batch:stream ref="playerFileItemReader"/>
                <batch:stream ref="playerFileItemWriter1"/>
                <batch:stream ref="playerFileItemWriter2"/>
                </batch:streams>
                </batch:chunk>
                </batch:tasklet>
                </batch:step>
                </batch:job>
                 
                <!-- INFRASTRUCTURE SETUP -->
                 
                <bean id="compositeWriter" class="SeparatorCompositeItemWriter">
                 
                <!--  <bean id="compositeWriter" class="org.springframework.batch.item.support.CompositeItemWriter"> -->
                <property name="delegates">
                <list>                
                <ref bean="playerFileItemWriter1" />
                <ref bean="playerFileItemWriter2" />
                </list>
                </property>
                </bean>
                 
                <bean id="playerFileItemReader" scope="step" class="org.springframework.batch.item.file.FlatFileItemReader">
                <!--  <property name="resource" value="classpath:input/player.csv" /> -->
                <!-- <property name="resource" value="file:e:\player.csv" />         -->
                <property name="resource" value="#{jobParameters['inputFile']}" />
                
                <!--<property name="resource">-->
                <!--<bean class="org.springframework.core.io.FileSystemResource" scope="step">-->
                <!--<constructor-arg value="#{jobParameters[inputFile]}" />-->
                <!--</bean>-->
                <!--</property>-->
                
                 <property name="lineMapper">   
                <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">                
                <property name="lineTokenizer">                
                <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                <property name="delimiter" value=","/>
                <property name="names" value="ID,lastName,firstName,position,debutYear,finalYear" />
                </bean>
                </property>
                <property name="fieldSetMapper">
                <bean class="PlayerFieldSetMapper" />
                </property>
                </bean>
                </property>
                </bean>
                 
                <bean id="careerProcessor" class="CareerProcessor" />
                 
                <bean id="playerFileItemWriter1" class="org.springframework.batch.item.file.FlatFileItemWriter">
                <property name="name" value="fw1" /> <!--  This is import. a way distinguish writes -->
                <property name="resource" value="file:e:\output1.csv" /> 
                <property name="shouldDeleteIfExists" value="true"/>
                <property name="lineAggregator">
                <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
                <property name="delimiter" value=","/>
                <property name="fieldExtractor">
                <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
                <property name="names" value="id,firstName,lastName,position,debutYear,finalYear,careerLength"/>
                </bean>
                </property>
                </bean>
                </property>
                </bean>
                 
                <bean id="playerFileItemWriter2" class="org.springframework.batch.item.file.FlatFileItemWriter">
                <property name="name" value="fw2" /> <!--  This is import. a way distinguish writes -->
                <property name="resource" value="file:e:\output2.csv" /> 
                <property name="shouldDeleteIfExists" value="true"/>
                <property name="lineAggregator">
                <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
                <property name="delimiter" value=","/>
                <property name="fieldExtractor">
                <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
                <property name="names" value="id,firstName,lastName,position,debutYear,finalYear,careerLength"/>
                </bean>
                </property>
                </bean>
                </property>
                </bean>    
                 
                </beans>
                actaully the above job working fine for me, but here the out put file names are hard coded. But i want to give/generate/assign the file names at runtime. How can we handle this?

                Comment


                • #9
                  same thing as the input file: step scope, job parameters, and SpEL.

                  Comment


                  • #10
                    Originally posted by arno View Post
                    same thing as the input file: step scope, job parameters, and SpEL.
                    As per your advice I am passing the output file names as outputFile1 & outputFile2 in the jobParameters, but I am getting classcastexception which I have pasted further down below, also I am pasting the Job.xml where apporpriate changes have been made.
                    Pls help me!!!

                    SERVLET CODE:
                    Code:
                    	protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
                    	    JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
                    	    jobParametersBuilder.addString("inputFile","file:e:/player.csv");
                    	    jobParametersBuilder.addString("outputFile1","file:e:/output1.csv");
                    	    jobParametersBuilder.addString("outputFile2","file:e:/output2.csv");
                    	    jobParametersBuilder.addDate("schedule.time", date);
                    	    JobParameters jobParameters= jobParametersBuilder.toJobParameters();
                    	    try
                    	    {
                    		status = launcher.run((
                                               org.springframework.batch.core.Job)job,jobParameters);
                    	    }
                    	  .....
                    	  ....
                    	}
                    Exception Stack trace:
                    Code:
                    ERROR - Encountered an error executing the step: class java.lang.ClassCastException: $Proxy3
                    java.lang.ClassCastException: $Proxy3
                    	at SeparatorCompositeItemWriter.write(SeparatorCompositeItemWriter.java:23)
                    	at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:155)
                    	at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:136)
                    	at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:207)
                    	at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:191)
                    	at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:70)
                    	at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264)
                    	at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:67)
                    	at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:352)
                    	at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:212)
                    	at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
                    	at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:239)
                    	at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:197)
                    	at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:348)
                    	at org.springframework.batch.core.job.flow.FlowJob.access$100(FlowJob.java:43)
                    	at org.springframework.batch.core.job.flow.FlowJob$JobFlowExecutor.executeStep(FlowJob.java:137)
                    	at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
                    	at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
                    	at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
                    	at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:105)
                    	at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:250)
                    	at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:110)
                    	at java.lang.Thread.run(Unknown Source)
                    Job.xml
                    Code:
                     
                    <?xml version="1.0" encoding="UTF-8"?>
                    <beans xmlns="http://www.springframework.org/schema/beans"
                    xmlns:batch="http://www.springframework.org/schema/batch" 
                    xmlns:aop="http://www.springframework.org/schema/aop" xmlns:tx="http://www.springframework.org/schema/tx"
                    xmlns:p="http://www.springframework.org/schema/p" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                    xsi:schemaLocation="
                    http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
                    http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.0.xsd
                    http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.0.xsd
                    http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.0.xsd">
                     
                    <description>
                    <![CDATA[Sample showing usage of CompositeItemWriter.
                    Note that when two writers of the same class are used simultaneously 
                    they need to be distinguished using the 'name' property 
                    (see the two FlatFileItemWriters used in this example).]]>
                    </description>
                     
                    <import resource="MEMORY-JOBREPOSITORY.xml"/>   
                    <!-- <import resource="DB-JOBREPOSITORY.xml"/> --> 
                     
                    <bean id="dynamicJobParameters" class="DynamicJobParameters"/>    
                     
                    <batch:job id="fileToFilesJob" incrementer="dynamicJobParameters" job-repository="jobRepository">
                    <batch:step id="step1">
                    <batch:tasklet transaction-manager="jobRepository-transactionManager" >
                    <batch:chunk reader="playerFileItemReader" processor="careerProcessor" writer="compositeWriter" 
                    commit-interval="1">
                    <batch:streams>
                    <batch:stream ref="playerFileItemReader"/>
                    <batch:stream ref="playerFileItemWriter1"/>
                    <batch:stream ref="playerFileItemWriter2"/>
                    </batch:streams>
                    </batch:chunk>
                    </batch:tasklet>
                    </batch:step>
                    </batch:job>
                     
                    <!-- INFRASTRUCTURE SETUP -->
                     
                    <bean id="compositeWriter" class="SeparatorCompositeItemWriter">
                     
                    <!--  <bean id="compositeWriter" class="org.springframework.batch.item.support.CompositeItemWriter"> -->
                    <property name="delegates">
                    <list>                
                    <ref bean="playerFileItemWriter1" />
                    <ref bean="playerFileItemWriter2" />
                    </list>
                    </property>
                    </bean>
                     
                    <bean id="playerFileItemReader" scope="step" class="org.springframework.batch.item.file.FlatFileItemReader">
                    <!--  <property name="resource" value="classpath:input/player.csv" /> -->
                    <!-- <property name="resource" value="file:e:\player.csv" />         -->
                    <property name="resource" value="#{jobParameters['inputFile']}" />
                    
                    <!--<property name="resource">-->
                    <!--<bean class="org.springframework.core.io.FileSystemResource" scope="step">-->
                    <!--<constructor-arg value="#{jobParameters[inputFile]}" />-->
                    <!--</bean>-->
                    <!--</property>-->
                    
                     <property name="lineMapper">   
                    <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">                
                    <property name="lineTokenizer">                
                    <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                    <property name="delimiter" value=","/>
                    <property name="names" value="ID,lastName,firstName,position,debutYear,finalYear" />
                    </bean>
                    </property>
                    <property name="fieldSetMapper">
                    <bean class="PlayerFieldSetMapper" />
                    </property>
                    </bean>
                    </property>
                    </bean>
                     
                    <bean id="careerProcessor" class="CareerProcessor" />
                     
                    <bean id="playerFileItemWriter1" scope="step" class="org.springframework.batch.item.file.FlatFileItemWriter">
                    <property name="name" value="fw1" /> <!--  This is import. a way distinguish writes -->
                    <!--<property name="resource" value="file:e:\output1.csv" /> -->
                    <property name="resource" value="#{jobParameters['outputFile1']}" />
                    <property name="shouldDeleteIfExists" value="true"/>
                    <property name="lineAggregator">
                    <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
                    <property name="delimiter" value=","/>
                    <property name="fieldExtractor">
                    <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
                    <property name="names" value="id,firstName,lastName,position,debutYear,finalYear,careerLength"/>
                    </bean>
                    </property>
                    </bean>
                    </property>
                    </bean>
                     
                    <bean id="playerFileItemWriter2" scope="step" class="org.springframework.batch.item.file.FlatFileItemWriter">
                    <property name="name" value="fw2" /> <!--  This is import. a way distinguish writes -->
                    <!--<property name="resource" value="file:e:\output2.csv" /> -->
                    <property name="resource" value="#{jobParameters['outputFile2']}" />
                    <property name="shouldDeleteIfExists" value="true"/>
                    <property name="lineAggregator">
                    <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
                    <property name="delimiter" value=","/>
                    <property name="fieldExtractor">
                    <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
                    <property name="names" value="id,firstName,lastName,position,debutYear,finalYear,careerLength"/>
                    </bean>
                    </property>
                    </bean>
                    </property>
                    </bean>    
                     
                    </beans>

                    Comment


                    • #11
                      looks like the exception comes from your own code (SeparatorCompositeItemWriter class). What are you doing in this class? Don't forget you can't manipulate the delegates with their class names, as they're step-scoped proxies (i.e. don't cast the delegates as FileFileItemWriter).

                      Comment


                      • #12
                        Originally posted by arno View Post
                        looks like the exception comes from your own code (SeparatorCompositeItemWriter class). What are you doing in this class? Don't forget you can't manipulate the delegates with their class names, as they're step-scoped proxies (i.e. don't cast the delegates as FileFileItemWriter).
                        Hi arno,

                        I am doing separation of the files based on careerLength condition.
                        Code:
                        @SuppressWarnings("unchecked")
                        public class SeparatorCompositeItemWriter implements ItemWriter 
                        {
                         
                        private List delegates;
                         
                        public void setDelegates(ItemWriter[] delegates) {
                        this.delegates = Arrays.asList(delegates);
                        }
                         
                        public void write(List items) throws Exception 
                        {
                        FlatFileItemWriter write1 = (FlatFileItemWriter)delegates.get(0);
                        FlatFileItemWriter write2 = (FlatFileItemWriter)delegates.get(1);
                         
                        List file1Items = new ArrayList<Player>();
                        List file2Items = new ArrayList<Player>();
                         
                        for(Object player: items)
                        {
                        Player pl = (Player)player;
                        if(pl.getCareerLength()< 25 ) file1Items.add(player);
                        else file2Items.add(player);
                        }
                         
                        write1.write(file1Items);
                        write2.write(file2Items);
                        }
                        }
                        actaully this file going smoothly before the multiple fine name change. But now its showing error ? what is wrong in this file? Can you please help on this.

                        Comment


                        • #13
                          ok, so you're doing exactly what I mentioned in my previous post:
                          Code:
                          FlatFileItemWriter write1 = (FlatFileItemWriter)delegates.get(0);
                          FlatFileItemWriter write2 = (FlatFileItemWriter)delegates.get(1);
                          First, it can't work because the delegates are proxies (they're step-scoped). Second, you don't need to do that. Do it this way:
                          Code:
                          ItemWriter write1 = (ItemWriter)delegates.get(0);
                          ItemWriter write2 = (ItemWriter)delegates.get(1);
                          so try to make it work and then we'll see how to use generics.

                          Comment


                          • #14
                            Hi Arno,

                            Greatful thanks to you arno.
                            So nice of you!. Your ideas help's me a lot.

                            Actually, while processing a input csv file, - for each record/row in the csv file - can we validate it against a DB table, before pushing it to DB.

                            Regards,
                            Rams

                            Comment

                            Working...
                            X