Announcement Announcement Module
Collapse
No announcement yet.
MultiFileItemReader prior step generates resources FlatFileItemReader error Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • MultiFileItemReader prior step generates resources FlatFileItemReader error

    Hello,

    Spring Batch 2.1.9
    Framework 3.2.2

    I have a Multi Resource Item Reader that extends MultiResourceItemReader and simply injects my stepExecutionListener. I do this as the prior step in the job has created the FileSystemResource Resources (and recorded it with the JobExecution ExecutionContext) to pass to it as the directory for the run is dynamically created.

    The multi resource item reader then hands off to a ResourceAwareItemReaderItemStream composite reader. If I hard code the resources value - job runs and reads all *.txt in the dynamically created folder.

    Code:
    <bean id="reader" class="com.know.commercial.file.MultiFileItemReader" scope="step" >
    <property name="stepExecutionListener" ref="stepExecutionListener" />
    <property name="delegate" ref="compositeInvoiceItemReader" />
    <property name="resources" value="file://home/xx/Downloads/ftp/get/abc/517fd777ccf2111d24a3e44b/*.txt" />
    <property name="strict" value="false" />
    </bean>
    I read in the docs that I need to use a @BeforeStep in the MultiResourceItemReader if I am assigning the resources dynamically so I have:

    Code:
    public class MultiFileItemReader extends MultiResourceItemReader<File> {
    	private StepExecutionListenerCtxInjector stepExecutionListener;
    
    	
    	@BeforeStep
    	public void retrieveInterStepFileResources() {
    		Resource[] resources = (Resource[]) stepExecutionListener.getJobExecutionCtx().get("ftpResources");
    		System.out.println("first resource: " + resources[0]);
    		super.setResources(resources);
    	}
    	
    	public void setStepExecutionListener(StepExecutionListenerCtxInjector stepExecutionListener) {
    		this.stepExecutionListener = stepExecutionListener;
    	}
    
    }
    The job runs and outputs to system the first resource but I get:

    Caused by: java.lang.IllegalStateException: Input resource must exist (reader is in 'strict' mode): file [/home/xx/Downloads/ftp/get/abc/518010d6ccf2e13090294d1d/*.txt] at org.springframework.batch.item.file.FlatFileItemRe ader.doOpen(FlatFileItemReader.java:250)
    Which makes me suspect that my configuration timing is not right for FlatFileItemReader. I tried passing a reference to my MultiFileItemReader to it in its constructor (which as it is ResourceAware I didn't think I had to do) :

    Code:
    public class CompositeInvoiceItemReader implements ResourceAwareItemReaderItemStream {
    	private MultiFileItemReader itemReader;
    	private FlatFileItemReader delegate;
    	
    
    	public CompositeInvoiceItemReader(MultiFileItemReader itemReader) {
    		super();
    		this.itemReader = itemReader;
    		this.delegate.setResource(this.itemReader.getCurrentResource());
    		this.delegate = new FlatFileItemReader();
    		this.delegate.setLineMapper(new CompositeInvoiceLineMapper());
    		this.delegate.setStrict(true);
    	}
    ....
    but get
    Code:
    java.lang.IllegalArgumentException: Resources must be set
    My MultiFileItemReader is scope step along with compositeInvoiceItemReader.

    Feel I am close but final step(s) escape me.

    When I check the BATCH_JOB_EXECUTION_CONTEXT SHORT_CONTEXT column the output is the same when I create the FileSystemResource dynamically and when I hard code it in the resources property of the bean in the xml. Really appears to be timing....

    Thanks for looking - please advise.

  • #2
    I think you want to assign the resource file dynamically if I am correct than try to provide your file as job parameter or in before job or in job execution context please refer this link
    http://static.springsource.org/sprin...tml#step-scope

    I think this will resolve ur problem

    Comment


    • #3
      Hi Sandeep33 - thanks for the suggestion.

      The reader (MultiFileItemReader which extends MultiResourceItemReader) is using scope="step".

      The reader:delegate CompositeInvoiceReader is also using scope="step".

      If reader:resources is not a hard coded "file://....." the FlatFileItemReader doesn't get handed a file in the resource.

      I have tried

      Code:
      <property name="resources" value="#{jobExecutionContext[ftpResources]}" />
      I have tried lazy-init="true" both for the MultiResource and the CompositeReader - each time:

      Code:
      Caused by: java.lang.IllegalStateException: Input resource must exist (reader is in 'strict' mode): file [/home/xx/Downloads/ftp/get/abc/51813398ccf259858ce1e778/*.txt]
      	at org.springframework.batch.item.file.FlatFileItemReader.doOpen(FlatFileItemReader.java:250)
      The file mentioned is actually the MultiResource dynamically assigned resources property. The MultiResource is not handing any *.txt files within that folder to the CompositeReader (or doing it too late).

      Still stumped. So all works if resources hard coded but when I try to dynamically assign the resources - FlatFileItemReader generates the above error.

      Comment


      • #4
        I think reader not able to get the file path properly
        check the below links:
        http://stackoverflow.com/questions/1...-with-spring-3
        http://stackoverflow.com/questions/1...not-refreshing
        http://stackoverflow.com/questions/8...classpath?rq=1

        Comment


        • #5
          MultiFileItemReader prior step generates resources FlatFileItemReader error - SOLVED

          Sandeep33,

          I checked out those links - and your thinking is along the right lines.

          Hard coded the resources values looks like

          Code:
          <property name="resources" value="file://home/xx/Downloads/ftp/get/abc/5181721fccf22652f034deb3/*.txt" />
          So in a prior tasklet step I did the following:

          Code:
          Resource[] resources = getMultiResourceItemReaderValidResources(folders);
          stepExecutionListener.getJobExecutionCtx().put("ftpResources", resources);
          Code:
          private Resource[] getMultiResourceItemReaderValidResources(HashSet<String> folders) {
          		ArrayList<String> validExts = InvoiceFileExtensionCheck.validFileExtensions();
          		ArrayList<Resource> resource = new ArrayList<Resource>();
          		
          		Iterator<String> inner = validExts.iterator();
          		Iterator<String> outer = folders.iterator();
          		while (outer.hasNext()) {
          			while (inner.hasNext()) {
          				 resource.add(new FileSystemResource(outer.next() + "/*." + inner.next()));
          			}
          		}
          		return resource.toArray(new Resource[resource.size()]);
          	}
          which I thought is what I need to do to match what the resources expects for value.

          When I changed it to:

          Code:
          private String[] getMultiResourceItemReaderValidResources(HashSet<String> folders) {
          	ArrayList<String> validExts = InvoiceFileExtensionCheck.validFileExtensions();
          	ArrayList<String> resource = new ArrayList<String>();
          		
          	Iterator<String> inner = validExts.iterator();
          	Iterator<String> outer = folders.iterator();
          	while (outer.hasNext()) {
          		while (inner.hasNext()) {
          			 resource.add("file:/" + (outer.next() + "/*." + inner.next()));
          		}
          	}
          	return resource.toArray(new String[resource.size()]);
          }
          i.e. using a String array and not making it a Resource array - it works!.

          Kicking myself for not trying that... Well done and thanks!

          Peter

          Comment


          • #6
            That's great peter if you dont have problem or wont mind it could you send me your code base on mail id :[email protected]

            Comment


            • #7
              Sandeep33,

              My impression of the Spring forum is a place to ask for help but much of the research, testing, tweaking, learning is done (and expected) by the submitter. The gurus don't jump in right away to solve the problem and circumvent the learning process. I have submitted 17 posts since starting Spring framework last December. I have spent a LOT of time reading, writing and rewriting and as you can see - still a long way to go.

              So with that said - first, I am not at liberty to share the code base and second do not wish to shortcut your learning process. I have not implemented anything that I haven't gleaned from the forum, docs, books on Spring, Spring batch and long hours.

              If you submit a thread and ask for help - happy to help if I can. This way all current and future members benefit.

              Best wishes in your learning.

              Comment

              Working...
              X