Announcement Announcement Module
Collapse
No announcement yet.
Composite ItemWriter Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Composite ItemWriter

    Hi,

    I had gone through the Composite ItemWriter, and it looks like the implementation of Composite ItemWriter is totally dependent on the chain which makes it Composite.

    My use case is : Read an input file (Item Reader), Perform some validation (Item Processor) and then write them (Item Writer). But here is the catch

    (1) I will have to process multiple input files with same prefix and write a single output file. For this, I have configured the step to have <scope=step> so that it gets intialized only when the step is actually called. The input for the file would be set in JobExecution context.
    Question here is, with a single key, I can set only one value. Now i might have multiple values, so do I set the value as
    Code:
    <bean id="flatFileItemReader" scope="step"
    class="org.springframework.batch.item.file.FlatFileItemReader">
    <property name="resource" value="#{jobExecutionContext['input.file.name']}" />
    </bean>
    ?

    Is this equivalent to
    Code:
    <bean id="flatFileItemReader" scope="step"
    class="org.springframework.batch.item.file.FlatFileItemReader">
    <property name="resource" value="classpath:/input/file-*" />
    </bean>
    ?

    2) Now, for the writing part. I need to write the data to a database and also write it to a file. The database update is for the application, the writing to file is to be used for generating report. Does springbatch provide anything like Chained Item Writers ? I ll be using FlatFileItemWiter out of the box, as it covers up all my scenarions. But it was mentioned, its not thread safe. Can some one tell me, why it's NOT?

  • #2
    Originally posted by rohitmohta87 View Post
    Is this equivalent to
    Code:
    <bean id="flatFileItemReader" scope="step"
    class="org.springframework.batch.item.file.FlatFileItemReader">
    <property name="resource" value="classpath:/input/file-*" />
    </bean>
    ?
    No it's not the same. A FlatFileItemReader accepts a single Resource as it's input, and you are trying to give it a Resource[] array. Have you tried using MultiResourceItemReader?

    2) Now, for the writing part. I need to write the data to a database and also write it to a file. The database update is for the application, the writing to file is to be used for generating report. Does springbatch provide anything like Chained Item Writers ?
    I don't understand why CompositeItemWriter is not useful to you. Can you explain a bit more?

    I ll be using FlatFileItemWiter out of the box, as it covers up all my scenarions. But it was mentioned, its not thread safe. Can some one tell me, why it's NOT?
    Because it is stateful, in particular for restart data and things like footer callbacks. If you don't use the state you might get away with using it in a multi-threaded step. Your mileage may vary.

    Comment


    • #3
      Hi Dave,

      Yes, I am using MultiResoureItemReader, to pass [] of resource.

      For the 2nd part, the input what I will be getting from file will have to be inserted in database. There are some validations done in the Stored Procedure(SP) and then inserted. If it fails, then the SP will be returning appropriate error message. This error message will have to be written in file.
      If we see CompositeItemProcessor, the item processed by ItemProcessor1 is passed to ItemProcessor2. So if I change a value a variable in ItemProcessor1, it will get reflected in ItemProcessor2.
      Is it the same case with CompositeItemWriter? Since ItemWriter's write method has a return type as void.

      And regarding FlatFileItemWriter, I didn't quite get it. I might have misunderstood, but when we say its stateful, we mean it can store some persistent value , and here its number of data written. The count which gets updated when an update method is called. So, only one thread should be accessing the method or atleast the count variable right? So, that we have a synchronized increment?
      Could you please explain the below also?
      you don't use the state you might get away with using it in a multi-threaded step. Your mileage may vary.
      Last edited by rohitmohta87; May 1st, 2011, 03:58 AM.

      Comment


      • #4
        In simple words, for Case 2 - if my input has 10 records, my output file will have 12 records. Additional two being, whether success or failure and if failure what is the reason.
        Validation of data happens in multiple point, in memory we have one and the other is in SP.
        For now, I am using compositeitem processor. Which will append all the data in a bean, and that bean is passed to FlatFileItemWriter.

        Comment


        • #5
          Originally posted by rohitmohta87 View Post
          And regarding FlatFileItemWriter, I didn't quite get it. I might have misunderstood, but when we say its stateful, we mean it can store some persistent value , and here its number of data written. The count which gets updated when an update method is called. So, only one thread should be accessing the method or at least the count variable right? So, that we have a synchronized increment?
          In the general case simply synchronizing access to the shared state is not sufficient to make a component thread safe. I looked at the FlatFileItemWriter in more detail though, and as far as I can tell the only use for the state is to prevent a header being written twice, so if you are not using a header callback you are probably OK, and you may even find that it works fine with a header (since that is only used in open()). In other words FlatFileItemWriter may in fact be pretty safe to use in a multi-threaded step. It won't (of course) be safe to use in multiple concurrent steps, but you can prevent that happening by using StepScope.

          Back to your specific problem. I still haven't understood what that is. You want to write some extra records at the end of a step? Maybe use a StepExecutionListener (and ItemStream if you want it to be restartable)?

          Comment


          • #6
            Thanks for the explanation on FlatFileItemWriter :-)

            My Usecase, in simple layman's language is this (this is the current scenario)
            FlatFileItemReader : Read data from multiple files

            CompositeItemProcessor : It has 3 ItemProcessors. 2 of them make checks on the data - memory check. 3rd one, inserts data by calling a StoredProcedure. Now, when I say insert - it doesn't mean a plain insert. The data from file is passed to Sp. In SP also we have some validation work. If all fine, then insert. Else SP will return the error code.

            FlatFileItemWrier : Writes the original content of input file in a separate file, and also mentioned whether it was success/failure and if failed - the error code.

            So u see, I my itemwriter would require 2 more values - Result and Result Desc which could be sealed off by ItemProcessors i.e. populated by them.

            My thinking was, as we have compositeItemProcessors - the result of one itemprocessor is an input to next in list. Similarly, do we have compositeItemWriters? Changes made by one ItemWrier we used by next in list?

            And later, I realized - my direction of earlier thought was wrong. ItemWriters are used for writing. With a compositeItemWriter - I can write to 2 different output source i.e. same output object.

            Comment

            Working...
            X