Announcement Announcement Module
Collapse
No announcement yet.
Multi-line Multi record flat file Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Multi-line Multi record flat file

    Hello,

    I have a txt file which has multi line records and each of this record has its own records.

    [Data.1]
    "Header","2","2011-07-05 00:00:01",
    "Price","2132467",35
    "Price","1184067",35
    "Price","4981716",25
    "Price","4940699",70

    [Data.2]
    "Header","2","2011-07-03 00:00:11",
    "Price","2551467",45
    "Price","1184067",25
    "Price","4981716",65
    "Price","4940699",80
    ..

    I have used "MultiLineProductRecordSeparatorPolicy" to read each [Data]. But I don't know how I can get the "Price" records in this Data. Also it seems that "linesToSkip" property does not work when I use multiLineRecordSeparatorPolicy.

    <bean id="flatFileDataReader" class="org.springframework.batch.item.file.FlatFil eItemReader">
    <property name="linesToSkip" value="31"/>
    <property name="recordSeparatorPolicy" ref="multiLineRecordSeparatorPolicy"/>
    <property name="lineMapper" ref="priceLineMapper"/>
    </bean>

    Thanks,
    Asal

  • #2
    Multi Line Records

    Did you try multi line records in general? (see multi line records en detail also)

    like so

    Code:
    <bean id="orderFileLineMapper"
          class="org.spr...PatternMatchingCompositeLineMapper">
        <property name="tokenizers">
            <map>
                <entry key="Data*" value-ref="dataTokenizer" />
                <entry key="Header*" value-ref="headerTokenizer" />
                <entry key="Price*" value-ref="priceTokenizer" />
            </map>
        </property>
        <property name="fieldSetMappers">
            <map>
                <entry key="Data*" value-ref="dataFieldSetMapper" />
                <entry key="Header*" value-ref="headerFieldSetMapper" />
                <entry key="Price*" value-ref="priceFieldSetMapper" />
            </map>
        </property>
    </bean>

    Multi Line Records Examples

    If you need to get one unified line, take a look at the Spring Batch Samples, e.g. the multi-line example(github source)

    Possible Problem with your input file

    If you use a DelimitedLineTokenizer i am almost sure the following lines will be count as one, due to the
    ,
    at the end of the "Header" line.

    Code:
    "Header","2","2011-07-05 00:00:01",
    "Price","2132467",35
    Last edited by michael.lange; Oct 7th, 2011, 02:33 AM.

    Comment


    • #3
      Multi line records

      Hey,

      Thanks for your reply. Yes I have seen the PatternMatchingCompositeLineMapper. But that is used when you have different types of records. But in my case I have one multi-line record (Data) and I want to relate the Price and Header to one Data Object. If I read each Price as one record, then how can I know that each Price record relates to what Data record?

      Cheers,
      Asal

      Comment


      • #4
        for your usecase i pointed to the example

        Multi Line Records Examples

        If you need to get one unified line, take a look at the Spring Batch Samples, e.g. the multi-line example(github source)
        a AggregateItemReader is used there, it should be no problem to adjust the example to one, where you keep the data-id for each price record

        one other solution could be:
        • go with the "read each line" example
        • in processor or writer decide - if header-line, keep data-id in StepExecutionContext, do not persist (do not call DAO layer, wrapped jdbcWriter, whatever)
        • if price-line, extract data-id from StepExecutionContext, persist price
        • next header-line - just overwrite data-id in context
        • and so on

        Comment


        • #5
          Multi line records

          Thanks for your reply. I got just another question. How can I access StepExecutionContext in the ItemWriter? I read in the documents that I can have a StepListener for each Step. But I couldn't find out how I can access the StepExecutionContext in the writer.

          Thanks,
          Asal

          Comment


          • #6
            to access the StepExecutionContext you can get the writer to be a listener too, just create a custom writer


            example job.xml

            Code:
                <job id="skipJob" xmlns="http://www.springframework.org/schema/batch">
                    <step id="skipJobStep">
                        <tasklet>
                            <chunk 
                                reader="itemReader" 
                                writer="itemWriter" 
                                commit-interval="5">
                            </chunk>
                        </tasklet>
                        <listeners>
                            <listener ref="itemWriter" />
                        </listeners>            
                    </step>
                </job>
            its important on which level you use the listeners element, the spring batch doc example is for chunklisteners
            Last edited by michael.lange; Oct 11th, 2011, 08:32 AM.

            Comment


            • #7
              Multi line records

              Thanks for your reply. I got it to work finally!

              Asal

              Comment

              Working...
              X