Announcement Announcement Module
Collapse
No announcement yet.
reading and writing to different flat files Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • reading and writing to different flat files

    Hi,
    I am trying to read from a list of fixed length flat files and want to write them sequentially in to another flat file. I used MultiResourceReader for ItemReader. For ItemWriter I used the FlatFileItemWriter. I am using PassThroughLineMapper and PassThroughLineAggregator.
    Writer is hanging after writing first 1 or 2 lines.
    I did try 1 file read and write to another file. Same behavior is observed.
    I really do not want to transform content in to objects. I just want to write bytes as they are. Can someone guide me to any such example or is it that i have to use java file nio for the same.
    I appreciate your help.
    Here is the configuration for simple file reader and writer with SomeDTO which has only setter and getter for Item which is a String.
    <job id="fileJob" xmlns="http://www.springframework.org/schema/batch">
    <step id="step" parent="step1"/>
    </job>

    <step id="step1" xmlns="http://www.springframework.org/schema/batch">

    <tasklet transaction-manager="transactionManager">
    <chunk writer="itemWriter" reader="itemReader" commit-interval="10" >

    </chunk>
    <listeners>
    <listener ref="stepExcecutionListener" />
    <listener ref="chunkListener" />
    <listener ref="itemReaderListener" />
    <listener ref="itemWriterListener" />
    <listener ref="itemSkipListener" />
    </listeners>
    </tasklet>
    </step>

    <bean id="itemReader" class="org.springframework.batch.item.file.FlatFil eItemReader" scope="step">
    <property name="resource" value="file:target/output/outfilepartition0.txt" />
    <property name="lineMapper">
    <bean class="org.springframework.batch.item.file.mapping .DefaultLineMapper">
    <property name="lineTokenizer">
    <bean
    class="org.springframework.batch.item.file.transfo rm.FixedLengthTokenizer">
    <property name="names" value="item" />
    <property name="columns" value="1-19" />
    </bean>
    </property>
    <property name="fieldSetMapper">
    <bean
    class="org.springframework.batch.item.file.mapping .BeanWrapperFieldSetMapper">
    <property name="targetType"
    value="org.someorg.batch.poc.partition.common.Some DTO" />
    </bean>
    </property>
    </bean>
    </property>
    </bean>

    <bean id="itemWriter" class="org.springframework.batch.item.file.FlatFil eItemWriter" scope="step">
    <property name="resource" value="file:target/output/outfilepartition.txt" />
    <property name="lineAggregator">
    <bean
    class="org.springframework.batch.item.file.transfo rm.FormatterLineAggregator">
    <property name="fieldExtractor">
    <bean
    class="org.springframework.batch.item.file.transfo rm.BeanWrapperFieldExtractor">
    <property name="names" value="item" />
    </bean>
    </property>
    <property name="format" value="%-20s" />
    </bean>
    </property>
    </bean>
    When this runs, it hangs.
    Here is the output:
    INFO SimpleChunkListener:12 - Calling the chunk read/write...

    INFO SimpleChunkListener:16 - chunk read/write complete!

    INFO SimpleChunkListener:12 - Calling the chunk read/write...
    INFO DataItemReadListener:31 -
    Hanged.

    Let me know, if something wrong in configuration. Help will be much appreciated.
    Last edited by vkotian; Aug 16th, 2010, 11:40 AM.

  • #2
    Nothing wrong with the reader and writer configuration that I can see. It's a single threaded step, so I can't think of any obvious way to hang it through a concurrency error. I guess I suspect your listeners; how are they implemented.

    (Please use [code][/code] tags to post code and stack traces.)

    Comment


    • #3
      reading and writing to different flat files

      Thank you for replying Dave. Here I have done some more.
      I just removed all other listeners except chunk listener.
      Here is how chunk listener is implemented.
      public class SimpleChunkListener {
      private final static Log logger = LogFactory.getLog(SimpleChunkListener.class);
      @BeforeChunk
      public void beforeChunk(){
      logger.info("Calling the chunk read/write...");
      }
      @AfterChunk
      public void afterChunk(){
      logger.info("chunk read/write complete!");
      }
      }
      Here is some more information. The file that I am reading is generated from partition job. it has 10 lines of string data. I'm not sure if it is not understanding EOF.

      Also here is more output with all the listeners.
      08:43:41,262 INFO SimpleChunkListener:12 - Calling the chunk read/write...
      08:43:41,309 INFO DataItemReadListener:31 - [email protected]
      08:43:41,309 INFO DataItemReadListener:31 - [email protected]
      08:43:41,309 INFO DataItemReadListener:31 - [email protected] d
      08:43:41,309 INFO DataItemReadListener:31 - [email protected] 4
      08:43:41,309 INFO DataItemReadListener:31 - [email protected] 2
      08:43:41,309 INFO DataItemReadListener:31 - [email protected]
      08:43:41,309 INFO DataItemReadListener:31 - [email protected]
      08:43:41,309 INFO DataItemReadListener:31 - [email protected] 5
      08:43:41,309 INFO DataItemReadListener:31 - [email protected]
      08:43:41,309 INFO DataItemReadListener:31 - [email protected]
      08:43:41,325 INFO FlatFileFixedLengthWriterListener:64 - [[email protected] , [email protected] , [email protected] d, [email protected] 4, [email protected] 2, [email protected] , [email protected] , [email protected] 5, [email protected] , [email protected]]
      08:43:41,325 INFO SimpleChunkListener:16 - chunk read/write complete!
      08:43:41,325 INFO SimpleChunkListener:12 - Calling the chunk read/write...
      08:43:41,325 INFO DataItemReadListener:31 -

      Hangs.

      Comment


      • #4
        Thread dump

        Also here is the thread dump.

        2010-08-17 08:33:43
        Full thread dump Java HotSpot(TM) Client VM (16.3-b01 mixed mode, sharing):

        "RMI TCP Connection(2)-10.44.52.91" daemon prio=6 tid=0x03bf4c00 nid=0x938 runnable [0x051df000]
        java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream. java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStre am.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStre am.java:237)
        - locked <0x24480418> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(FilterInputStream.j ava:66)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages( TCPTransport.java:517)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandl er.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandl er.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

        Locked ownable synchronizers:
        - <0x2448cc98> (a java.util.concurrent.locks.ReentrantLock$NonfairSy nc)

        "JMX server connection timeout 14" daemon prio=6 tid=0x0438a400 nid=0x160c in Object.wait() [0x0518f000]
        java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x244805c8> (a [I)
        at com.sun.jmx.remote.internal.ServerCommunicatorAdmi n$Timeout.run(ServerCommunicatorAdmin.java:150)
        - locked <0x244805c8> (a [I)
        at java.lang.Thread.run(Thread.java:619)

        Locked ownable synchronizers:
        - None

        "RMI Scheduler(0)" daemon prio=6 tid=0x02fba800 nid=0xdec waiting on condition [0x0513f000]
        java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x24480650> (a java.util.concurrent.locks.AbstractQueuedSynchroni zer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(L ockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchroni zer$ConditionObject.awaitNanos(AbstractQueuedSynch ronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.ja va:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$D elayedWorkQueue.take(ScheduledThreadPoolExecutor.j ava:583)
        at java.util.concurrent.ScheduledThreadPoolExecutor$D elayedWorkQueue.take(ScheduledThreadPoolExecutor.j ava:576)
        at java.util.concurrent.ThreadPoolExecutor.getTask(Th readPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

        Locked ownable synchronizers:
        - None


        "Finalizer" daemon prio=8 tid=0x00a6a000 nid=0x1538 in Object.wait() [0x02e1f000]
        java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x295c0bb0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue .java:118)
        - locked <0x295c0bb0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue .java:134)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finali zer.java:159)

        Locked ownable synchronizers:
        - None

        "Reference Handler" daemon prio=10 tid=0x00a68800 nid=0x133c in Object.wait() [0x02dcf000]
        java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x295c0c38> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Refer ence.java:116)
        - locked <0x295c0c38> (a java.lang.ref.Reference$Lock)

        Locked ownable synchronizers:
        - None

        "main" prio=6 tid=0x003fc400 nid=0x1544 waiting on condition [0x0060e000]
        java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x29bbc318> (a java.util.concurrent.Semaphore$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSu pport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchroni zer.parkAndCheckInterrupt(AbstractQueuedSynchroniz er.java:747)
        at java.util.concurrent.locks.AbstractQueuedSynchroni zer.doAcquireSharedInterruptibly(AbstractQueuedSyn chronizer.java:905)
        at java.util.concurrent.locks.AbstractQueuedSynchroni zer.acquireSharedInterruptibly(AbstractQueuedSynch ronizer.java:1217)
        at java.util.concurrent.Semaphore.acquire(Semaphore.j ava:286)
        at org.springframework.batch.core.step.tasklet.Taskle tStep$ChunkTransactionCallback.doInTransaction(Tas kletStep.java:377)
        at org.springframework.transaction.support.Transactio nTemplate.execute(TransactionTemplate.java:130)
        at org.springframework.batch.core.step.tasklet.Taskle tStep$2.doInChunkContext(TaskletStep.java:261)
        at org.springframework.batch.core.scope.context.StepC ontextRepeatCallback.doInIteration(StepContextRepe atCallback.java:76)
        at org.springframework.batch.repeat.support.RepeatTem plate.getNextResult(RepeatTemplate.java:367)
        at org.springframework.batch.repeat.support.RepeatTem plate.executeInternal(RepeatTemplate.java:214)
        at org.springframework.batch.repeat.support.RepeatTem plate.iterate(RepeatTemplate.java:143)
        at org.springframework.batch.core.step.tasklet.Taskle tStep.doExecute(TaskletStep.java:247)
        at org.springframework.batch.core.step.AbstractStep.e xecute(AbstractStep.java:196)
        at org.springframework.batch.core.job.SimpleStepHandl er.handleStep(SimpleStepHandler.java:135)
        at org.springframework.batch.core.job.flow.JobFlowExe cutor.executeStep(JobFlowExecutor.java:61)
        at org.springframework.batch.core.job.flow.support.st ate.StepState.handle(StepState.java:60)
        at org.springframework.batch.core.job.flow.support.Si mpleFlow.resume(SimpleFlow.java:144)
        at org.springframework.batch.core.job.flow.support.Si mpleFlow.start(SimpleFlow.java:124)
        at org.springframework.batch.core.job.flow.FlowJob.do Execute(FlowJob.java:135)
        at org.springframework.batch.core.job.AbstractJob.exe cute(AbstractJob.java:281)
        at org.springframework.batch.core.launch.support.Simp leJobLauncher$1.run(SimpleJobLauncher.java:120)
        at org.springframework.core.task.SyncTaskExecutor.exe cute(SyncTaskExecutor.java:48)
        at org.springframework.batch.core.launch.support.Simp leJobLauncher.run(SimpleJobLauncher.java:114)
        at some.org.batch.poc.partition.MultiFileJobPOCTest.t estLaunchJob(MultiFileJobPOCTest.java:37)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.junit.runners.model.FrameworkMethod$1.runRefle ctiveCall(FrameworkMethod.java:44)
        at org.junit.internal.runners.model.ReflectiveCallabl e.run(ReflectiveCallable.java:15)
        at org.junit.runners.model.FrameworkMethod.invokeExpl osively(FrameworkMethod.java:41)
        at org.junit.internal.runners.statements.InvokeMethod .evaluate(InvokeMethod.java:20)
        at org.springframework.test.context.junit4.statements .RunBeforeTestMethodCallbacks.evaluate(RunBeforeTe stMethodCallbacks.java:74)
        at org.springframework.test.context.junit4.statements .RunAfterTestMethodCallbacks.evaluate(RunAfterTest MethodCallbacks.java:82)
        at org.springframework.test.context.junit4.statements .SpringRepeat.evaluate(SpringRepeat.java:72)
        at org.springframework.test.context.junit4.SpringJUni t4ClassRunner.runChild(SpringJUnit4ClassRunner.jav a:240)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild( BlockJUnit4ClassRunner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner. java:193)
        at org.junit.runners.ParentRunner$1.schedule(ParentRu nner.java:52)
        at org.junit.runners.ParentRunner.runChildren(ParentR unner.java:191)
        at org.junit.runners.ParentRunner.access$000(ParentRu nner.java:42)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRu nner.java:184)
        at org.springframework.test.context.junit4.statements .RunBeforeTestClassCallbacks.evaluate(RunBeforeTes tClassCallbacks.java:61)
        at org.springframework.test.context.junit4.statements .RunAfterTestClassCallbacks.evaluate(RunAfterTestC lassCallbacks.java:70)
        at org.junit.runners.ParentRunner.run(ParentRunner.ja va:236)
        at org.springframework.test.context.junit4.SpringJUni t4ClassRunner.run(SpringJUnit4ClassRunner.java:180 )
        at org.eclipse.jdt.internal.junit4.runner.JUnit4TestR eference.run(JUnit4TestReference.java:46)
        at org.eclipse.jdt.internal.junit.runner.TestExecutio n.run(TestExecution.java:38)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRu nner.runTests(RemoteTestRunner.java:467)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRu nner.runTests(RemoteTestRunner.java:683)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRu nner.run(RemoteTestRunner.java:390)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRu nner.main(RemoteTestRunner.java:197)

        Locked ownable synchronizers:
        - None

        "VM Thread" prio=10 tid=0x00a67400 nid=0xf10 runnable

        "VM Periodic Task Thread" prio=10 tid=0x00a87800 nid=0x59c waiting on condition

        JNI global references: 819
        =======

        Comment


        • #5
          question

          1 more question is, I want to combine multiple files to one file, creating header and footer. Isn't the writing to a new file sequentially (append all other files content) using a separate class better(java 6 nio)? This can be used in a tasklet which can go as a sequential step in the job. Redundant step I see here is converting this to DTO and converting it back to String.
          Your suggestions and comments are welcome.

          Comment


          • #6
            Regarding the hung step: are you using @Transactional in your test case? That would definitely cause a problem, so just remove the annotation. Is that the problem?

            Comment


            • #7
              Problem resolved

              Thanks for pointing this out. Problem solved. Yes. It was @Transactional in junit.

              Comment

              Working...
              X