Announcement Announcement Module
Collapse
No announcement yet.
Appending to a Text File Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Appending to a Text File

    I'm thinking this is more a straight Spring question on the file resource than a question about FlatFileItemWriter.

    I was wondering how to setup a FlatFileItemWriter or the file resource so that writes appends to the flat file.

    There must be an appropriate injectable property for that but I'm not seeing.

    -Paul

  • #2
    Actually the file item writers intentionally only append on restart. (And, it's not a Spring Core concern as yet, because there is no Spring abstraction for output streams, only input streams.)

    Comment


    • #3
      Thanks for the quick response.

      Originally posted by Dave Syer View Post
      Actually the file item writers intentionally only append on restart. ...
      Darn, I was hoping to play with simple things like flat files to understand
      how ItemWriters work and play with tranformations without having to build up an entire world of DB (Hibernate) Writers.

      I was thinking that writing Items to a DB table is like appending, so FlatFileItemWriters would seem to be analogous to a DB.
      This seems like a certain missing parallelism when using a flat file as primary output for a step.

      Since I was doing this with Unit tests, I had hoped to inject the simplest
      ItemWriter that appended its result into a file, I'll have to rethink my little experiment.

      FWIW, I tried a hack to FlatFileItemWriter and added an openForAppend property, added one test to FlatFileItemWriter.OutputState.initializeBufferedW riter()

      if (!restarted && !openForAppend) {
      ...
      }

      And a quick test of a open(), write(), flush(), update(), write(), flush(), close(), open(), write() ... did the right thing. I was able to restore the position in the file to the correct update position at the next open() using the executionContext saved from the update() call.

      You're welcome to tell me where that goes off the rails when used in a Step of a SimpleJob, since my eventual goal is to wire it all up in a Job.

      Comment


      • #4
        I don't exactly see how having files appended helps you to test the features of Spring Batch sans database. If you have a real business use case that requires an append we would maybe add the flag as you describe (if you put it and your tests in a JIRA it will get someone's attention quickly). We probably won't extend this behaviour across the board, as you hoped, since appending to an XML file is likely to be wickedly expensive.

        Comment


        • #5
          Originally posted by Dave Syer View Post
          I don't exactly see how having files appended helps you to test the features of Spring Batch sans database. If you have a real business use case that requires an append we would maybe add the flag as you describe (if you put it and your tests in a JIRA it will get someone's attention quickly). We probably won't extend this behaviour across the board, as you hoped, since appending to an XML file is likely to be wickedly expensive.
          Thanks for the feedback, I'll consider what I might suggest in a JIRA after I get more up to speed on the whole framework. I would have to have a good understanding the appropriate executionContext at open and close, but all that is for another day. At this time I'm not claiming that flat file append is a forgone conclusion.

          A few comments about 'analogousness'. Appending to a text file is analogous to writing to a DB table when an insert adds a row to a table without regard to total rows already in the table and writing a line to a text file adds without regard to the total lines already in the file. Maybe it is just my particular problem space at the moment, but eventually after loading and transforming with multiple tables per step, I eventually will have a merge step which adds to a DB.

          As to extending "across the board"; yes adding any such explicit flag to anything but FlatFileItemWriter would certainly be inappropriate. I certainly agree that appending to an XML file seems unusual. Any such flat file appending might be appropriate as a separate subclass of FlatFileItemWriter, particularly if there's other code required to handle all cases in all states and others see it a a quirky and unusual variation.

          Thanks for the discussion and the interesting framework.

          -Paul

          Comment

          Working...
          X