Announcement Announcement Module
Collapse
No announcement yet.
Multiple Steps in a job writing to 1 file Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Multiple Steps in a job writing to 1 file

    Hey,

    What is the best way to setup a job with multiple steps to write to the same file.

    I have 2 steps which read different datasets from the database and I want both of them piped into the same file, but with my current setup the second write seems to overwrite the initial dataset.

    I should also mention that my steps that grabs data from the db and write to file are partitioned.

    I am using
    DB Reader - org.springframework.batch.item.database.JdbcCursor ItemReader
    Custom XML writer that extends StaxEventItemWriter with Marshaller - org.springframework.oxm.xstream.AnnotationXStreamM arshaller
    OutputResource - org.springframework.core.io.FileSystemResource

  • #2
    I think there is a flag in the writer that lets you append if the output already exists. It was added relatively recently if I recall correctly, so you should check your version to see if it is there.

    Comment


    • #3
      Hey Dave,

      Thanks for the reply.

      Are you talking about a flag on the writer? The only one I could find was 'overwriteOutput' which just controls if the writer automatically overwrites an exisiting file or not. I took a look in the source repository at the latest version of StaxEventItemWriter and didn't see anything there.

      If you could point me in the right direction that would be great, sounds like what you're describing is exactly what I need!

      Thanks

      Comment


      • #4
        You are correct. Appending to a file is a new feature (there is already a request logged http://jira.springframework.org/browse/BATCH-1224). You are probably best advised to add a merge step to your job. It should be pretty quick unless the files are enormous, and certainly quicker than any substantive business processing.

        Comment


        • #5
          I think I may just save off the position in the file, like you would for a restart and then append to the file as if its a restart if I have additional data I need to write.

          Append functionality I think generally would be valuable.

          Comment

          Working...
          X