Announcement Announcement Module
Collapse
No announcement yet.
Restarting in FlatFileItemWriter and LineCount Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Restarting in FlatFileItemWriter and LineCount

    Hi,

    I am trying to implement restart during writing of a FlatFileItemWriter and using the ItemStream to do that. In my update method I save the current line count and on the open method i initialize the "current.count" variable in the ExecutionContext. I have few issues and questions:

    1- Do I have to set this specific "current.count" attribute in the ExecutionContext? Apparently yes as otherwise it doesn't seem to work. Is there any constant or variable or method which represents this. I couldn't find any. RESTART_DATA_NAME is there but it is private (my writer extends FlatFileItemWriter).

    2- While updating this value I have to update the byte size and not the line count. How do I calculate that? I was initially saving (in the update) and using that value (simple line count) on restart (in open) to set the "current.count" attribute in ExecutionContext but that doesn't work and I have to use the actual byte size (something internally which is done by state.position/FileChannel).

    3- It is OK to share values between the 2 different ItemStreams where each represents a reader and a writer respectively. I want the Reader's ItemStream to user values updated by the Writer's Stream and I am using the ExecutionContext passed to the open/update methods to do that for those 2 streams.

    Please advise what am I missing here. Thanks.
    Last edited by haqyunus; Apr 19th, 2011, 03:57 PM.

  • #2
    I didn't really understand the issue. What is it that doesn't work?

    In any case it shouldn't matter that you are extending FlatFileItemWriter - if you store and retrieve your own restart data in the ExecutionContext you can calculate it and call it whatever you like. I suppose you might need to switch off the state-saving in the base class to prevent it from taking control of the file pointer.

    Why do you need to extend FlatFileItemWriter? Nothing illegal about that, but there are a lot of extension points in there already. If you explain the use case a bit we might be able to recommend an alternative approach.

    Comment


    • #3
      I apologize as I got confused myself and hence posted a semi-confused question.

      The requirement is simple: to maintain the last written pointer in the file so we can enable restart.

      I earlier thought that I would have to maintain the file pointer myself. For that purpose I again assumed that I only need to maintain the number of lines written and not the number of bytes written.

      Then I hit the issue that where to copy or store this value so the framework can use it. After poking in the parent class I found that current.count is the key that is used by the framework to maintain the file pointer (and it also maintains bytes and not plain line numbers.)

      Once resolving that, the problem was that how can I calculate the number of bytes written as each line can have different number of records.

      What I did was that in the open method of the Writer (via ItemStream):
      Code:
      if (executionContext.containsKey(getKey(Constants.RESTART_FILE_POINTER))) {
      savedWritePointer = executionContext.getLong(getKey(Constants.RESTART_FILE_POINTER));
      executionContext.putLong(getKey(Constants.CURRENT_FILE_POINTER),savedWritePointer);
      }
      super.open(executionContext);

      and in the update method:

      Code:
      if (executionContext.containsKey(getKey(Constants.CURRENT_FILE_POINTER))) {
      executionContext.putLong(getKey(Constants.RESTART_FILE_POINTER)
                ,executionContext.getLong(getKey(Constants.CURRENT_FILE_POINTER)));
      }
      super.update(executionContext);
      the value of CURRENT_FILE_POINTER is "current.count" as I believe that it is what the framework (at least the FlatFileItemWriter) uses for finding out what is the start point while writing a file.

      Is my understanding/design correct? It seems to be working so far though I haven't tested is extensively?

      By the way what do you mean by not extending from FlatFileItemWriter? Can you please explain that a bit. Thanks.

      Comment

      Working...
      X