Announcement Announcement Module
Collapse
No announcement yet.
ParallelJob/Multithreading - Sample - Clarification Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • ParallelJob/Multithreading - Sample - Clarification

    Hi,

    I went through the paralleljob sample and following is my understanding.
    1. The "loading" step is multithreaded using "taskExecutor".
    2. The "stagingReader", "stagingProcessor" & "tradeWriter" will be executed
    sequentially in a given thread.
    3. The primary keys are retrieved first using synchronization and then
    read, process, write is performed.

    The question that i have is:
    1. Can reading the primary keys itself be multithreaded? I am not sure,
    whether my question will be meaningful, but it just a thought in my
    mind.
    2. If there are million records to be processed, will there be any performance
    hit while reading the primary keys in one single shot? Or can this be
    read in chunks and then be multithreaded?
    3. For flat file input, is multithreading possible?

    Thanks.

    --venkat

  • #2
    Anyone can help on this thread?

    --venkat

    Comment


    • #3
      Try to be patient. The best way to get an answer is actually to leave your post with 0 replies.

      1. The "loading" step is multithreaded using "taskExecutor".
      Yes.
      2. The "stagingReader", "stagingProcessor" & "tradeWriter" will be executed
      sequentially in a given thread.
      Yes.
      3. The primary keys are retrieved first using synchronization and then
      read, process, write is performed.
      Yes. There is also a synchronization in the read() method of the reader, so that multiuple threads don't get the same item.
      1. Can reading the primary keys itself be multithreaded? I am not sure,
      whether my question will be meaningful, but it just a thought in my
      mind.
      Anything is possible. You could load the keys in a separate thread and allow the processing to start before they are complete, for instance. I doubt you could process them much faster, but if you try it and it helps then be sure to let us know.
      2. If there are million records to be processed, will there be any performance
      hit while reading the primary keys in one single shot? Or can this be
      read in chunks and then be multithreaded?
      You could strategise the reading for sure. Our experience is that it isn't worth the effort until you get to the point where you are prepared to put some more effort into it, and then you probably want to use a PartitionHandler anyway (then the StagingItemReader is redundant). A million records is pretty small beer.
      3. For flat file input, is multithreading possible?
      In principle. In practice we don't support it (yet) because it is very difficult to store the restart data if items are not processed sequentially.

      Comment


      • #4
        Thanks a lot for your reply Dave.

        And kindly apologize for replying to my own thread. I was expecting your
        reply, but since there was a delay, i replied to the post. Anyway apologize.

        Yes, sure i will implement this and will let you all know whether multithreading
        in reading primary keys works out or not.

        Thanks
        venkat

        Comment

        Working...
        X