Announcement Announcement Module
Collapse
No announcement yet.
File reading by multiple nodes from same FS Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • File reading by multiple nodes from same FS

    We have a spring integration application which is running in a cluster on multiple nodes. A NFS drive is mounted across all nodes and is readable by all nodes.

    On NFS there is a folder which is being watched by file:inbound-channel-adapter of all nodes with a polling interval of 5 seconds.

    The problem is 1) One node should not pick up a file which has been picked up by other node

    2) If a file is picked up by a thread on one node, second thread which runs after polling interval should not pick it up- I have in place AcceptOnceFileListFilter for this one, but wondering if there is a better way.

    One of options which we have looked at already:

    1) Create a record in DB with unique filename and every other node checks on same DB before starting processing

    Have to yet look at NIOLocker - but need some inputs on it's reliability and possible options in this scenario.

  • #2
    Hello!

    Is it what are you looking for?
    https://github.com/artembilan/spring...eLockInCluster
    Sorry I've done it on the Spring Integration sources, but it can be simply ported to any application.
    Run it and you'll see that each file is read only once.
    So, what I've found:
    if you use <file:nio-locker/> your file will be locked until you don't kill the process or don't invoke NioFileLocker#unlock: http://static.springsource.org/sprin...l#file-reading
    And before that you can't neither read file not create a new FileChannel on it.

    Nevertheless the JavaDocs of FileLock say the real locks depend of platform.

    HTH

    Take care,
    Artem

    Comment


    • #3
      Hi. I have a similar requirement. Did you come up with a robust solution?

      Thanks,
      Alan.

      Comment


      • #4
        @Alan

        For now we have kept the greedy solution where every node looks at the filesystem and creates a record in DB. Whoever gets to a DB record first gets chance to process the file and rest of them have to abandon processing of the file from that point onwards!

        Eventually here is what our plan:
        - Have a mediator look at file system and create message and push them to a queue.
        - Nodes will get messages from queue - that way one node gets only one file
        - We plan to have the queue implemented in RabbitMQ
        - If you look at EIP book - this is a problem of greed for resources by node- for which best solution is a mediator pattern!

        Hope this helps!

        Comment

        Working...
        X