Announcement Announcement Module
Collapse
No announcement yet.
Data/node affinity in clustered app Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Data/node affinity in clustered app

    I have a clustered app which allows for N JVMs to be added. I already have discovery and group management taken care of. I want to introduce Spring Integration to simplify my messaging architecture (now written using custom JMS).

    The problem I'm trying to solve is this:

    My "worker" nodes are intended to load and cache expensive data. Worker instance "A" may have a cache for NYC, Worker instance "B" may have a cache for Paris, etc. If a request comes into the system for something about NYC, and NYC's cached data is already on worker "A", I want to route the request to that specific worker.

    How do I model this kind of system in Spring Integration? Should each live worker have its own channel, and use a router for each channel? If so, should I be I dynamically creating channels when nodes join the system and use a DynamicRouter to route to them, and implement affinity that way?

    thanks!

  • #2
    Hello!
    H-m-m...
    It looks like you need Hadoop.

    Take care,
    Artem Bilan

    Comment


    • #3
      @Artem
      It looks like you need Hadoop.
      How can Hadoop fit in here? I'm not a Hadoop expert but what from Hadoop are you suggesting to be used here? MapReduce? Which IMO doesn't fit in here.

      @iangreen
      You seem to be going the right way. You will have one entry point into the system, a router, which will route your request to the appropriate node for processing.

      Comment


      • #4
        Hi, Amol

        I don't try to solve the task. I just look at this as a solution.
        It reminds me HDFS and distributing tasks between nodes...

        Comment


        • #5
          Originally posted by Cleric View Post
          Hello!
          H-m-m...
          It looks like you need Hadoop.

          Take care,
          Artem Bilan
          Nah, gridgain looks much better to me last I checked. Thanks for your suggestion though.

          I decided to use a jms message converter on the local outbound side and selector expression on the remote inbound side. Seems decent enough

          Thanks all

          Comment

          Working...
          X