Announcement Announcement Module
Collapse
No announcement yet.
How to have concurrent message processing while keeping related messages in order Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • How to have concurrent message processing while keeping related messages in order

    I feel like I'm 90% of the way to a solution, but I need a nudge.

    My situation: I have a growing volume of messages that require near-real-time processing. Processing of an individual message normally takes less than one second, but the volume is getting to the point where we're in danger of falling behind during bursts of messages. Most of the time is used up waiting on network I/O, and overall system load is low, so there's plenty of room for concurrent processing to improve throughput.

    The primary constraining factor: Related messages have to be sent out in the same order they're received. "Related" is usually defined as the original source of the message, where there are many (tens of thousands) of sources. There's no set number or sequence of messages that will come from any one source, and messages from a source will often be spread out over days or months but can also come very close together.

    An almost-solution: ActiveMQ's message groups provide support for this exact problem. They give me concurrent processing of unrelated messages along with guaranteed order of related messages by ensuring that related messages (defined by the JMSXGroupID header) are always delivered in order to a single consumer. Then I simply add consumers to add concurrent processing of unrelated messages. I wanted to use this by changing a few direct channels into JMS channels that have concurrent consumers on them. The problem is that SI message headers aren't translated into JMS message headers when a message is sent to a JMS channel, so this approach won't work. Theoretically, I should be able to achieve this using a JMS outbound adapter followed by a JMS message driven adapter, but this is all part of the same message flow and is conceptually just an ExecutorChannel with the additional "grouping" feature.

    What's the "right" way to solve this in Spring Integration?

  • #2
    After some poking and some trial and error, I've come to the preliminary conclusion that all that's really needed is an option in SubscribableJmsChannel to let it apply a JmsHeaderMapper to messages it's sending to the queue, like what's done in the JmsSendingMessageHandler. Unfortunately, that's not something I can easily add from the outside, and it's worth noting here that JmsChannelFactoryBean and the related JMS channel classes aren't very open to extension. I had to do a lot of reflection, some questionable packaging, and too much copy-pasting to make my proof-of-concept work. However it does work. Is there a reason not to add this feature to Spring Integration?

    Comment


    • #3
      This can be done already by simply subclassing the SimpleMessageConverter - after calling super.toMessage(), add the custom header.

      Inject it into the channel using the message-converter attribute.

      Comment


      • #4
        True. That's way easier. I'd still vote for built-in support for header mapping when putting a message on a JMS channel. It seems like a natural and useful thing to do.

        Thanks for the help!

        Comment


        • #5
          Feel free to open an 'Improvement' JIRA issue: https://jira.springsource.org/browse/INT.

          If you really feel strongly, consider submitting a pull request... https://github.com/SpringSource/spri...ONTRIBUTING.md

          Perhaps a 'mapped-headers' attribute (comma delimited list) would be appropriate (it probably doesn't make sense to map all the headers, given that the full message is stored anyway). The mapped-headers would be those headers copied to JmsMessage on the send side, but there'd be no special mapping on the receive side (because all the headers will be restored anyhow).

          Comment

          Working...
          X