Announcement Announcement Module
No announcement yet.
Serial reading by many consumers off one queue Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Serial reading by many consumers off one queue

    Hi there

    I have a rabbitmq cluster with a queue that I retrieve messages off. I have a consumer that runs, for redundancy, on two servers. The consumer handles a message and updates a value in memcached that a web application will read.

    Currently, only half of the messages' data ends up in memcached since each consumer updates memcached while the other consumer is reading a message.

    I am trying to figure out how to configure my queue to only allow one consumer to read a message at a time, and only allow the next message to be popped after the consumer is finished processing.

    I have tried setting acknowledgeMode=MANUAL and prefetchCount=1 and then using Channel.basicAck(), but the next message is still provided to the other consumer before I have ack'd the current message.

    This is the first time I am working with spring and rabbit and I am probably missing something.

    Any suggestions how I can have two consumers on one queue reading in serial?

    Thanks in advance for any help.


  • #2
    If you have a single queue with two consumers, the broker will only send each message to one of the consumers. Prefetch is 1 by default.

    I am not familiar with memcached - are you saying it can't handle concurrent updates? Seems strange.

    There is no way (AFAIK) to configure Rabbit to hold off sending a message to consumer2 until consumer1 has acked his message. And, I'm not sure why you'd want to do that - you'd be better off having one server handle all the work and fail over to the other one if the first one goes down.

    Maybe I'm misunderstanding your question.


    • #3
      Thanks for the prompt reply.

      The data we keep does not live in a database, it is created as users use the system and we keep a list of the N most recent events. As an event occurs, the php web application writes a message to a rabbit queue. These messages are then picked up by the consumer and added to the list in memcache, removing the oldest event and adding the newly received event occurrence.

      What happens is the following:

      Event occurs in web app (e.g. some prize was won)
      Message added to queue
      Consumer1 picks up message
      Consumer1 reads list of events from cache, removes oldest item, adds newest one and saves data back to cache.
      While the above step is happening, another message arrives.
      Consumer2 gets this message and does the same as above.
      Consumer1 finishes its work, writes data back to the cache and acks the message.
      Consumer2 finishes its work, writes data back to the cache and acks the message.

      The problem is thus that Consumer2 overwrites the data just set into cache by Consumer1.

      So I was wondering if spring has some configuration for a SimpleMessageListenerContainer to only get the next message once the queue is in a "i am not awaiting any acks" state.


      • #4
        Sounds like you need a cache that supports transactions. Or you need to implement your own optimistic locking scheme - looks like memcached has a CAS operator that might help you...

        Like I said, I am not aware of any way to configure a queue so it won't deliver a message to a second consumer while it's waiting for an ack from the first one.


        • #5
          You also might want to ask your question on the Rabbit list since this is Rabbit question rather than a Spring-AMQP question (and the Rabbit experts are over there - they might have another solution for you).


          • #6
            Thank you very much for the quick help, Gary!

            We're looking at approaching the problem from a different angle, I was indeed barking up the wrong tree.