Announcement Announcement Module
Collapse
No announcement yet.
Achieving fast consumption rate Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Achieving fast consumption rate

    Hi,

    With RabbitMQ and Spring AMQP, I have to achieve is a faster consumption of each message. The scenario is like this:

    I have to send emails/SMSes to different people after consuming the messages internally. I want to just post a message and consume it faster by having multiple consumers ready. The message should be consumed only by one consumer…

    Should I make a no. of consumers using a shared rabbitmq queue in which each consumer consumes the message posted on the exchange, deletes it from the queue and then after passes that message to a new thread for processing ?

    But having a single queue might hamper the performance.

    Is there any other way which can cater my requirement ? any suggestions?

    Regards,
    Kshitiz

  • #2
    You should probably begin by looking at the support for "Message-driven POJOs" that is part of Spring AMQP. It is modeled after Spring's JMS support and provides the same general options for supporting multiple consuming threads per actual Consumer instance.

    Here's a quick link:
    http://static.springsource.org/sprin...single/#d0e288

    You will want to explore the various properties that can be configured on the MessageListener container.

    Comment


    • #3
      Thanks Mark,

      I am using a message Handler which is just a POJO, thanks to Spring AMQP , but now I am facing a situation. I want to hold acknowledging the message so that it stays live on the rabbitMQ till my POJO's handle message gets completed successfully.

      On rabbitMQ mailing list , I have been told to consume with no_ack=false and then send a basic.ack after the processing is complete. How can we achieve that from this POJO or its configuration ?

      Regards,
      Kshitiz Garg

      Comment


      • #4
        We have transactions covered fully in the RabbitTemplate now, but the asynch use case (message listeners) is not finished yet. However, I think if you set autoAck=false in the message listener container you should see the acks going after the POJO listener has completed.

        Comment


        • #5
          Thanks Dave, I will certainly try that, please let me know later if u come to know about its confirmation

          Comment


          • #6
            Hi,

            Is there any provision that even if I use a POJO message Handler, the queue name property of the associated listener is inserted programatically?

            Regards,
            Kshitiz Garg

            Comment


            • #7
              Originally posted by Mark Fisher View Post
              You should probably begin by looking at the support for "Message-driven POJOs" that is part of Spring AMQP. It is modeled after Spring's JMS support and provides the same general options for supporting multiple consuming threads per actual Consumer instance.

              Here's a quick link:
              http://static.springsource.org/sprin...single/#d0e288

              You will want to explore the various properties that can be configured on the MessageListener container.
              Hi Mark,

              we have a message handler POJO associated with a listener container, If we want different instances/thread of this POJO to handle each new message , is it an automatic process?
              if not, what option I should set on the associated listener..

              Comment


              • #8
                The listener container has a 'concurrentConsumers' property. Here's the JavaDoc from that setter:
                Code:
                /**
                 * Specify the number of concurrent consumers to create. Default is 1.
                 * <p>
                 * Raising the number of concurrent consumers is recommendable in order to
                 * scale the consumption of messages coming in from a queue. However, note
                 * that any ordering guarantees are lost once multiple consumers are
                 * registered. In general, stick with 1 consumer for low-volume queues.
                 */

                Comment


                • #9
                  Thanks a lot mark, that's exactly the thing I am looking for as the order of messages is not important and that will enhance the consumption rate ....

                  Now I have 20 exchanges, and 4 output queues having 20 concurrent consumers consuming messages coming from each of these 4 queues. Do you think that this mismatch between no. of exchanges and queues, (exchanges are many and queues are few) is not good?

                  Actually initially I thought that I will make 4 different type of queues(email/http/sms/ftp) and 4 different types of handlers(email/http/sms/ftp) for each one of those 20 exchanges....

                  This design is trying to scale the no. of POJO handler instances vertically , i.e. across different queues (instead of horizontal scaling in one queue, achieved through concurrentConsumers property).

                  But it seems that this design is not possible as we bind the handlers with queues while configuring the associated listeners beforehand......Can you please suggest any configuration/design changes to achieve this design, if it's possible?

                  Regards,
                  Kshitiz Garg

                  Comment


                  • #10
                    Can you explain the "bindings" you have set up between those Exchanges and Queues?

                    By the way, are you using Spring Integration for the various adapters (mail, ftp, etc)?

                    Comment


                    • #11
                      I am not using spring integration yet.

                      Here is the problem: I have 20 exchanges, and I want to make 4 different type of queues(email/http/sms/ftp) and 4 different types of handlers(email/http/sms/ftp) for each one of those 20 exchanges....

                      I am getting a topicName from outside the system, so I create one exchange , i.e. (topic+"FanOutExchange") like this:

                      FanoutExchange exchange = new FanoutExchange(topicDbObj.getName()+Constants.EXCH ANGE);

                      Then since I have 4 different protocols of subscriber's subscriptions(email/http/sms/ftp), I create 4 queues like this:
                      Queue queue = new Queue(subObj.getProtocol()+Constants.QUEUE);

                      And then I bind them together like this:

                      Binding binding = new Binding(queue, exchange);

                      This adds to 20 exchanges and corresponding 80 queues.

                      Let's move onto POJO handlers' side...

                      I want to have 4 POJO handlers, just Like I have 4 queues, for each exchange, but the irony is this that these 4 POJO handlers should be nothing but different instances of the same 4 classes..How Can I achieve such a setting?

                      Can Spring integration help me here?
                      Last edited by stephanion2002; Oct 13th, 2010, 10:59 AM.

                      Comment


                      • #12
                        Originally posted by Mark Fisher View Post
                        The listener container has a 'concurrentConsumers' property. Here's the JavaDoc from that setter:
                        Code:
                        /**
                         * Specify the number of concurrent consumers to create. Default is 1.
                         * <p>
                         * Raising the number of concurrent consumers is recommendable in order to
                         * scale the consumption of messages coming in from a queue. However, note
                         * that any ordering guarantees are lost once multiple consumers are
                         * registered. In general, stick with 1 consumer for low-volume queues.
                         */
                        Hi Mark,

                        My Handler POJO does not have any specific instance variables initializations except some DIs. If I set concurrent consumers property=20 , then does that mean I will have 20 "different" objects which will not intermingle with the same data ? Do I have to do something else to make this part of my application "thread-safe" ? Should I make my handler POJO's scope=protoype for this ?

                        Thanks,
                        Kshitiz

                        Comment


                        • #13
                          The concurrentConsumers value will not cause multiple instances of your listener's delegate object to be created. It does mean that multiple threads may be invoking that object concurrently.

                          Comment


                          • #14
                            Thanks Mark

                            Comment


                            • #15
                              Hi,
                              Few days back, I had posted this query:

                              Originally posted by stephanion2002 View Post
                              Thanks Mark,

                              I am using a message Handler which is just a POJO, thanks to Spring AMQP , but now I am facing a situation. I want to hold acknowledging the message so that it stays live on the rabbitMQ till my POJO's handle message gets completed successfully.

                              On rabbitMQ mailing list , I have been told to consume with no_ack=false and then send a basic.ack after the processing is complete. How can we achieve that from this POJO or its configuration ?

                              Regards,
                              Kshitiz Garg
                              Dave Syer from springsource had replied to the above quote:

                              We have transactions covered fully in the RabbitTemplate now, but the asynch use case (message listeners) is not finished yet. However, I think if you set autoAck=false in the message listener container you should see the acks going after the POJO listener has completed
                              In my case, I am starting another thread which completes an email sending work, and I want to explicitly set basic.ack=true from this new thread so that I can be sure that messages are going to be deleted from RabbitMQ only after the completion of the message processing.

                              Is there any workaround for this ? Desperately waiting for an answer

                              Many Thanks,
                              Kshitiz Garg

                              Comment

                              Working...
                              X