Announcement Announcement Module
Collapse
No announcement yet.
AMQP hanging for unknown reason Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • AMQP hanging for unknown reason

    Hi,


    having a problem with amqp and spring integration. I am running a Spring batch program controlled by amqp messages. Everything goes fine but from time to time the app is straight hanging and doing nothing. Tried a couple of times to find the issue, however none of the threads seam to be deadlocked (according to jconsole at lease). Made a jstack dump, maybe someone can help me figure it out. I was thinking that opening shell executions was the root of the problem for the failures but I've build in a safety feature that kills any hanging shell process that takes too long (with junit test so I'm pretty confident it's working). This is occurring not very often but still often enough to hang the application once or twice a week.
    The queue is piling up and there are non-acked messages (the ones that seam to be hanging) in it all the time. The application can't even be closed via ctrl+c, have to kill it from the system.

    The dump was taken after ctrl+c was pressed so there's less going on I believe.

    Any clues anyone? Would appreciate any advice (or maybe there's some smarter tool to figure it out?)

    thanks,
    Piotr

  • #2
    Hi!

    Maybe this issue on Linux where your RabbitMQ works:
    http://blog.thecodingmachine.com/con...er-application
    BTW, you can see node metrics from home page of Rabbit Web Admin Console.

    HTH,
    Artem

    Comment


    • #3
      Don't think its the file and/or socket descriptors. I have a default limit on 1024 and currently use just 4 channels at the same time and about 25 file desctiptors (out of 1024) so still way in the green.

      Comment


      • #4
        What version of Spring-AMQP and rabbit client are you using?

        It looks to me like thread-2 is waiting infinitely for a response to a close().

        "Thread-2" prio=10 tid=0x00007f5260061800 nid=0x50c1 in Object.wait() [0x00007f5253183000]
        java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at com.rabbitmq.utility.BlockingCell.get(BlockingCell .java:50)
        - locked <0x00000000e11d19a8> (a com.rabbitmq.utility.BlockingValueOrException)
        at com.rabbitmq.utility.BlockingCell.get(BlockingCell .java:65)
        - locked <0x00000000e11d19a8> (a com.rabbitmq.utility.BlockingValueOrException)
        at com.rabbitmq.utility.BlockingCell.uninterruptibleG et(BlockingCell.java:111)
        - locked <0x00000000e11d19a8> (a com.rabbitmq.utility.BlockingValueOrException)
        at com.rabbitmq.utility.BlockingValueOrException.unin terruptibleGetValue(BlockingValueOrException.java: 37)
        at com.rabbitmq.client.impl.AMQChannel$BlockingRpcCon tinuation.getReply(AMQChannel.java:349)
        at com.rabbitmq.client.impl.AMQConnection.close(AMQCo nnection.java:777)
        ...
        This is probably a question for the Rabbit folks over on their mailing list.

        (I think Spring-AMQP could use a flavor of the close that specifies a timeout, which (according the rabbit javadocs) would force the socket to close.

        Comment


        • #5
          Hi Gary,

          thanks for the info, seams like I'm pretty much sentenced to fight with their mailing list...

          Should I open a JIRA issue for the second thing (so a SI AMQP safety feature I guess) ?

          Comment


          • #6
            Please do open a JIRA; I am hoping to get 1.2 RC1 out this week, with GA to follow shortly so I should be able to fit this in.

            Comment


            • #7
              Amazing,

              thanks a lot Gary!

              Opening issue now.

              Comment

              Working...
              X