Announcement Announcement Module
Collapse
No announcement yet.
Memory Issue in production Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Memory Issue in production

    I am using the snapshot version ( i think close to when RC1 released)

    Setup

    2 producers , producing at about 800 msg/sec

    I have have consumers running in 4 different boxes each with 5 threads (5 concurrent consumers in each box)

    I have prefetch set to 10000 and txSize to 10000 cause I didnt want the consumer to fetch too much and run out of memory. Channel is transacted.
    I am using a durable queue, direct exchange.

    I see from the mgmt console that each of the consumer have the load evenly distributed. But with time the memory stamp of the consumer is slowly increasing.It started of using 240 MB and in a weeks time it is close to 1.9 G.

    I profiled the app in dev env and see java.util.concurrent.LinkedBLockingQueue$Node objects growing in size with time.

    Why is it this object is constantly growing? in spite of the mgmnt console showing there are 0 messages to the acknowledged? And also the queue size on the server is 0.

    Thanks in advance,
    Suresh

  • #2
    I am assuming of there are no acks remaining the queue size in the consumer should reduce.

    Dave /Mark any suggestions am I doing something wrong here?

    Comment


    • #3
      10000 is way higher than I would have thought optimum for prefetch or tx size, so you might want to re-think that. But it doesn't really explain the apparent memory leak. If you can get some more detail about who holds the references to the leaking queues it would help a lot. I can do some profiling myself next week, but any extra information you can provide will get us there faster. It would help if you upgraded to the latest snapshot.

      Comment


      • #4
        I did the profiling using latest commit '862920929d656fa33673'. Application (very similar to your described one) worked for a few hours (produced/consumed 12M messages). Did not notice any memory leaks though. It seems memory leak problem is solved. Is any chance you can confirm that?

        /Tomas

        Comment


        • #5
          The issue seems to have been resolved with the latest snapshot, I did notice it in the RC1 version when I upgraded to the latest snapshot its not there anymore

          On a side not why is prefetch value of 10000 not desirable. To one of my earlier queries Dave had replied

          "AMQP has a prefetch limit ("Quality of Service") that you can set on the channel before you consume any messages. Spring AMQP exposes that as a prefetchCount property in the message listener container. But if you are auto-acking my reading is that this has no effect (did you try it?). Maybe you have to consider acking the messages? You can set the txSize (in the current master version of Spring AMQP, or nightly snapshots) in the message listener container quite high to make it more efficient. If it works, but isn't as fast as you like we could consider optimisations. "


          So i set the txsize and prefecth to 10000. I am still unclear on how exactly they both are related.

          Thanks,
          Suresh

          Comment

          Working...
          X