Announcement Announcement Module
Collapse
No announcement yet.
What diagnostic action to take when Maven hangs ? Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • What diagnostic action to take when Maven hangs ?

    Hi,

    I have a Maven build that hangs and I wonder what to do to get some feedback like a message or stack trace or dump..

    I'm stuck trying to run some integration tests http://forum.springsource.org/showth...d=1#post426888

    Thanks

  • #2
    If you're on linux / mac:

    1. in the hanging shell, hit CTRL-Z - this will suspend the program and give you a pid
    2. type bg [ENTER] - this will send the program to the background
    3. Do a kill -3 of the process - it will dump a large amount of text - make sure your command line terminal history window is long enough to capture it all - will be several thousand lines of text
    (alternative to #3 - you can try jstack with the pid, it is a Java stack trace generator)
    4. You can see what threads are waiting on other threads from there (maybe, sometimes it's a tight CPU loop)

    One best practice is to do several thread dumps in a row - and compare where the threads are hanging. Chances are with Maven you only have a few threads running at most since it is a build tool, but if you are running Spring integration tests, then the Maven tool may have fired up Spring and spawned threads in that VM.

    Another option, which will run very slowly, but will work, is to run mvnDebug instead of mvn when you run it. It should sit there paused, waiting for you to connect. You could try connecting with jvisualvm or jconsole and watching the heap, cpu, etc., from there. You'd need to make something stop suspending it, so you can always hook up your IDE, like Eclipse, to it using remote debugging on the same JVM port that it exposes.

    Ken

    Comment


    • #3
      Hi Ken,

      First of all, let me say one thing: Lucky were Ken's co-workers.

      I ran your commands:

      -------------------------------------------------------
      T E S T S
      -------------------------------------------------------
      ^Z
      [1]+ Stopped mvn clean test
      stephane@stephane-ThinkPad-X60:learnintouch> bg
      [1]+ mvn clean test &
      stephane@stephane-ThinkPad-X60:learnintouch>

      But it didn't show a process number.

      Here is the process number from another terminal:

      stephane 4798 1749 99 00:32 pts/1 00:02:49 /home/stephane/programs/jdk1.6.0_29/bin/java -class
      stephane 4822 4798 0 00:33 pts/1 00:00:00 /bin/sh -c cd /home/stephane/dev/java/projects/spri
      stephane 4824 4822 95 00:33 pts/1 00:00:22 /home/stephane/programs/jdk1.6.0_29/jre/bin/java -j
      stephane 4837 1759 0 00:34 pts/2 00:00:00 ps -ef

      And the kill command showed:

      stephane@stephane-ThinkPad-X60:learnintouch> kill -3 4824
      stephane@stephane-ThinkPad-X60:learnintouch> 2012-10-10 00:36:09
      Full thread dump Java HotSpot(TM) Server VM (20.4-b02 mixed mode):
      "Low Memory Detector" daemon prio=10 tid=0x08fce400 nid=0x12e3 runnable [0x00000000]
      java.lang.Thread.State: RUNNABLE
      "C2 CompilerThread1" daemon prio=10 tid=0x08fcc800 nid=0x12e2 waiting on condition [0x00000000]
      java.lang.Thread.State: RUNNABLE
      "C2 CompilerThread0" daemon prio=10 tid=0x08fca800 nid=0x12e1 waiting on condition [0x00000000]
      java.lang.Thread.State: RUNNABLE
      "Signal Dispatcher" daemon prio=10 tid=0x08fc9000 nid=0x12e0 waiting on condition [0x00000000]
      java.lang.Thread.State: RUNNABLE
      "Finalizer" daemon prio=10 tid=0x08fb7400 nid=0x12df in Object.wait() [0x7fd77000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x84455fa8> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue .java:118)
      - locked <0x84455fa8> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue .java:134)
      at java.lang.ref.Finalizer$FinalizerThread.run(Finali zer.java:159)
      "Reference Handler" daemon prio=10 tid=0x08fb5c00 nid=0x12de in Object.wait() [0x7fdc8000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x84455f88> (a java.lang.ref.Reference$Lock)
      at java.lang.Object.wait(Object.java:485)
      at java.lang.ref.Reference$ReferenceHandler.run(Refer ence.java:116)
      - locked <0x84455f88> (a java.lang.ref.Reference$Lock)
      "main" prio=10 tid=0x08f36400 nid=0x12da runnable [0xb6976000]
      java.lang.Thread.State: RUNNABLE
      at java.lang.ClassLoader.findLoadedClass0(Native Method)
      at java.lang.ClassLoader.findLoadedClass(ClassLoader. java:949)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:2 91)
      - locked <0x84444dd8> (a sun.misc.Launcher$AppClassLoader)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launche r.java:301)
      - locked <0x84444dd8> (a sun.misc.Launcher$AppClassLoader)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:2 47)
      at org.springframework.core.type.classreading.Recursi veAnnotationAttributesVisitor.visitEnd(AnnotationA ttributesReadingVisitor.java:167)
      at org.springframework.asm.ClassReader.a(Unknown Source)
      at org.springframework.asm.ClassReader.accept(Unknown Source)
      at org.springframework.asm.ClassReader.accept(Unknown Source)
      at org.springframework.core.type.classreading.SimpleM etadataReader.<init>(SimpleMetadataReader.java:54)

      I can see some threads are waiting for another one.

      Sorry I haven't done that before so I'm not the best hunter in the wild.

      Tomorrow I shall try to have a harder look at it and use the mvn debugger as well.

      Thank you.

      Comment


      • #4
        I can run the debugger all right with the command:

        Code:
        stephane@stephane-ThinkPad-X60:learnintouch> mvnDebug -DforkMode=never test -Dtest=AddressIntegrationTest
        Preparing to Execute Maven in Debug Mode
        Listening for transport dt_socket at address: 8000
        I can run the debug configuration in Eclipse and it attaches to the remote debugger.

        But then nothing much happens within Eclipse and I don't know how to step in the source code.

        I attached a screenshot of the Eclipse pane.

        Attachment
        Attached Files

        Comment


        • #5
          Stephane,

          Did you set a breakpoint in your test?

          You can right-click on the thread in the debugger and suspend it - I'd start with the main thread. My theory (based on absolutely nothing but gut instinct) is that you have a Spring config that is loading and takes forever, perhaps running out of memory on the way up. If you right-click on the main thread and pause it, likely you'll be in some area of the configuration.

          Comment


          • #6
            Hi Ken,

            I did pause the thread and set a break point in my first test but it was never reached.

            I suppose to see what is going on with the Spring Roo configuration I would need to have access to its source code for the debugger...

            Comment


            • #7
              I have 3 Gb of ram on a Linux Mint box. Strange that this would not be enough for Maven to build a Spring Roo app.

              EDIT

              Maybe there is some way to tell Spring to be slow and memory conservative than fast and memory intensive...

              Comment


              • #8
                Originally posted by stephaneeybert View Post
                Hi Ken,

                I did pause the thread and set a break point in my first test but it was never reached.

                I suppose to see what is going on with the Spring Roo configuration I would need to have access to its source code for the debugger...
                huh... you should be able to download the source - Spring distributes a Roo Java source jar on Maven central so if you get a thread to pause you should be able to double click on the stack frame of interest. Same with the Spring Framework and many others if it asks to download the source just click to make it do so.

                Clearly there is something you are triggering that should be looked at.

                Comment


                • #9
                  If I understood the idea of Roo being a tool and not a framework, the Roo source code is not needed in the debugging process as Roo is simply absent from the application at run time.

                  Maven has downloaded the source code of Spring and this source code is where I should look to see where it hangs.

                  The thing is I don't know where to put any break point.. not sure it is even in Spring I should look into.

                  Sorry for my modest ability on this call.
                  Last edited by stephaneeybert; Oct 18th, 2012, 04:44 AM.

                  Comment


                  • #10
                    Aargh. Sorry! I was up late prepping for an add-on presentation at SpringOne and you did ask me a question about source code.... My mind is not so clear at that time of night.

                    The Spring framework source is definitely available. So if it is happening in the container startup, you should be able to see. The same with JPA, Hibernate, etc.

                    Can you suspend it? Any info on what methods/classes it stops on?

                    Don't worry about a breakpoint- just right click on the threads in the debugger and click the option to suspend the thread. I would start with the main one.

                    Ken


                    Originally posted by stephaneeybert View Post
                    If I understood the idea of Roo being a tool and not a framework, the Roo source code is not needed in the debugging process as Roo is simply absent from the application at run time.

                    Maven has downloaded the source code of Spring and this source code is where I should look to see where it hangs.

                    The thing is I don't know where to put any break point.. not sure it is even in Spring I should look into.

                    Sorry for my modest ability on this call.

                    Comment


                    • #11
                      Hi Ken,

                      I'm the one who is sorry to see I disturb someone of your value to the Spring community with my little 2 cents issue.

                      There is no hurry on my side as this is a personal hobby project.

                      Yes, I was able to suspend the main thread easily.

                      I don't really know what this suspend means though. From your last message I understand the thread will ease in and stop at a method which should tell us something. That means it will not stop at a random location based on exactly when I suspend it..?

                      Let me try to suspend it again and see which method comes up.

                      Comment


                      • #12
                        I did suspend the main thread.

                        I took some snapshots hoping they would tell you more than my words...

                        Attachment

                        Attachment

                        Attachment

                        Attachment
                        Attached Files

                        Comment


                        • #13
                          I also bought another laptop computer, installed a different Linux on it, Lubuntu 12.10 and installed Maven 3 and the Java Spring Roo stack.

                          I ran the mvn clean test command and it hanged at exactly the same point and in the same fashion.

                          Quite consistent.

                          Comment


                          • #14
                            Hi Ken,

                            Some update on the issue I had. Indeed Maven was hanging, but it was due to another known issue I stumbled upon in the past.

                            See my latest post at http://forum.springsource.org/showth...ple-Roo-script

                            Kind Regards,

                            Comment

                            Working...
                            X