Announcement Announcement Module
Collapse
No announcement yet.
tcserver - Application does not start up (Imbalanced frame stack exception) Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • tcserver - Application does not start up (Imbalanced frame stack exception)

    I have successfully installed the tcserver and tried mulitple smaller applications and was able to view the insight dashboard.
    However when I deployed my application (real), it doesn't start up and on the insight dashboard there are two request context start & context destroyed.

    I searched and tried to increase the insight.max.frames to 6000 and even more than 20000 but still the error persists.

    Any help is appreciated.

    In the log I get the following error
    ERROR 2012-03-12 11:24:37,890 [pool-3-thread-1] TransactionInterceptor.completeTransactionAfterThr owing(419) | Application exception overridden by rollback exception
    java.lang.IllegalStateException: Imbalanced frame stack! (exit() called too many times)
    at com.springsource.insight.intercept.trace.ThreadLoc alFrameBuilder.exit(ThreadLocalFrameBuilder.java:6 1)
    at com.springsource.insight.collection.DefaultOperati onCollector.exit(DefaultOperationCollector.java:11 1)
    at com.springsource.insight.collection.DefaultOperati onCollector.exitAbnormal(DefaultOperationCollector .java:85)
    at com.springsource.insight.plugin.jdbc.JdbcPreparedS tatementOperationCollectionAspect.ajc$afterThrowin g$com_springsource_insight_plugin_jdbc_JdbcPrepare dStatementOperationCollectionAspect$6$bef8f336(Jdb cPreparedStatementOperationCollectionAspect.aj:120 )
    at org.apache.commons.dbcp.DelegatingPreparedStatemen t.executeQuery(DelegatingPreparedStatement.java:10 0)
    at org.hibernate.jdbc.AbstractBatcher.getResultSet(Ab stractBatcher.java:208)
    at org.hibernate.loader.Loader.getResultSet(Loader.ja va:1869)
    at org.hibernate.loader.Loader.doQuery(Loader.java:71 8)
    at org.hibernate.loader.Loader.doQueryAndInitializeNo nLazyCollections(Loader.java:270)
    at org.hibernate.loader.Loader.loadEntity(Loader.java :1953)
    at org.hibernate.loader.entity.AbstractEntityLoader.l oad(AbstractEntityLoader.java:86)
    at org.hibernate.loader.entity.AbstractEntityLoader.l oad(AbstractEntityLoader.java:76)
    at org.hibernate.persister.entity.AbstractEntityPersi ster.load(AbstractEntityPersister.java:3270)
    at org.hibernate.event.def.DefaultLoadEventListener.l oadFromDatasource(DefaultLoadEventListener.java:49 6)
    at org.hibernate.event.def.DefaultLoadEventListener.d oLoad(DefaultLoadEventListener.java:477)
    at org.hibernate.event.def.DefaultLoadEventListener.l oad(DefaultLoadEventListener.java:227)
    at org.hibernate.event.def.DefaultLoadEventListener.o nLoad(DefaultLoadEventListener.java:147)
    at org.hibernate.impl.SessionImpl.fireLoad(SessionImp l.java:1080)
    at org.hibernate.impl.SessionImpl.immediateLoad(Sessi onImpl.java:1018)
    at org.hibernate.proxy.AbstractLazyInitializer.initia lize(AbstractLazyInitializer.java:176)
    at org.hibernate.Hibernate.initialize(Hibernate.java: 428)

    Caused by: java.lang.IllegalStateException: Imbalanced frame stack! (exit() called too many times)
    at com.springsource.insight.intercept.trace.ThreadLoc alFrameBuilder.exit(ThreadLocalFrameBuilder.java:6 1)
    at com.springsource.insight.collection.DefaultOperati onCollector.exit(DefaultOperationCollector.java:11 1)
    at com.springsource.insight.collection.DefaultOperati onCollector.exitNormal(DefaultOperationCollector.j ava:67)
    at com.springsource.insight.plugin.springtx.Transacti onOperationCollectionAspect.ajc$afterReturning$com _springsource_insight_plugin_springtx_TransactionO perationCollectionAspect$3$2114d8d(TransactionOper ationCollectionAspect.aj:69)
    at org.springframework.transaction.support.AbstractPl atformTransactionManager.rollback(AbstractPlatform TransactionManager.java:823)
    at org.springframework.transaction.interceptor.Transa ctionAspectSupport.completeTransactionAfterThrowin g(TransactionAspectSupport.java:411)
    at org.springframework.transaction.interceptor.Transa ctionInterceptor.invoke(TransactionInterceptor.jav a:114)
    at org.springframework.aop.framework.ReflectiveMethod Invocation.proceed(ReflectiveMethodInvocation.java :172)
    at org.springframework.aop.interceptor.ExposeInvocati onInterceptor.invoke(ExposeInvocationInterceptor.j ava:89)
    at org.springframework.aop.framework.ReflectiveMethod Invocation.proceed(ReflectiveMethodInvocation.java :172)
    at org.springframework.aop.framework.JdkDynamicAopPro xy.invoke(JdkDynamicAopProxy.java:202)
    at $Proxy95.searchMovies(Unknown Source)

  • #2
    In Insight version 1.5, the property for setting the max frames is actually: "insight-max-frames" - we changed this to "insight.max.frames" for future releases but our documentation is inconsistent.

    What this normally means:
    Most likely your real application probably makes a number of sequential calls to a database during initialization which exceeds our default of 1000 frames. The error message is innocuous, the particular trace generated from that set of calls is discarded and the monitoring should proceed normally.

    What concerns me:
    You wrote that: "However when I deployed my application (real), it doesn't start up and on the insight dashboard there are two request context start & context destroyed."

    Just to clarify, are you saying that there are two traces from your real application, context start/stop; but that your application doesn't start at all? This seems to indicate that the agent caught a handful of traces during deployment and undeploy. I would make sure the application works as expected w/o the agent, then I would test with just the agent setenv.sh settings to make sure our changes to the instance setenv do not affect your application (but no insight-agent webapp).

    Comment


    • #3
      Thanks for the response John

      I tried with -Dinsight-max-frames (as you suggested above) changing it from my previous setting of -Dinsight.max.frames but still the error persists
      I changed this property in setenv.sh adding as one of the JVM paramter. Please let me know if i need any other change or the above setting is not correct.

      In the application that I am deploying (we are using hibernate) and there are lot of database call at startup to cache the data.

      When I said that the application shows two context start/stop. It means my application context is destroyed as soon as it starts. I tried my other application which doesn't have that many database calls and it deploys correctly and shows the trace on the dashboard.

      FYI :- the version of tc server I am using is vfabric-tc-server-developer-2.6.3.RELEASE.

      Any other help or information is appreciated.


      Thanks for your response.



      Regards
      Ashok Soni
      Last edited by [email protected]; Mar 13th, 2012, 12:05 PM.

      Comment


      • #4
        Also just to update after some time on the dashboard, I am able to see the initial trace (start up calls) as well along with the context destroyed and unable to access my application after that.

        Comment


        • #5
          I see the issue (or at least part of it). We fixed this some time ago after branching for 1.5.1, but while the fix was a candidate for the release branch but we decided not to merge it. The change is relatively simple and has proven reliable through a couple of different releases (AppInsight 1.0 and our CloudFoundry release).

          If you want, I can do a support release of the 1.5.1 agent with this fix which you can test. If it works, I can make sure the requisite service release is created and available. The next version of vFabric/Insight 1.8.3 should be OK.

          You may still see the max frames exception, but should be innocuous to your application.

          What do you think?

          Comment


          • #7
            Download:

            http://maven.springframework.org/rel...-1.5.1.SR5.zip

            Backup the existing insight-agent template and extract this into the templates directory. Then just create your template as you normally would.

            As I said, this will still probably print a message when your application starts up about frame counts, but you can ignore it. Please let me know how this works.

            Comment


            • #8
              Thanks for your reply but I got that resolved by increasing the insight-max-frames to a much higher value (around 750000)

              But what I am noticing is that there is a big change in the response time reported by spring insight and when I run the test directly on tomcat and measure the load time on apache jmeter (it seems that there is an overhead of more than 100%). May be I might not have configured any parameter or something....

              Thanks once again

              Regards
              Ashok Soni

              Comment


              • #9
                While Insight does have an impact on the response time, it shouldn't be that high - that said it really depends on the application. If most of your requests are extremely short-lived in the first place the additional overhead of monitoring those requests is not worth the effort. You can exclude certain urls from being monitored in the configuration file. Other performance concerns include how many particular plugins are hit by your application. Basic servlet applications with no JDBC hit fewer insight plugins, and have a much lower overhead.

                In any case, we are creeping up on the next vFabric release, which includes Insight 1.8.3 - When compared to 1.5.1 this version has many more knobs to tune for performance; and even though it's a bit far off I'm really excited about the release after that.

                Comment

                Working...
                X