Announcement Announcement Module
Collapse
No announcement yet.
outof memory issue Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • outof memory issue

    Hi

    I have an app that is using spring integration to connect over tcp to a 3rd party system. Is needs to send the message over tcp get the response and send it back out of my system. I need to be sure that i send a request over tcp and wait for the response before sending another.

    To test this i send http requests and use the information to generate an object and send it through the tcp connection.

    I am getting an increasing number of threads associated with my applications process that eventually results in this.

    PHP Code:
    19-May-2010 09:40:28 org.apache.catalina.core.StandardWrapperValve invoke
    SEVERE
    Servlet.service() for servlet mybenchapp threw exception
    java
    .lang.OutOfMemoryErrorunable to create new native thread
        at java
    .lang.Thread.$$YJP$$start0(Native Method)
        
    at java.lang.Thread.start0(Thread.java)
        
    at java.lang.Thread.start(Thread.java:597)
        
    at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:703)
        
    at java.util.concurrent.ThreadPoolExecutor.prestartCoreThread(ThreadPoolExecutor.java:1373)
        
    at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:223)
        
    at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:366)
        
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.schedule(ReschedulingRunnable.java:73)
        
    at org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.schedule(ThreadPoolTaskScheduler.java:164)
        
    at org.springframework.integration.endpoint.AbstractPollingEndpoint.doStart(AbstractPollingEndpoint.java:187)
        
    at org.springframework.integration.endpoint.AbstractEndpoint.start(AbstractEndpoint.java:84)
        
    at org.springframework.integration.config.ConsumerEndpointFactoryBean.start(ConsumerEndpointFactoryBean.java:179)
        
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:164)
        
    at org.springframework.context.support.DefaultLifecycleProcessor.access$1(DefaultLifecycleProcessor.java:153)
        
    at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:331)
        
    at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:142)
        
    at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:107)
        
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:885)
        
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:426)
        
    at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
        
    at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
        
    at com.securetrading.controllers.HelloController.sendToGateway(HelloController.java:114)
        
    at com.securetrading.controllers.HelloController.handleRequest(HelloController.java:58)
        
    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
        
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:771)
        
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:716)
        
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
        
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)
        
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
        
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
        
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        
    at java.lang.Thread.run(Thread.java:619

    when i profile my app i can see that loads of threads are getting crated with the name task-scheduler-1 through to 9. after looking for a reference to this name in the code i found that DefaultConfigurationBeanFactoryPostProcessor does this work when i dont define my own taskScheduler. what im not sure about is why there are hundereds of these threads with the same name task-scheduler-1 through to 9. from what i can see ThreadPoolExecutor.addIfUnderCorePoolSize is not getting met but the pool size default is 10.

    Can anyone tell me why this is happening?

    this is my config
    PHP Code:


                  
    <si:gateway id="gateway"
                    
    service-interface="my.class.GatewayInterface"
                    
    default-request-channel="input"
                    
    default-reply-channel="transformedChannel"/>
        
        <
    si:channel id="input" />

        
        <
    si:service-activator input-channel="input"
                              
    output-channel="requestChannel" 
                              
    ref="transactionEncoder" 
                              
    method="processTransaction" />

        <
    bean id="transactionEncoder"
              
    class="my.class.EncodingService" />



                     <
    ip:outbound-gateway id="tcpOutGateway" 
                              
    host="10.3.21.28"
                             
    port="40033" 
                             
    message-format="custom" 
                             
    request-channel="requestChannel"
                             
    reply-channel="replyChannel" 
                             
    custom-socket-writer-class-name="my.class.SocketWriter"
                             
    custom-socket-reader-class-name="my.class.SocketReader"/>

        <
    si:channel id="replyChannel">
            <
    si:queue />
            
        </
    si:channel>

    <
    si:transformer ref="transactionDecoder" 
                        
    input-channel="replyChannel"
                        
    output-channel="transformedChannel"  
                        
    method="transform"/>
     
     <
    bean id="transactionDecoder"
              
    class="my.class.DecodingService" />
        

    <
    si:channel id="transformedChannel">
        <
    si:queue capacity="5"/>
        </
    si:channel>


    <
    si:poller max-messages-per-poll="1" 
                   
    id="defaultPoller"
                   
    default="true">
            <
    si:interval-trigger interval="30000" />
        </
    si:poller
    My thoughts are that this might be something to do with the way my poller is working.

    Im using 2.0.0M4
    Last edited by leeo; May 19th, 2010, 07:09 AM.

  • #2
    If you only see threads with names between 1 and 9, then it seems like you are indeed staying under 10 threads. Are you seeing the number of threads elsewhere (e.g. JConsole) or just in the log?

    Also, what version of Spring Integration are you using?... and is this a production application or something in-development?

    Thanks,
    Mark

    Comment


    • #3
      im using 2.0.0M4 this is in developement at the moment so i can change any of my config or even my solution as a whole. when i look at the systems processes i see my tomcat java.exe running the more requests i send to the app the more the number of threads grows and blows up out of memory at about 2500. Its as if no threads are ever released

      im using YourKit Java Profiler and i see like 300 threads each with the name task-scheduler-1, another 300 odd for task-scheduler-2 and so on up to 9. should i not just have 9 in total or am i missing something

      Comment


      • #4
        Those are thread names, so they are unique, right? There should never be more than one thread named e.g. "task-scheduler-5". Are you sure that you are seeing hundreds of *active* threads?

        By the way, you might want to consider our new TCP gateway support:
        http://static.springsource.org/sprin...l#tcp-gateways

        Comment


        • #5
          Judging from the stacktrace you are creating your own instance of an ApplicationContext... For EVERY request that comes in... Why? If you want to run out of resources that is the way to go...

          Code:
              
              at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:885)
              at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:426)
              at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
              at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
          
          Last edited by Marten Deinum; May 19th, 2010, 08:56 AM.

          Comment


          • #6
            As far as i am aware i was using the new tcp gateway support i see this in the log when its running
            PHP Code:
            10457 [http-8080-1DEBUG org.springframework.integration.ip.tcp.SimpleTcpNetOutboundGateway  handler 'org.springframework.integration.ip.tcp.SimpleTcpNetOutboundGateway#0' sending reply Message: [Payload=[B@23a1fd][Headers={$replyChannel=org.springframework.integration.channel.MessageChannelTemplate$TemporaryReplyChannel@1f025cb$errorChannel=org.springframework.integration.channel.MessageChannelTemplate$TemporaryReplyChannel@1f025cb$timestamp=1274273603242$history=[gatewaychannel:inputchannel:requestChanneltcpOutGateway], $id=11abb638-2836-4dc7-9559-b89f79193f8c}] 
            as for the threads issue i see them in yourkit java profiler and jconsole

            please review the attached images for screen shots of what i see

            Comment


            • #7
              See my previous post...

              Comment


              • #8
                Ahh this could indeed be bad i will try to stop doing that now marten.


                what i had in my code is
                PHP Code:

                ApplicationContext context 
                = new ClassPathXmlApplicationContext("si-config.xml");
                GatewayInterface gateway =  (GatewayInterfacecontext.getBean("gateway"); 
                i take it i only need to do this once and not for every request

                Comment


                • #9
                  You should never do it .

                  Let spring load the configuration file and inject the gateway into your controller...

                  Comment


                  • #10
                    ahh right i just got that from some example i will have a look how to do as you suggest now. cheers

                    Comment


                    • #11
                      Sorry, I didn't see anything related to the TCP adapter in your configuration, so I assumed you were rolling your own.

                      Marten is definitely right though. You don't want to create a new ApplicationContext at runtime.

                      Comment


                      • #12
                        Hi can you give me some pointers on how to do this ie how do i tell spring about my config file si-config.xml

                        also with injection do you mean annotate my controller with something like


                        Code:
                        @Autowired
                        private GatewayInterface gateway;

                        also i changed my GatewayInterface method adding
                        @Gateway(requestChannel = "input")

                        now in the code my gateway instance is always null

                        Comment


                        • #13
                          The same way you tell it to load the xml file which contains your controller (or the component-scan to detect your controller).

                          Comment


                          • #14
                            Im now using the component scan and @autowire to inject this value into my controller and i have the desired behaviour. However my requirement is that i need a new Socket for each request i send. The current implementation works fine for the first request but on subsequent requests it attempts to use the same instance of Socket to send the second request. If there a way to force the socket to get recreated or should i be looking at creating a new instance of the SimpleTcpNetOutboundGateway for each request. Given that performance is an issue what would be the best approach.

                            Comment


                            • #15
                              This should work if the remote end closes the socket after sending the reply. However, it does not work due to a defect in the code.

                              Please create an 'INT' JIRA issue at http://jira.springframework.org.

                              I will try to get a fix into tonight's nightly build.

                              Thanks for finding this issue.

                              Comment

                              Working...
                              X