Announcement Announcement Module
Collapse
No announcement yet.
TCP Endpoint - binary incoming transfer - high cpu load Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • TCP Endpoint - binary incoming transfer - high cpu load

    Hello,

    I've configured TCP endpoint for incoming binary transfers (large eg. 1-5MB binary files) as follow:

    Code:
    <bean id="binarySerializer"
          class="org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer">
              <property name="maxMessageSize" value="10000000" />
        </bean>
        <bean id="binaryDeserializer"
          class="org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer">
              <property name="maxMessageSize" value="10000000" />
        </bean>
    
        <!-- connection factory -->
        <int-ip:tcp-connection-factory id="binary_cf" type="server" port="5001"
                                   so-tcp-no-delay="false" 
                                   lookup-host="false" so-receive-buffer-size="100000"
                                   using-nio="true" deserializer="binaryDeserializer" 
                                   serializer="binarySerializer" />   
    
        <!-- gateways -->
        <int-ip:tcp-inbound-gateway id="gateway_binary"
                                request-channel="binary_handler" reply-channel="binary_reply"
                                connection-factory="binary_cf" reply-timeout="10000" />
    
    <int:service-activator input-channel="binary_handler" output-channel="binary_reply"
                       ref="binaryHandlerBean" method="handleBinary"/>

    and binary activator is as follow:

    Code:
    public class BinaryActivator {
    
        private Logger log = Logger.getLogger(BinaryActivator.class);
    
        @Autowired
        @Qualifier("generic")
        private BinaryHandler binaryHandler;
    
        public byte[] handleBinary(byte[] msg) {
            log.debug("Binary Handler: Handle binary object. Payload size: " + msg.length);
            return binaryHandler.handleBinaryObject(msg);
        }
    }
    With the above configuration when sending files from external endpoint (non Spring based - standard TCP C# implementation with ByteArrayLengthHeaderSerializer payload preparation) the CPU utilization is very high. The load is gone just after reaching handleBinary function in the activator. I was trying nio and non-nio implementation. I think something is wrong with my implementation, but I would gladly ask someone with more experience ! Please help

    Slawek

  • #2
    I just ran a test and saw no issues, even with debug logging turned on a megabyte was transferred in about 100ms.

    How is the client sending data? In very small chunks?

    Run with debug logging (and NIO), and look for messages like this...

    Code:
    2012-06-21 18:22:28,103 [pool-2-thread-3] DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 4 bytes, buffer is now at 4 of 4
    2012-06-21 18:22:28,103 [pool-2-thread-3] DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Message length is 1000000
    2012-06-21 18:22:28,103 [pool-2-thread-3] DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1020 bytes, buffer is now at 1020 of 1000000
    ...
    
    2012-06-21 18:22:28,211 [pool-2-thread-3] DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 580 bytes, buffer is now at 1000000 of 1000000

    Comment


    • #3
      The test. 10 files each 1MB of data. Sending through one open tcp connection in a sequence. Sending from C# application (linux, mono. Between each files different communication on the other tcp channel (command data) - e.g. registering file transfer.
      Transfer of each file last circa 1-1.5 sec. Tomcat load in this time is ca 100% cpu load and the web application is a bit unresponsive.
      The test was runned on the linux and on win xp test machine.

      I've got another tcp endpoint opened for the command channel (protobuf communication) with much smaller packet sizes and even with a cumulative transfer c.a. 1 MB the load is very small.

      Log:
      Code:
      DEBUG: org.springframework.integration.ip.tcp.connection.TcpNioConnection - Message sent [Payload=[B@13ffa58][Headers={timestamp=1340351992338, id=737eb0e7-f242-49cc-8253-372d76176687, ip_address=127.0.0.1, ip_connection_seq=8, ip_hostname=127.0.0.1, ip_tcp_remote_port=39576, ip_connection_id=127.0.0.1:39576:89658411-647d-484c-99d8-888f017f77d9}]
      DEBUG: org.springframework.integration.ip.tcp.connection.TcpNioConnection - 127.0.0.1:36640:d9ed2b9a-826c-4883-b223-98d69c170615 Reading...
      DEBUG: org.springframework.integration.ip.tcp.connection.TcpNioConnection - Read 61440 into raw buffer
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 4 bytes, buffer is now at 4 of 4
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Message length is 1024005
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1020 bytes, buffer is now at 1020 of 1024005
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1024 bytes, buffer is now at 2044 of 1024005
      [...]
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1024 bytes, buffer is now at 56316 of 1024005
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1024 bytes, buffer is now at 57340 of 1024005
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1024 bytes, buffer is now at 58364 of 1024005
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1024 bytes, buffer is now at 59388 of 1024005
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1024 bytes, buffer is now at 60412 of 1024005
      DEBUG: org.springframework.integration.ip.tcp.connection.TcpNioConnection - 127.0.0.1:36640:d9ed2b9a-826c-4883-b223-98d69c170615 Reading...
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1024 bytes, buffer is now at 61436 of 1024005
      DEBUG: org.springframework.integration.ip.tcp.connection.TcpNioConnection - Read 61440 into raw buffer
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1024 bytes, buffer is now at 62460 of 1024005
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1024 bytes, buffer is now at 63484 of 1024005
      [...]
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1024 bytes, buffer is now at 1023996 of 1024005
      DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 9 bytes, buffer is now at 1024005 of 1024005
      DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'binary_handler', message: [Payload=[B@b529c8][Headers={timestamp=1340351992515, id=40b70fc3-1cf5-4107-a31d-b61d15f49fd7, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_hostname=127.0.0.1, ip_connection_seq=1, ip_tcp_remote_port=36640, ip_connection_id=127.0.0.1:36640:d9ed2b9a-826c-4883-b223-98d69c170615}]
      DEBUG: org.springframework.integration.handler.ServiceActivatingHandler - ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor@15385c6] received message: [Payload=[B@b529c8][Headers={timestamp=1340351992515, id=40b70fc3-1cf5-4107-a31d-b61d15f49fd7, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_hostname=127.0.0.1, ip_connection_seq=1, ip_tcp_remote_port=36640, ip_connection_id=127.0.0.1:36640:d9ed2b9a-826c-4883-b223-98d69c170615}]
      DEBUG: [...cut package name...].integration.BinaryActivator - Binary Handler: Handle binary object. Payload size: 1024005
      DEBUG: [...cut package name...].services.LogServiceImpl - LogEntry src: BinaryServiceImpl identify: handle_binary summary: Obsługa transferu binarnego
      DEBUG: [...cut package name...].services.BinaryServiceImpl - Handle binary PUT for msg length: 1024005
      DEBUG: [...cut package name...].services.BinaryServiceImpl - Handle binary PUT hashId: 5363bcc8
      DEBUG: [...cut package name...].services.LogServiceImpl - LogEntry src: BinaryServiceImpl identify: handle_binary_put summary: Obsługa zapisu danych (PUT)
      DEBUG: org.springframework.integration.handler.ServiceActivatingHandler - handler 'ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor@15385c6]' sending reply Message: [Payload=[B@473dc][Headers={timestamp=1340351993879, id=57447007-1fe7-4f7d-917d-0b41202d7da8, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_connection_seq=1, ip_hostname=127.0.0.1, ip_tcp_remote_port=36640, ip_connection_id=127.0.0.1:36640:d9ed2b9a-826c-4883-b223-98d69c170615}]
      DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'binary_reply', message: [Payload=[B@473dc][Headers={timestamp=1340351993879, id=57447007-1fe7-4f7d-917d-0b41202d7da8, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_connection_seq=1, ip_hostname=127.0.0.1, ip_tcp_remote_port=36640, ip_connection_id=127.0.0.1:36640:d9ed2b9a-826c-4883-b223-98d69c170615}]
      DEBUG: org.springframework.integration.handler.BridgeHandler - org.springframework.integration.handler.BridgeHandler@6b8de2 received message: [Payload=[B@473dc][Headers={timestamp=1340351993879, id=57447007-1fe7-4f7d-917d-0b41202d7da8, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_connection_seq=1, ip_hostname=127.0.0.1, ip_tcp_remote_port=36640, ip_connection_id=127.0.0.1:36640:d9ed2b9a-826c-4883-b223-98d69c170615}]
      DEBUG: org.springframework.integration.handler.BridgeHandler - handler 'org.springframework.integration.handler.BridgeHandler@6b8de2' sending reply Message: [Payload=[B@473dc][Headers={timestamp=1340351993879, id=57447007-1fe7-4f7d-917d-0b41202d7da8, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_connection_seq=1, ip_hostname=127.0.0.1, ip_tcp_remote_port=36640, ip_connection_id=127.0.0.1:36640:d9ed2b9a-826c-4883-b223-98d69c170615}]
      DEBUG: org.springframework.integration.channel.DirectChannel - postSend (sent=true) on channel 'binary_reply', message: [Payload=[B@473dc][Headers={timestamp=1340351993879, id=57447007-1fe7-4f7d-917d-0b41202d7da8, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_connection_seq=1, ip_hostname=127.0.0.1, ip_tcp_remote_port=36640, ip_connection_id=127.0.0.1:36640:d9ed2b9a-826c-4883-b223-98d69c170615}]
      DEBUG: org.springframework.integration.channel.DirectChannel - postSend (sent=true) on channel 'binary_handler', message: [Payload=[B@b529c8][Headers={timestamp=1340351992515, id=40b70fc3-1cf5-4107-a31d-b61d15f49fd7, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1ca4240, ip_hostname=127.0.0.1, ip_connection_seq=1, ip_tcp_remote_port=36640, ip_connection_id=127.0.0.1:36640:d9ed2b9a-826c-4883-b223-98d69c170615}]
      Sample C# code for sending
      Code:
      private TcpClient m_clientBinary;
      private IPEndPoint m_endpointBinary;
      
      m_clientBinary = new TcpClient();
      try {
        m_endpointBinary = new IPEndPoint(IPAddress.Parse(m_host), m_portBinary);
        m_clientBinary.Connect(m_endpointBinary);				
      } catch (System.Net.Sockets.SocketException ex) {
        throw new CommException(ex.Message);
      }
      
      
      public void SendToBinary(byte[] buffer) 
      {
      int bufLen = buffer.Length;
      NetworkStream stream = m_clientBinary.GetStream();
      stream.Write(Utils.ReverseBytes(BitConverter.GetBytes(bufLen)), 0, 4);
      stream.Write(buffer, 0 , buffer.Length);
      stream.Flush();			
      }

      Comment


      • #4
        After replying yesterday, I realized (with NIO), we are not using the receiveBufferSize every where we should; however, I would not expect this to affect the case when you are not using NIO; you said you still saw problems then.

        I have a fix on this branch https://github.com/garyrussell/sprin...mmits/INT-2635 that should help (with NIO).

        Any chance you can try it?

        Comment


        • #5
          ok, i'll try it in the next week.

          I'll check again also without NIO. I was checking this configuration without NIO some time ago, and have some problems with that (dunno know what that was because it was on the beginning of the project and maybe something else was a problem) and since then I was not using that configuration.

          Thanks for a support !

          Comment


          • #6
            I've checked again with detailed logging of debug message time.

            Without NIO:

            Code:
            22 cze 2012 20:33:08,204 DEBUG: org.springframework.integration.channel.DirectChannel - postSend (sent=true) on channel 'protobuf_input', message: [Payload=[B@166ebeb][Headers={timestamp=1340389988192, id=4d050500-ffdc-4f80-b887-c9bf349e1ac2, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@ad2a5, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@ad2a5, ip_hostname=127.0.0.1, ip_connection_seq=10, ip_tcp_remote_port=43848, ip_connection_id=127.0.0.1:43848:0f92a258-034b-4789-9707-692884951d8b}]
            22 cze 2012 20:33:08,204 DEBUG: org.springframework.integration.ip.tcp.connection.TcpNioConnection - Message sent [Payload=[B@8b6e75][Headers={timestamp=1340389988204, id=aa265c8b-7d38-44ad-bee9-22ed48a00bcb, ip_address=127.0.0.1, ip_connection_seq=10, ip_hostname=127.0.0.1, ip_tcp_remote_port=43848, ip_connection_id=127.0.0.1:43848:0f92a258-034b-4789-9707-692884951d8b}]
            22 cze 2012 20:33:08,208 DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 4 bytes, buffer is now at 4 of 4
            22 cze 2012 20:33:08,209 DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Message length is 1024005
            22 cze 2012 20:33:08,209 DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 65536 bytes, buffer is now at 65536 of 1024005
            [...]
            22 cze 2012 20:33:08,211 DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 40965 bytes, buffer is now at 1024005 of 1024005
            22 cze 2012 20:33:08,211 DEBUG: org.springframework.integration.ip.tcp.connection.TcpNetConnection - Message received [Payload=[B@e053][Headers={timestamp=1340389988211, id=129850e2-94f2-4d40-85f4-8e92b51acb58, ip_address=127.0.0.1, ip_connection_seq=3, ip_hostname=127.0.0.1, ip_tcp_remote_port=37986, ip_connection_id=127.0.0.1:37986:9e8752b4-d1b7-485f-8584-c832f548ba6f}]
            22 cze 2012 20:33:08,211 DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'binary_handler', message: [Payload=[B@e053][Headers={timestamp=1340389988211, id=f863d3cf-37a5-4f48-88eb-8b2600bf1c9d, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1878f19, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1878f19, ip_hostname=127.0.0.1, ip_connection_seq=3, ip_tcp_remote_port=37986, ip_connection_id=127.0.0.1:37986:9e8752b4-d1b7-485f-8584-c832f548ba6f}]
            22 cze 2012 20:33:08,212 DEBUG: org.springframework.integration.handler.ServiceActivatingHandler - ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor@a422a1] received message: [Payload=[B@e053][Headers={timestamp=1340389988211, id=f863d3cf-37a5-4f48-88eb-8b2600bf1c9d, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1878f19, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@1878f19, ip_hostname=127.0.0.1, ip_connection_seq=3, ip_tcp_remote_port=37986, ip_connection_id=127.0.0.1:37986:9e8752b4-d1b7-485f-8584-c832f548ba6f}]
            22 cze 2012 20:33:09,551 DEBUG: [[ cut package name ]].integration.BinaryActivator - Binary Handler: Handle binary object. Payload size: 1024005
            22 cze 2012 20:33:09,551 DEBUG: [[ cut package name ]].services.LogServiceImpl - LogEntry src: BinaryServiceImpl identify: handle_binary summary: Obsługa transferu binarnego
            22 cze 2012 20:33:09,551 DEBUG: [[ cut package name ]].services.BinaryServiceImpl - Handle binary PUT for msg length: 1024005
            22 cze 2012 20:33:09,551 DEBUG: [[ cut package name ]].services.BinaryServiceImpl - Handle binary PUT hashId: fbdfcc05
            With NIO

            Code:
            22 cze 2012 20:42:57,147 DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1024 bytes, buffer is now at 1024000 of 1024005
            22 cze 2012 20:42:57,147 DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 5 bytes, buffer is now at 1024005 of 1024005
            22 cze 2012 20:42:57,147 DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'binary_handler', message: [Payload=[B@5dd779][Headers={timestamp=1340390577147, id=6a2365a3-c989-42cb-85d7-0be3e2831a37, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@ed5ea9, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@ed5ea9, ip_hostname=127.0.0.1, ip_connection_seq=6, ip_tcp_remote_port=33149, ip_connection_id=127.0.0.1:33149:850a416b-edfe-4760-b6ca-b9572bf84de0}]
            22 cze 2012 20:42:57,147 DEBUG: org.springframework.integration.handler.ServiceActivatingHandler - ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor@62af74] received message: [Payload=[B@5dd779][Headers={timestamp=1340390577147, id=6a2365a3-c989-42cb-85d7-0be3e2831a37, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@ed5ea9, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@ed5ea9, ip_hostname=127.0.0.1, ip_connection_seq=6, ip_tcp_remote_port=33149, ip_connection_id=127.0.0.1:33149:850a416b-edfe-4760-b6ca-b9572bf84de0}]
            22 cze 2012 20:42:58,574 DEBUG: [[ cut package name ]].integration.BinaryActivator - Binary Handler: Handle binary object. Payload size: 1024005
            22 cze 2012 20:42:58,574 DEBUG: [[ cut package name ]].services.LogServiceImpl - LogEntry src: BinaryServiceImpl identify: handle_binary summary: Obsługa transferu binarnego
            22 cze 2012 20:42:58,574 DEBUG: [[ cut package name ]].services.BinaryServiceImpl - Handle binary PUT for msg length: 1024005
            22 cze 2012 20:42:58,574 DEBUG: [[ cut package name ]].services.BinaryServiceImpl - Handle binary PUT hashId: 13df5aa3
            It seems that something (??) is blocking (copying ??) data between service activating (org.springframework.integration.handler.ServiceAc tivatingHandler) and real activator function (my code in initial post). Any ideas. Maybe I'm doing something wrong - please bear in mind i'm no expert in this area

            Comment


            • #7
              Originally posted by Gary Russell View Post
              After replying yesterday, I realized (with NIO), we are not using the receiveBufferSize every where we should; however, I would not expect this to affect the case when you are not using NIO; you said you still saw problems then.

              I have a fix on this branch https://github.com/garyrussell/sprin...mmits/INT-2635 that should help (with NIO).

              Any chance you can try it?
              Hi Gary, I've tested your branch of spring-integration. No difference from the previous (stable) version. Nonetheless as i have written in next posts the long timeout is between these lines. Any ideas ?

              Code:
              23:54:31,788 DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 1024 bytes, buffer is now at 1023996 of 1024005
              23:54:31,788 DEBUG: org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer - Read 9 bytes, buffer is now at 1024005 of 1024005
              23:54:31,789 DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'binary_handler', message: [Payload=[B@8ee06][Headers={timestamp=1341179671789, id=e994c61f-9694-4ea7-87c3-fca11ff334b3, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@42df6c, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@42df6c, ip_hostname=127.0.0.1, ip_connection_seq=9, ip_tcp_remote_port=41188, ip_connection_id=127.0.0.1:41188:6eae2c4a-8eb9-4c55-9a0b-1ab1fb779d6d}]
              23:54:31,789 DEBUG: org.springframework.integration.handler.ServiceActivatingHandler - ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor@4f3e0c] received message: [Payload=[B@8ee06][Headers={timestamp=1341179671789, id=e994c61f-9694-4ea7-87c3-fca11ff334b3, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@42df6c, ip_address=127.0.0.1, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@42df6c, ip_hostname=127.0.0.1, ip_connection_seq=9, ip_tcp_remote_port=41188, ip_connection_id=127.0.0.1:41188:6eae2c4a-8eb9-4c55-9a0b-1ab1fb779d6d}]
              23:54:32,454 DEBUG: [..package..].integration.BinaryActivator - Binary Handler: Handle binary object. Payload size: 1024005

              Comment


              • #8
                Interesting; that seems to eliminate the TCP adapters as the cause of your high cpu issue.

                Given your configuration, the framework does very little between preSend() on the channel and actually invoking your method. I don't see that a large payload should make any different there.

                Do you have any channel interceptors configured?

                Can you try to get a thread dump (e.g. using jstack) while in this state?

                Comment


                • #9
                  Sorry for a delay. Please find in the attachment the stack dump of the running process. Some information:
                  - tomcat 7
                  - spring with scheduler threads
                  - vaadin as a GUI layer with pooler thread
                  - spring integration for tcp/ip message handling

                  I think the problem is in the "pool-85-thread-2" thread.

                  The spring-integration for the binary part is exact as in the first post. There is the same configuration for protobuf port but I think it does not interfere(?). The routing for the byte[] message is from tcp-inbound-gateway (using tcp-connection-factory) directly to the my binary activator which takes byte[] as a function parameter and return also with a byte[] response. Nothing more.
                  I see from the dump that there is some kind of conversion of integration Message object (?) to byte[] array and thus copying large arrays. But this is only my guessing. Maybe I should take another activator function signature in order to avoid useless data duplication ?
                  Last edited by slawek.mikula; Jul 4th, 2012, 03:38 AM.

                  Comment


                  • #10
                    Thanks; I think this is due to a change in SpEL in Spring 3.1 that causes conversion on all arguments, even if they don't need it; Spring Integration uses SpEL under the covers to invoke the service method.

                    Is there any chance you could try with Spring 3.0.7 to confirm my suspicion?

                    I just had to make a change to avoid another issue because of this problem.

                    Looks like we either need to handle all such cases in S.I., or escalate it to the Core Spring team.

                    Comment


                    • #11
                      This is not the final fix, but it should work around it for you...

                      Code:
                      git clone https://github.com/garyrussell/spring-integration.git
                      git checkout cpu
                      ./gradlew install
                      Then update your pom to use 2.1.4.BUILD-SNAPSHOT

                      Please open a JIRA issue here... https://jira.springsource.org/browse/INT

                      Comment


                      • #12
                        Yup, you got it ! Thanks!

                        After downgrading spring to 3.0.7 i've got 10MB file in 300ms and everything is working right now very well. !

                        Comment


                        • #13
                          Originally posted by Gary Russell View Post
                          This is not the final fix, but it should work around it for you...

                          Code:
                          git clone https://github.com/garyrussell/spring-integration.git
                          git checkout cpu
                          ./gradlew install
                          Then update your pom to use 2.1.4.BUILD-SNAPSHOT

                          Please open a JIRA issue here... https://jira.springsource.org/browse/INT
                          Thanks, i'll try it. Tomorrow i'll open this regression issue ticket.

                          Comment


                          • #14
                            Created: https://jira.springsource.org/browse/INT-2650

                            But i assume as a assignee You already know it

                            Comment


                            • #15
                              Thanks; the fix is in master now; we will probably cut a 2.1.4 at the end of the month; will that work for you? Or, do you need it sooner?

                              Comment

                              Working...
                              X