Announcement Announcement Module
Collapse
No announcement yet.
OutOfMemory at FileTransfer Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • OutOfMemory at FileTransfer

    Hi folks,

    just experiencing a "interesting" scenario.

    We are transfering files via Spring Integration int-sftp:outbound-channel-adapter to a remote directory. So far so good. On a long term test scenario the file system of the remote server ran out of free space.
    The spring application was still able to create zero-byte-size files with the standard .writing file extension on the remote system, BUT in the java heap we can see a rapid increase of org.springframework.integration.file.remote.handle r.FileTransferringMessageHandler in size.

    The following table is a quick overview what we see when we analyze a heap dump taken just before the application runs into the OOM error.

    Class Name Shallow Heap Retained Heap
    java.util.HashMap$Entry[8192] @ 0xdc6c95f8 32.784 570.039.728
    table java.util.HashMap @ 0xc4ddddf0 48 570.039.776
    map java.util.HashSet @ 0xc4dddde0 16 570.039.792
    c java.util.Collections$SynchronizedSet @ 0xc4ddddc8 24 570.039.816
    allocated org.springframework.integration.util.SimplePool @ 0xc4ccae80 48 570.140.360
    pool org.springframework.integration.file.remote.sessio n.CachingSessionFactory @ 0xc4c87830 24 570.140.384
    sessionFactory org.springframework.integration.file.remote.handle r.FileTransferringMessageHandler @ 0xc4c877a0 80 570.145.344
    value java.util.concurrent.ConcurrentHashMap$HashEntry @ 0xc4e00a98 32 32
    Is this a bug, that SI, or the underlying sftp library can't realize that it is not able to put the file completely successfull into the remote directory or is it a problem that we have to address on the remote system that we just do not run out of free disk space?

    Any help or comments are greatly appreciated!

    Thx in advance
    Cheers
    Peter
    Last edited by peter0601; Aug 20th, 2013, 03:40 AM. Reason: typo

  • #2
    Can you provide your configuration (at least of the sftp adapter(s)) and a log snippet?

    Comment


    • #3
      allocated org.springframework.integration.util.SimplePool @ 0xc4ccae80
      On the face of it, it looks like the session cache is growing without limit; which is the default behavior when cache-sessions="true" (default).

      This is probably a bug in SI, but I'll take a look.

      You can limit the cache size (http://static.springsource.org/sprin...ession-caching) which will avoid the OOM, but if the sessions are not being returned, your app won't work without a restart when the server condition is resolved.

      Comment


      • #4
        Hi Gary,

        thanks a lot for your reply. We are giving the Cached Session Factory a try and restart our test szenario again to see if we still can re-produce the behaviour.

        About the config. We are pretty much using the configuration of the dynamic ftp example.
        But for the sake of completeness, here we go :-)

        Code:
        <bean id="sftpClientFactory" class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
                <property name="host" value="${host}"/>
                <property name="user" value="${username.sftp}"/>
                <property name="password" value="${password}"/>
                <property name="port" value="${port}"/>
            </bean>
        
        <int-sftp:outbound-channel-adapter id="ftpOutbound"
                                               channel="toFtpChannel"
                                               remote-directory="${remote.directory}"
                                               session-factory="sftpClientFactory"
                                               remote-filename-generator-expression="payload.getName()+'.txt'"/>
        Last edited by peter0601; Aug 20th, 2013, 07:57 AM.

        Comment


        • #5
          Thanks; it's a bug in the cached session management (when cached sessions are closed, while they are removed from the pool for the purposes of re-use, a reference is retained). Changing the session cache size, unfortunately, won't be a work-around.

          I have opened a JIRA issue https://jira.springsource.org/browse/INT-3112

          Comment

          Working...
          X