Announcement Announcement Module
Collapse
No announcement yet.
2.0.4.RELEASE - make sure your SFTP directories have a trailing slash! Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • 2.0.4.RELEASE - make sure your SFTP directories have a trailing slash!

    It seems like something has changed between 2.0.3 and 2.0.4 where if you previously had an SFTP inbound gateway with a remote directory configured, and it did not have a trailing slash, you might start seeing the following exception after upgrading:

    Code:
    org.springframework.core.NestedIOException: failed to read file; nested exception is 2: No such file
    	at org.springframework.integration.sftp.session.SftpSession.read(SftpSession.java:100)
    	at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.read(CachingSessionFactory.java:137)
    	at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.copyFileToLocalDirectory(AbstractInboundFileSynchronizer.java:176)
    	at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:138)
    	at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.receive(AbstractInboundFileSynchronizingMessageSource.java:144)
    	at org.springframework.integration.endpoint.SourcePollingChannelAdapter.doPoll(SourcePollingChannelAdapter.java:89)
    	at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:146)
    	at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:144)
    	at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:207)
    	at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
    	at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
    	at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
    	at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:202)
    	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:51)
    	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
    	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    	at java.lang.Thread.run(Thread.java:680)
    Caused by: 2: No such file
    	at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
    	at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:1741)
    	at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:1011)
    	at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:986)
    	at org.springframework.integration.sftp.session.SftpSession.read(SftpSession.java:96)
    	... 22 more
    I haven't tracked it down to any specific code change in 2.0.4.RELEASE, but my project definitely works in 2.0.3 and generates the above error with 2.0.4. Adding a trailing slash to the remote directory resolves the issue.

  • #2
    This is actually strange
    We did do some work here for 2.0.4 but it mainly involved exposing remoteFileSeparator for configuration since we got some issues related to fileSeparator character specific to the platform. However, the default value remains the same '/' and it was never appended at the end;
    For example
    Code:
    2.0.3 AbstractInboundFileSynchronizer
    private String remoteFileSeparator = "/";
    . . .
    private void copyFileToLocalDirectory(String remoteDirectoryPath, F remoteFile, File localDirectory, Session session) throws IOException {
    		String remoteFileName = this.getFilename(remoteFile);
    		String remoteFilePath = remoteDirectoryPath + remoteFileSeparator + remoteFileName;
    ...
    }
    
    2.0.4 AbstractInboundFileSynchronizer
    private String remoteFileSeparator = "/";
    . . .
    private void copyFileToLocalDirectory(String remoteDirectoryPath, F remoteFile, File localDirectory, Session session) throws IOException {
    		String remoteFileName = this.getFilename(remoteFile);
    		String remoteFilePath = remoteDirectoryPath + remoteFileSeparator + remoteFileName;
    ...
    }
    and i can't see any changes to the read method of SftpSession between 2.0.3 and 2.0.4

    Come to think of it, what version of JSCH is on your classpath?
    Also, could you provide more info about your client/server environment so I can try to reproduce it here.

    Comment


    • #3
      I thought of JSCH too, but it hasn't changed between the releases. I'm just pulling it in as a transitive dependency:

      Code:
      [INFO] +- org.springframework.integration:spring-integration-sftp:jar:2.0.4.RELEASE:compile
      [INFO] |  +- com.jcraft:jsch:jar:0.1.42:compile
      [INFO] |  +- org.springframework.integration:spring-integration-stream:jar:2.0.4.RELEASE:compile
      [INFO] |  \- org.springframework:spring-context-support:jar:3.0.5.RELEASE:compile
      
      [INFO] +- org.springframework.integration:spring-integration-sftp:jar:2.0.3.RELEASE:compile
      [INFO] |  +- com.jcraft:jsch:jar:0.1.42:compile
      [INFO] |  +- org.springframework.integration:spring-integration-stream:jar:2.0.3.RELEASE:compile
      [INFO] |  \- org.springframework:spring-context-support:jar:3.0.5.RELEASE:compile
      So I stepped through org.springframework.integration.file.remote.synchr onizer.AbstractInboundFileSynchronizer#copyFileToL ocalDirectory and remoteFilePath definitely did not contain a slash in the line:

      Code:
      				session.read(remoteFilePath, fileOutputStream);
      I'll give you some more details in a bit. Here's what my SFTP config looks like:

      Code:
          <sftp:inbound-channel-adapter
                  id="sftpInboundAdapter"
                  session-factory="sftpSessionFactory"
                  cache-sessions="true"
                  remote-directory="${sftp.dir}"
                  filename-regex="prefix.*\.txt"
                  delete-remote-files="true"
                  local-directory="#{ sftpDirectory.absolutePath }"
                  auto-create-local-directory="true"
                  channel="convertIncomingFiles">
              <integration:poller fixed-delay="5000" error-channel="errorChannel"/>
          </sftp:inbound-channel-adapter>
      The sftpDirectory is a java.util.File instance created in the Spring beans context, and sftp.dir is resolved by a PropertyPlaceholderConfigurer.

      Comment


      • #4
        It seems to be related to "INT-1885 added support for empty remote-file-separator to the FTP Inbound Channel Adapter" - this fragment in org.springframework.integration.file.config.Abstra ctRemoteFileInboundChannelAdapterParser#parseSourc e:

        Code:
        		String remoteFileSeparator = element.getAttribute("remote-file-separator");
        		if (remoteFileSeparator != null){
        			synchronizerBuilder.addPropertyValue("remoteFileSeparator", remoteFileSeparator);
        		}
        Initializes the separator to empty unless you explicitly provide a value in the XML config. So the below works fine in 2.0.3, but not in 2.0.4:

        Code:
        <?xml version="1.0" encoding="UTF-8"?>
        <beans xmlns="http://www.springframework.org/schema/beans"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xmlns:sftp="http://www.springframework.org/schema/integration/sftp"
               xmlns:integration="http://www.springframework.org/schema/integration"
               xmlns:stream="http://www.springframework.org/schema/integration/stream"
               xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
                   http://www.springframework.org/schema/integration/sftp http://www.springframework.org/schema/integration/sftp/spring-integration-sftp-2.0.xsd
                   http://www.springframework.org/schema/integration/stream http://www.springframework.org/schema/integration/stream/spring-integration-stream-2.0.xsd">
        
            <sftp:inbound-channel-adapter
                    id="sftpInboundAdapter"
                    session-factory="sftpSessionFactory"
                    cache-sessions="true"
                    remote-directory="input"
                    filename-regex="test.*\.txt"
                    delete-remote-files="true"
                    local-directory="/tmp/files"
                    auto-create-local-directory="true"
                    channel="files">
                <integration:poller fixed-delay="5000" error-channel="errorChannel"/>
            </sftp:inbound-channel-adapter>
        
            <bean id="sftpSessionFactory" class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
                <property name="host" value="vm"/>
                <property name="port" value="22"/>
                <property name="user" value=".."/>
                <property name="password" value="..."/>
            </bean>
        
            <integration:channel id="files"/>
        
            <integration:transformer
                    input-channel="files" output-channel="fileNames"
                    expression="'File downloaded: ' + payload.name + '.'"/>
        
            <integration:channel id="fileNames"/>
        
            <stream:stderr-channel-adapter append-newline="true" channel="fileNames"/>
        
        </beans>
        Changing the SFTP inbound adapter by adding an explicit remote-file-separator setting like so fixes it again:

        Code:
            <sftp:inbound-channel-adapter
                    id="sftpInboundAdapter"
                    session-factory="sftpSessionFactory"
                    cache-sessions="true"
                    remote-directory="input"
                    remote-file-separator="/"
                    filename-regex="test.*\.txt"
                    delete-remote-files="true"
                    local-directory="/tmp/files"
                    auto-create-local-directory="true"
                    channel="files">
                <integration:poller fixed-delay="5000" error-channel="errorChannel"/>
            </sftp:inbound-channel-adapter>
        The cause for all this is that element.getAttribute("remote-file-separator") returns an empty string even if the XML element has no such attribute, so changing it like so will fix the default behavior back to what it was.

        The solution is to change org.springframework.integration.file.config.Abstra ctRemoteFileInboundChannelAdapterParser#parseSourc e like so:

        Code:
        		if (element.hasAttribute("remote-file-separator")){
        			synchronizerBuilder.addPropertyValue("remoteFileSeparator", element.getAttribute("remote-file-separator"));
        		}
        I've opened an issue and attached a patch with unit tests in JIRA - INT-1921.

        Comment

        Working...
        X