Announcement Announcement Module
Collapse
No announcement yet.
ftp inbound channel adapter sending files to channel, that have already been download Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • ftp inbound channel adapter sending files to channel, that have already been download

    Currently I have an FTP Inbound channel adapter configured with "channel=nullChannel", downloading files from the "remote-directory", and storing them in the "local-directory".

    I have another server that reads "local-directory", and uses the files.

    I want to change this however, and along with downloading to "local-directory", I also want to put these files into another folder, as they are downloaded.

    I thought simple, create a new channel "newFileChannel", add a file outbound channel adapter, and change the channel on ftp inbound channel adapter from "nullChannel" to "newFileChannel". however, it seems that this does not actually take the files as they are downloaded from FTP, but rather from the existing files in "local-directory". this is not the usecase that I want, the reason basically being that "local-directory" is never emptied (reasons irrelevant I think).

    suggestions on how I can achieve my UC? that is, each time a file is downloaded, and saved to "local-directory", add this to a channel.

    Thanks

    Code:
    <int:channel id="newFileChannel" />
    	
    	<int-file:outbound-channel-adapter directory="#{integrationProperties['secondary.staging.folder']}" channel="newFileChannel" filename-generator="customFileNameGenerator" />
    	
    	<bean id="customFileNameGenerator" class="CustomFileNameGenerator" >
    	</bean>
    	
    	<int-ftp:inbound-channel-adapter id="myFtpChannelAdapter"
     				channel="newFileChannel" 
    				session-factory="ftpSessionFactory"
    				charset="UTF-8"
    				auto-create-local-directory="true"
    				delete-remote-files="#{integrationProperties['ftp.delete.remote.files']}"
    				remote-directory="#{integrationProperties['ftp.folder.base'] + newspaperFolders['foo']}"
    				remote-file-separator="/"
    				local-directory="#{integrationProperties['ftp.download.folder.base'] + newspaperFolders['foo']}" 
    				filter="pnFilter">
    			<int:poller fixed-delay="100000" max-messages-per-poll="-1" ></int:poller>
    <!-- 			<int:poller cron="0 50 2 * * MON-FRI" max-messages-per-poll="-1" /> -->
    	</int-ftp:inbound-channel-adapter>

  • #2
    You need to write a custom FileListFilter that only accepts new files and ignores all files that are there during startup. However, this cannot be configured using the namespace until the 3.0 release. It was added and is available in the 3.0.0.M2 milestone.

    https://jira.springsource.org/browse/INT-2892

    For 2.2.x you would have to wire up the adapter, poller etc using <bean/> declarations.

    Comment


    • #3
      Thank you very much Gary.

      As my current implementation/application is only using "int-ftp:inbound-channel-adapter" I think I can safely use 3.0.0.M2 instead of 2.1.0.RELEASE.

      could you clarifiy?

      will "local-filter" "execute" after ftp download?
      the reason I ask this is because the poller on my "int-ftp:inbound-channel-adapter" is a once a day cron.

      I dont want to get into the situation where I have to wait 24 hours after the file has been downloaded for it to be picked up and put onto int-ftp:inbound-channel-adapter's "channel"


      or do you think it might just be better for me to have an "int-file:inbound-channel-adapter" with this custom filter, polling every couple of seconds?

      Comment


      • #4
        thinking about this more, and it seems kind of obvious and probably simpler to just use file inbound channel adapter and a customer filter, unless I am completely missing something

        I dont need to upgrade my integration then, and dont need to define my ftp adapter as a bean instead of using the int-ftp namespace

        Comment


        • #5
          Yes, it's used after the download; the filter is used to filter the remote files to be downloaded then, after the download, the local-filter is applied when determining which messages to send.

          Previously, the local-filter was always an AcceptOnceFileListFilter. This caused problems for other users, who wanted to be able to process the same filename each day. Hence we added the local-filter attribute.

          Comment


          • #6
            Yes, that would work too - have the ftp adapter send to nullChannel and use a file adapter - thats really exactly what's going on under the skins.

            That's a great work-around for others too. Thanks!

            Just be sure to filter out any files appended with ".writing" because that is appended (by default) when files are in the process of being downloaded. Here's the code that sets up the filter in the 'FileReadingMessgeSource' that's embedded in the ftp adapter...

            Code:
            Pattern completePattern = Pattern.compile("^.*(?<!" + this.synchronizer.getTemporaryFileSuffix() + ")$");
            return new CompositeFileListFilter<File>(Arrays.asList(
            		this.localFileListFilter,
            		new RegexPatternFileListFilter(completePattern)));
            Depending on how long the files generally take to be pulled down, you'd have to set the second poller up appropriately.

            Comment


            • #7
              thanks a stack for your help!

              Spring Integration really is an awesome project. I use it extensively, and situations that seem complex in the beginning always turn out to have a simple implementation. I guess that is the Spring way

              Comment

              Working...
              X