Announcement Announcement Module
Collapse
No announcement yet.
SFTP outbound channel adapter - delete source files? Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • SFTP outbound channel adapter - delete source files?

    Is it possible to wire things up in a way that an SFTP outbound channel adapter will delete the source files after uploading them?

    I am aware of the file transformer trick, but since the files I'm uploading are several hundred MB in size, I'd rather not convert them to bytes/string prior to uploading

  • #2
    You can do this
    Code:
    <int:publish-subscribe-channel id="inputChannel"/>
    	
    <int-sftp:outbound-channel-adapter id="sftpOutboundAdapterWithExpression"
    				session-factory="sftpSessionFactory"
    				channel="inputChannel"
    				order="1" . . ./>
    				
    <int:service-activator input-channel="inputChannel" order="2" expression="payload.delete()"/>
    In the above, the input-channel is a synchronous pub-sub with two subscribers.
    However as you can see the order of the Message dispatch to the subscribers is maintained via 'order' attribute of the two subscribers. In other words SFTP adapter will always be the first one to receive the Message and the deleting service-activator will always be the second and *only* after the first one completes successfully otherwise the message won't even reach the service-activator.
    Give it a shot and let me know.
    Although I am thinking we might need to make it simpler, so feel free to raise a feature request and we'll see how it goes

    Comment


    • #3
      Actually i just realized that file.delete() method returns boolean which means service-activator will be looking to where to send a reply message. So here you can do to things. If you want to ignore the boolean value then simply define your output-channel as 'nullChannel'
      Code:
      <int:service-activator input-channel="inputChannel" output-channel="nullChannel" order="2" expression="payload.delete()"/>
      Or if you do care you can send it to a predefined channel where you can check for boolean value and do whatever it is you want with it.

      Comment


      • #4
        Thank you for the quick reply! My sftp XSD (using 2.0.3.RELEASE?) doesn't seem to have an order attribute defined, and only setting order="2" on the service-activator didn't seem to have the desired effect.

        However I could implement the same idea with two channels and a wire-tap like so:

        Code:
            <integration:channel id="outboundFiles">
                <integration:interceptors>
                    <integration:wire-tap channel="outboundFilesToSftp"/>
                </integration:interceptors>
            </integration:channel>
        
            <integration:channel id="outboundFilesToSftp"/>
        
            <sftp:outbound-channel-adapter id="sftpOutboundAdapter"
                                           session-factory="sftpSessionFactory"
                                           channel="outboundFilesToSftp"
                                           charset="UTF-8"
                                           remote-directory="output-dir"/>
        
            <integration:service-activator id="cleanupAfterUpload"
                                           input-channel="outboundFiles"
                                           expression="payload.delete()"
                                           output-channel="nullChannel"/>
        The wire-tap copies the message in preSend, therefore the "tap" channel will receive its copy first - that must be our outbound channel adapter.

        Thanks again!

        Comment


        • #5
          'order' was added to the (s)ftp adapters recently...

          https://jira.springsource.org/browse/INT-1848

          If you can update to 2.0.4.BUILD-SNAPSHOT you should be able to use it.

          Comment


          • #6
            Thanks Gary!

            Yeah, wiretap would work, but we generally don't recommend to use wiretap for things other then logging/auditing. See if you can use snapshot build as Gary suggested. We'll be releasing 2.0.4 in few days anyway.

            Comment


            • #7
              Thanks! I think I can wait until 2.0.4 is out anyway.

              Is there a good way to go and pick up *specific* named files from a remote SFTP / FTP location from a trigger event? One way to do it is to build a service activator that opens a session directly on receipt of a list of files to fetch. It could even be built as an content enricher type filter which receives a list of file names and adds the files to the outgoing message. Is there anything along these lines already in SI?

              Comment


              • #8
                Thanks Oleg. The pub-sub channel with 2 subscribers approach works great! I am using 2.0.5.

                Comment


                • #9
                  I hope my question won't fork the post, but what can i do, when pub-sub channel with 1 or 2 subscribers (depends configuration) both can have delete-source-file se to true.

                  But only one can delete source file, the other will have a FileNotFoundException.

                  Comment


                  • #10
                    You need to show your configuration but one thing that will definitely cause this is if you put a task executor on the pub-sub channel because each leg will process on separate threads and, yes, the file might be deleted before the second one gets it.

                    Turn on DEBUG logging and you will see what's happening.

                    Comment


                    • #11
                      Yes that's exactly what's hapen.

                      But when source is a pub-sub, multiple outbound with delete-source-file set to true should be allowed, no ? (without log error)

                      Comment


                      • #12
                        Not sure what you mean by "source is a pub-sub"; the pub-sub channel is downstream of the adapter. The adapter doesn't know what happens downstream so when the poller thread returns to the adapter, the delete action is taken.

                        If you must go async on your pub-sub channel, you would need to add an aggregator somewhere in your flow so the main thread waits for the result.

                        Alternatively, delete the file yourself when both legs complete.

                        Comment

                        Working...
                        X