Announcement Announcement Module
Collapse
No announcement yet.
FTPSource questions Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • FTPSource questions

    Hi --

    Not sure if this is the best place to post these questions so I'll do it here for now. Push me to the JIRA if necessary...

    FTPSource looks like it continuously (i.e. very rapidly) polls an FTP site. Is there any way we can (a) set a poll interval and (b) set a maximum number of worker threads that poll the source?

    Also - once I'm done processing a from the FTP site, am I expected to manually remove it or do you foresee the ability to specify a "delete/move to another place on the source when completion" type facility which would stop me from having to do this? I'm thinking of maybe implementing a postReceive interceptor on my sink/last channel in the workflow. Does this sound reasonable? I understand there are transactional semantics to consider here so I'm going to have a better think about it!

    Thanks much!
    Joel

  • #2
    Apologies, my lack of understanding of the framework/architecture is obvious.

    For anyone else who might be interested, set the schedule period in your channel-adapter something similar to (period is in ms):

    Code:
    	<channel-adapter channel="ftpChannel" source="ftpSource">
    		<schedule period="10000" />
    	</channel-adapter>

    Comment


    • #3
      The forum is an excellent place to ask questions, so there is no reason to apologise.

      By the way in the current head the FtpSource has been refactored to allow more configuration. Now you take batches of changed files from the server in one poll, the batch size is configurable. Combined with a collection splitter this should give you the same source, but silence the network considerably.

      Code:
      	<bean id="ftpSource"
      		class="org.springframework.integration.adapter.ftp.FtpSource"
      		p:host="localhost" p:password="kaas" p:username="ftp-user"
      		p:remoteWorkingDirectory="ftp-test">
      		<constructor-arg>
      			<bean
      				class="org.springframework.integration.message.DefaultMessageCreator" />
      		</constructor-arg>
      	</bean>
      
      	<si:channel-adapter id="input" source="ftpSource" />
      
      	<si:splitter output-channel="output" input-channel="input" ref="defaultSplitter" method="split"/>
      
      	<bean id="defaultSplitter"
      		class="org.springframework.integration.util.CollectionSplitter" />
      
      	<si:channel id="output"/>

      Comment


      • #4
        The point you raise about deleting messages is particularly tricky from where I'm standing. The File and Ftp source have no way of being sure that a file isn't being written anymore, so during each poll they create messages for files that have been changed and not processed yet.

        If you would delete files at some point you could potentially delete a file that is still being written to. Especially with ftp the file is on another server and there is no such thing as transaction guarantees. The best way to deal with this in my opinion is to make one node responsible for write operations on the file (specifically including the delete). So FtpSource is just viewing the changing set of files and creating messages based on that, after processing the owner of the file is notified and potentially deletes the file.

        It is not the frameworks business to delete the file, I think. We could provide a Deleting*Target maybe...

        Comment


        • #5
          Hi Iwein,

          How has your example above changed for M6, I can't find a CollectionSplitter anymore?

          Comment


          • #6
            In M6, it is not necessary to provide ref/method with the <splitter/> element. When they are not provided, the default behavior should be what you are looking for: if the request Message contains a Collection or Array as payload, it will be split into individual Messages for each object in the Collection or Array.

            Hope this helps.
            -Mark

            Comment


            • #7
              Thank you for your help Mark,

              The ref attribute of the splitter element seems to be required in the schema though, so going without it doesn't seem to be an option, or am I missing something? The FtpSource example in the reference manual should probably also be updated. Thanks!

              Comment


              • #8
                Thanks for pointing this out. The "ref" attribute is required by the schema (which I am updating at this moment), but it is also required by the parser. Could you please open a JIRA issue for this?

                Thanks again,
                Mark

                Comment


                • #9
                  Sure. One other thing: By looking at the code for QueuedFTPClientPool I would suspect that if the poll period for the channel-adapter is more than the timeout configured on the ftp server, the pool will eventually contain connections that have been closed by the server, and these will be served from the pool with no checks. Now I haven't been able to test this (because of the problem above), but I would suspect it will happen. It would seem prudent to call sendNoOp() on the FTPClient instance and catch the potential exception before passing it forward.

                  http://commons.apache.org/net/apidoc...tml#sendNoOp()

                  Comment


                  • #10
                    Originally posted by fdurden View Post
                    It would seem prudent to call sendNoOp() on the FTPClient instance and catch the potential exception before passing it forward.

                    http://commons.apache.org/net/apidoc...tml#sendNoOp()
                    Thx for pointing that out. This one is also worth a JIRA issue.

                    If you want to be able to test before the schema updates it is almost trivial to write your own CollectionSplitter like this
                    Code:
                    public class CollectionSplitter {
                    
                    	@SuppressWarnings("unchecked")
                    	@Splitter public List split(Message<? extends Collection> message) {
                    		return new ArrayList(message.getPayload());
                    	}
                    }
                    (I actually copied the source from the testpackage where I have used it in my @Ignored integration tests)

                    If you create JIRA issues for the problems you find I will get to them next weekend. I really appreciate the effort you put into trying this out.

                    Comment


                    • #11
                      The timeline has changed somewhat on this obviously. Ftp related classes are going to be pushed out of the release, but the code is not going to be removed completely as there is obviously demand for it. We're discussing a separate project for extensions to the Spring Integration project at the moment.

                      Comment


                      • #12
                        Which library is FTPSource in?

                        iwein,

                        Where can I download the library which has this FTPSource class in it? I'm looking for a Spring API which connects to an FTP Server and uploads a file. Is that possible with Spring?

                        Thanks,

                        IMichel

                        Originally posted by iwein View Post
                        The forum is an excellent place to ask questions, so there is no reason to apologise.

                        By the way in the current head the FtpSource has been refactored to allow more configuration. Now you take batches of changed files from the server in one poll, the batch size is configurable. Combined with a collection splitter this should give you the same source, but silence the network considerably.

                        Code:
                        	<bean id="ftpSource"
                        		class="org.springframework.integration.adapter.ftp.FtpSource"
                        		p:host="localhost" p:password="kaas" p:username="ftp-user"
                        		p:remoteWorkingDirectory="ftp-test">
                        		<constructor-arg>
                        			<bean
                        				class="org.springframework.integration.message.DefaultMessageCreator" />
                        		</constructor-arg>
                        	</bean>
                        
                        	<si:channel-adapter id="input" source="ftpSource" />
                        
                        	<si:splitter output-channel="output" input-channel="input" ref="defaultSplitter" method="split"/>
                        
                        	<bean id="defaultSplitter"
                        		class="org.springframework.integration.util.CollectionSplitter" />
                        
                        	<si:channel id="output"/>

                        Comment


                        • #13
                          https://build.springframework.org/br...OT-21/artifact contains the artifacts you are looking for. If you want you can also check out the code and build it yourself (or encorporate the classes in your own build system). There is no official release of the Ftp adapters, but they are quite stable.

                          Comment


                          • #14
                            FTP Client Example

                            Iwien,

                            Do you have sample bean configuration you can send me? i.e: <bean> properties that are expected by the API, for example.

                            If not that is fine, but it would make it easier for me when I am trying to set it up.

                            Thanks!

                            IMichel

                            Comment


                            • #15
                              You can check out the config for the testcases:
                              https://fisheye.springsource.org/bro...ts-context.xml

                              If you install the SpringSource Tool Suite, you will be able to use code completion for all the properties, so you shouldn't have any trouble wiring things up.

                              Comment

                              Working...
                              X