Announcement Announcement Module
Collapse
No announcement yet.
Inbound channel adapter issue Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    I updated my project with 2.0.4 and tried it. I am still seeing the same behavior:

    Code:
    2011-05-09 19:09:00,053  DEBUG DefaultFtpSessionFactory [task-scheduler-5] (AbstractFtpSessionFactory.java:154) - Connected to server [localhost:21]
    2011-05-09 19:09:00,053  DEBUG DefaultFtpSessionFactory [task-scheduler-3] (AbstractFtpSessionFactory.java:154) - Connected to server [localhost:21]

    Comment


    • #17
      Did you set 'cache-sessions' attribute to 'false'?

      Comment


      • #18
        I don't know what it is worth, but the FTP log shows that two sessions are being established at the same time.


        Code:
        000631)5/9/2011 19:09:00 PM - (not logged in) (127.0.0.1)> USER fidinvest
        (000631)5/9/2011 19:09:00 PM - (not logged in) (127.0.0.1)> 331 Password required for fidinvest
        (000631)5/9/2011 19:09:00 PM - (not logged in) (127.0.0.1)> PASS **********
        (000631)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> 230 Logged on
        (000631)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> TYPE I
        (000631)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> 200 Type set to I
        (000631)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> SYST
        (000631)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> 215 UNIX emulated by FileZilla
        (000631)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> PORT 127,0,0,1,14,242
        (000631)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> 200 Port command successful
        (000631)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> LIST /
        (000631)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> 150 Opening data channel for directory list.
        (000631)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> 226 Transfer OK
        (000631)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> disconnected.
        (000632)5/9/2011 19:09:00 PM - (not logged in) (127.0.0.1)> USER fidinvest
        (000632)5/9/2011 19:09:00 PM - (not logged in) (127.0.0.1)> 331 Password required for fidinvest
        (000632)5/9/2011 19:09:00 PM - (not logged in) (127.0.0.1)> PASS **********
        (000632)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> 230 Logged on
        (000632)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> TYPE I
        (000632)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> 200 Type set to I
        (000632)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> SYST
        (000632)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> 215 UNIX emulated by FileZilla
        (000632)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> PORT 127,0,0,1,14,243
        (000632)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> 200 Port command successful
        (000632)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> LIST /
        (000632)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> 150 Opening data channel for directory list.
        (000632)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> 226 Transfer OK
        (000632)5/9/2011 19:09:00 PM - fidinvest (127.0.0.1)> disconnected.

        Comment


        • #19
          Yes, that is correct. Here is the updated config:

          Code:
          <int-ftp:inbound-channel-adapter id="fromFidFTPReader" local-directory="EncryptedFilesFromFid" 
          			channel="fromFidEncryptedBatchChannel" session-factory="ftpClientFactory" remote-directory="/" 
          			delete-remote-files="true"  filename-pattern="odt*.*" cache-sessions="false">
          		<int:poller cron="0 0/3 6-23 * * SUN-SAT"/>
          	</int-ftp:inbound-channel-adapter>

          Comment


          • #20
            Bobby

            I am still puzzled. I just spent an hour trying to figure this out and although I can explain some of what's going on, but not everything.
            1. The fact that there are two connections to the FTP server is because you have two FTP adapters (inbound and outbound), so even if you are using default CachedConnectionFactory, if cached connection is taken (by one of the adapters), another one will be created. So that's ok.
            2. The fact that you see different task-scheduler threads is also ok, since TaskScheduler creates a default thread pool.

            That leaves only one problem and that is what appears to be a duplicate file processing.
            Code:
            2011-05-09 11:42:00,833  INFO  FtpSession [task-scheduler-4] (FtpSession.java:73) - File have been successfully transfered to: Fid/odt05.txt.pgp
            2011-05-09 11:42:00,927  INFO  FtpSession [task-scheduler-8] (FtpSession.java:73) - File have been successfully transfered to: Fid/odt05.txt.pgp
            I was able to reproduce it (well, sort of) only if i upload file with the same name twice via FTP Outbound Adapter. In fact the log message that you are pointing to:
            Code:
            2011-05-09 11:42:00,833  INFO  FtpSession [task-scheduler-4] (FtpSession.java:73) - File have been successfully transfered to: Fid/odt05.txt.pgp
            . . . comes from the FTP Outbound Adapter (not the inbound) and since your flow begins with Http Inbound Gateway, could you please verify that you are not sending a duplicate file to the FTP Outbound Adapter. You can probably put some log statements in your transformer and see what goes through it and whats been sent to the transformer's output channel. In fact I would suggest to add the wiretap to 'convertChannel' and send it to a logger

            Let me know

            Comment


            • #21
              The middleware product I am working with does two things:
              1. Messages posted via http inbound gateway are accumulated, a batch file created, encrypted and posted to FTP site. This task always posts with the same file name - scf05.txt.pgp.
              2. When an encrypted file (always with the same file name - odt05.txt.pgp) is posted on the FTP site, it is picked up, decrypted, XML messages are generated from that batch file and posted via HTTP outbound adapter.

              I am thinking that the messages you were referring to, in my log - are from synchronizing with the FTP folder after the file is downloaded (the file on the FTP site is being deleted after being downloaded). So, the odt05.txt.pgp is the result of FTP inbound channel adapter as the FTP outbound channel adapter always posts with scf05.txt.pgp file name.

              The issue I opened is limited to inbound FTP processing (FTP outbound processing is not an issue at this point because I removed the file-inbound-channel-adapter to poll a directory and rather passing it to the the input channel of FTP outbound channel adapter. Before that two threads from file inbound channel adapter polled the same file, constructed two messages for the same file). The inbound file - odt05.txt.pgp is typically posted once for every two hours; whenever, it is posted, I see two threads going and picking up the same file and creating two messages and hence, two sets of XML messages are being posted as a result of inbound batch processing.

              Hope I answered your questions.

              Comment


              • #22
                That does explains it a little and is similar to the behavior I've observed yesterday while trying to reproduce.
                So, essentially it seems to be a timing issue (sort of artificial race condition). For example:
                - File has been uploaded via FTP outbound, downloaded via FTP inbound, deleted from remote directory by FTP inbound
                - meanwhile a second file with the same name goes through the same process
                Now it appears as a duplicate processing, but based on your explanation the only thing that is duplicate is the file name since it is always the same, bu the contents are different. Right?

                Comment


                • #23
                  That is correct. The file I post (scf05.txt.pgp) is consumed by a third-party application where as I consume the file (odt05.txt.pgp) posted by them from the same FTP site. For every post of the file (either scf or odt), file name is same but the contents are different.

                  Your statement is true in that for *every* odt file posted, duplicate processing is being performed as a result of FTP inbound channel adapter picking the same file twice.

                  Comment


                  • #24
                    Well, this is what I am trying to clarify.What is your definition of the same file, because it looks like it is always the same file if you judge it based on the name, but the contents of this file are different which makes it a different file. So it is not picked up twice, but rather due to the timing collision of upload/download two different files with the same name are being processed. Right?

                    Comment


                    • #25
                      By saying 'same file' - I mean by name AND content. Two copies are made (one copy by each thread) and messages are constructed. Out of the two threads, one thread gets to delete the file on FTP server.

                      To put it in perspective,
                      Task-scheduler-1 picks up odt05.txt.pgp posted at 1100am, say at 11:03:00:123
                      Task-scheduler-2 picks up the same (by name and content) odt05.txt.pgp posted at 1100 am exactly at 11:03:123
                      Task-scheduler-2 gets to delete the file from FTP server

                      A message is constructed by the copy of the file picked up by Task-scheduler-1 thread
                      Another message is constructed by the copy of the file picked up by Task-scheduler-2 thread

                      End result: the same batch file (by name and content) is processed twice.

                      Comment


                      • #26
                        Bobby

                        What you see is the output from both FTP outbound and inbound adapter
                        Task-scheduler-1 picks up odt05.txt.pgp posted at 1100am, say at 11:03:00:123 - FTP Outbound (thread A)

                        Task-scheduler-2 picks up the same (by name and content) odt05.txt.pgp posted at 1100 am exactly at 11:03:123 - FTP Inbound (thread B)

                        Task-scheduler-2 gets to delete the file from FTP server - FTP Inbound (thread B)

                        I saw exactly the same output yesterday.
                        You can easily verify it by populating remote directory with few files and run only FTP Inbound. You'll see there is no duplicate processing. There is no way in your configuration FTP Inbound could be processing the same file more then once based on how its implemented. File(s) first downloaded into local directory (synchronized). Only than the processing begins.

                        Comment

                        Working...
                        X