Announcement Announcement Module
Collapse
No announcement yet.
int-file:inbound-channel-adapter - file contents Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • int-file:inbound-channel-adapter - file contents

    Hallo,

    is possible to get a file readed every time the contents are changing. Once i process a file with a name, it will never be readed again (until i restart the process). There is a way to change this behavior ?

    tnx

    ticino

  • #2
    I guess you could implement your own filter (implementing the interface "FileListFilter") and check the last modified time to see if it has changed: the filter could keep a Map of every file it receives with its last modification date for instance.

    Then, inject this filter into your file inbound-channel-adapter.

    Does that sound good to you?

    Pierre

    Comment


    • #3
      yes, it worked, i implemented the following filter :

      Code:
      package com.mycompany.test.commons.si;
      
      import java.io.File;
      import java.util.HashMap;
      import java.util.Map;
      
      import org.springframework.integration.file.filters.AbstractFileListFilter;
      
      /**
       * @author ticino
       * @param <F>
       */
      public class ModifiedFileListFilter<F> extends AbstractFileListFilter<F> {
      
          private final Map<F, Long> fileModTime = new HashMap<F, Long>();
      
          private final Object monitor = new Object();
      
          /* (non-Javadoc)
           * @see org.springframework.integration.file.filters.AbstractFileListFilter#accept(java.lang.Object)
           */
          @Override
          protected boolean accept(F file) {
              synchronized (this.monitor) {
                  if (this.fileModTime.get(file) == null) {
                      // new file, never seen
                      fileModTime.put(file, ((File) file).lastModified());
                      return true;
                  }
                  if (this.fileModTime.get(file).compareTo(((File) file).lastModified()) == 0) {
                      return false;
                  }
                  if (this.fileModTime.get(file).compareTo(((File) file).lastModified()) < 0) {
                      // file was modified, update timestamp
                      fileModTime.put(file, ((File) file).lastModified());
                      return true;
                  }
                  // modified in past
                  return false;
              }
          }
      }
      and configured :

      Code:
      	<bean id="modifiedFileListFilter" class="com.mycompany.test.commons.si.ModifiedFileListFilter" />
      
      
      <int-file:inbound-channel-adapter
      		directory="indir" auto-create-directory="true" channel="mychannel"
      		filter="modifiedFileListFilter">
      	</int-file:inbound-channel-adapter>
      every file that i modify into indir directory is then processed .

      Thanks for the hint, work great.

      ticino
      Last edited by ticino; Feb 21st, 2011, 05:16 PM.

      Comment


      • #4
        Pierre's recommendation is right on. Basically, when you add your own filter, the default AcceptOnceFileListFilter will not be included. If you need more than one, you can use our CompositeFileListFilter.

        HTH,
        Mark

        Comment


        • #5
          I done it by extending AbstractFileListFilter, seem to work ... something wrong with it ?

          Comment


          • #6
            ftp

            how could I achieve this with an ftp adapter ?

            Comment


            • #7
              You should be able to do it the same exact way - via 'filter' attribute. In fact the same filter implementation would work for File/Ftp/Sftp":
              Code:
              <bean id="modifiedFileListFilter" class="com.mycompany.test.commons.si.ModifiedFileListFilter" />
              
              <int-ftp:inbound-channel-adapter
              		directory="indir" channel="mychannel"
              		filter="modifiedFileListFilter">
              </int-ftp:inbound-channel-adapter>

              Comment


              • #8
                Originally posted by ticino View Post
                I done it by extending AbstractFileListFilter, seem to work ... something wrong with it ?
                Nope, it looks even better

                Glad it worked !

                Comment


                • #9
                  Hello.
                  Guys, I think we should not save files in memory: it increases the useage of memory with each file.
                  I think it is enough to save only file name and lastModified proprties.

                  Hope I'm right

                  Artem

                  Comment


                  • #10
                    Dear Cleric,

                    The content of the files is not saved in memory, only the File objects are. File objects contain only the path in memory (and an integer -- prefixLength) : this shouldn't cost more than the path itself (as a String) then.

                    Pierre

                    Comment


                    • #11
                      ftp

                      filter:
                      Code:
                      public class FtpModifiedFileListFilter extends AbstractFileListFilter<FTPFile>{
                      	private final Map<String, Calendar> fileModTime = new HashMap<String, Calendar>();
                      	private final Object monitor = new Object();
                      	
                      	Log logger = LogFactory.getLog(getClass());
                      	
                      	protected boolean accept(FTPFile file) {
                      		synchronized (this.monitor) {
                      			if(this.fileModTime.containsKey(file.getName())){
                      				// got it
                      			}else{
                      				fileModTime.put(file.getName(), file.getTimestamp());
                                      logger.info("new file:"+file.getName());
                                      return true;
                      			}
                      			if(this.fileModTime.get(file.getName()).before(file.getTimestamp())){
                      				logger.info("modified:"+file.getName());
                      				this.fileModTime.put(file.getName(), file.getTimestamp());
                      				return true;
                      			}else{
                      				logger.info("not modified:"+file.getName());
                      			}
                      			return false;
                      		}
                      	}
                      }
                      config:
                      Code:
                      <int:channel id="ftp-copied-files"/>
                      	
                      	<int:logging-channel-adapter channel="ftp-copied-files"/>
                      
                          <int-ftp:inbound-channel-adapter channel="ftp-copied-files"
                           	local-directory="file://#{systemProperties['user.home']}/received_ftp_files"
                           	session-factory="ftp-session-factory" remote-directory="stock" delete-remote-files="false"
                           	filter="file-filter">
                            	<int:poller task-executor="ftp-executor" fixed-delay="25000"/>
                          </int-ftp:inbound-channel-adapter>
                          
                          <bean id="file-filter" class="org.springframework.integration.file.filters.CompositeFileListFilter">
                          	<constructor-arg ref="file-list-filters"/>
                          </bean>
                          
                          <util:list id="file-list-filters">
                          	<bean class="org.wims.ftp.FtpModifiedFileListFilter"/>
                          	<bean class="org.springframework.integration.ftp.filters.FtpSimplePatternFileListFilter">
                          		<constructor-arg value="test-ext.csv"/>
                          	</bean>
                          </util:list>
                      result:
                      Code:
                      INFO  FtpModifiedFileListFilter - new file:banden.txt
                      INFO  FtpModifiedFileListFilter - new file:data-input.txt
                      INFO  FtpModifiedFileListFilter - new file:test-ext.csv
                      INFO  FtpSession - File have been successfully transfered to: stock/test-ext.csv
                      DEBUG FileReadingMessageSource - Added to queue: [C:\Users\wim\received_ftp_files\test-e
                      INFO  FileReadingMessageSource - Created message: [[Payload=C:\Users\wim\received_ftp_fi
                      DEBUG DirectChannel - preSend on channel 'ftp-copied-files', message: [Payload=C:\Users\
                      DEBUG LoggingHandler - org.springframework.integration.handler.LoggingHandler#0 received
                      INFO  LoggingHandler - C:\Users\wim\received_ftp_files\test-ext.csv
                      DEBUG DirectChannel - postSend (sent=true) on channel 'ftp-copied-files', message: [Payl
                      INFO  FtpModifiedFileListFilter - not modified:banden.txt
                      INFO  FtpModifiedFileListFilter - not modified:data-input.txt
                      INFO  FtpModifiedFileListFilter - not modified:test-ext.csv
                      INFO  FtpModifiedFileListFilter - not modified:banden.txt
                      INFO  FtpModifiedFileListFilter - not modified:data-input.txt
                      INFO  FtpModifiedFileListFilter - modified:test-ext.csv
                      INFO  FtpSession - File have been successfully transfered to: stock/test-ext.csv
                      INFO  FtpModifiedFileListFilter - not modified:banden.txt
                      INFO  FtpModifiedFileListFilter - not modified:data-input.txt
                      INFO  FtpModifiedFileListFilter - not modified:test-ext.csv
                      the filter detects the modified file, but never gets to the channel ?

                      Comment


                      • #12
                        ftp

                        okay , AcceptOnceFileListFilter is still added in AbstractInboundFileSynchronizingMessageSource extended by FtpInboundFileSynchronizingMessageSource

                        ftp:inbound-channel-adapter has no option prevent-duplicates="false" as in file:inbound-channel-adapter
                        Last edited by wims.tijd; Feb 24th, 2011, 11:39 AM.

                        Comment

                        Working...
                        X