Announcement Announcement Module
Collapse
No announcement yet.
File Inbound gradual slowdown Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • File Inbound gradual slowdown

    Hi,
    For my application I have a need to process large chunk of files from input folder and pass it to downstream for other processing like db write, archiving the files etc. To start with I have 250k files in the input folder. I use Spring File Integration for upstream processing, some correlation tools to match/pair the input files and then insert rows to db for a successful match. Finally the input files are archived. I are seeing that the application slows down drastically as it is polling more files:

    File Start File End Elapsed Time(secs) Processing Rate
    1 2000 24.491 40.83
    2001 4000 26.536 37.68
    10001 12000 35.423 28.23
    20001 22000 46.345 21.58
    30001 32000 56.967 17.55
    42001 42000 67.245 14.87
    50001 52000 77.295 12.94
    60001 62000 107.502 9.30
    70001 72000 226.209 4.42
    80001 82000 334.054 2.99
    The slowdown is gradual and consistent. Initially I thought it could be because of the correlation tool being used or may be beacause of the db write operation. So I trimmed down the application and in the new basic version, I just have file inbound adapter pass the files to a transformer which just logs them to a log file. Even in this basic version, I am seeing same slowdown.
    Here is the code configuation used:
    Code:
    <channel id="filesIn"/>
    <channel id="mailMsgTransformer"/>
    <beans:bean id="customTrigger" class="org.springframework.scheduling.support.PeriodicTrigger">
            <beans:constructor-arg value="10"/>
            <beansroperty name="fixedRate" value="false"></beansroperty>
            <beansroperty name="initialDelay" value="0" />
    </beans:bean>
    
    <file:inbound-channel-adapter channel="filesIn"
                                    directory="#[fileserver.input.path]" scanner="directoryScanner" filename-pattern="*.xml">
                                    <poller trigger="customTrigger"/>
    <transformer ref="mailTransformerBean" input-channel="filesIn"
                                    method="transform" output-channel="nullChannel"/>
    <beans:bean id="mailTransformerBean" class="...Transformer"></beans:bean>
    <beans:bean id="directoryScanner"       class="...CustomDirectoryScanner">
                    <beans:property name="numberOfThreads" value="4" />
                    <beans:property name="queueSize" value="10" />
    </beans:bean>
    The files come in request/response pair so I have written a CustomDirectoryScanner which extends DefaultDirectoryScanner. This scanner has custom threads which pairs the files and puts them to filesIn channel.
    The Transformer class is just for printing the files as and when they arrive:
    Code:
    public Message<?> transform(Message<?> message) {
           Object payload = message.getPayload();
           log.info("Sending file {} to output channel.", ++i);
           return message;
    }
    Now as you can see I have very little custom code except for the DirectoryScanner where I do some business logic.

    On analyzing the heap memory usage, cpu utilization I didnt see anything abnormal and the appliation was well withing the limits. So I doubt if it has to do with memory leak. Thread contention could be another issue but if it were because of thread contention, the slowdown will be much random than this. Currently I am seeing the same pattern of slowdown at the similar rate for multiple runs.

    I am not sure where am I going wrong here. Any inputs would be greatly appreciated.
Working...
X