Announcement Announcement Module
Collapse
No announcement yet.
multiple http-outbound gateway to download files Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • multiple http-outbound gateway to download files

    hi

    i have a question about multiple http outbound connection.
    in my use case i need to download files via http, i'm using this:
    PHP Code:
    <int:channel id="fileList"></int:channel>
    <
    int-http:outbound-gateway id="httpOutboundGateway"         
            
    url="{outboundGatewayUrl}" request-channel="fileList"
            
    reply-channel="channel13" http-method="GET"
            
    expected-response-type="byte[]" charset="UTF-8">
            <
    int-http:uri-variable name="outboundGatewayUrl" expression="payload" />
    </
    int-http:outbound-gateway>
    <
    int:channel id="channel13"></int:channel>
    <
    int-file:outbound-channel-adapter id="dwonloadChannel" channel="channel13"
            
    directory-expression="'${fileToProcess.dir}'" 
            
    filename-generator-expression="'file-'.concat(headers['jms_timestamp']).concat('.ready')"
            
    auto-create-directory="true" /> 
    this works fine (but if you have better option, i'd be happy to know )
    Note-the fileList is populated with msg' by a splitter, which may produce 10-60 links at once.
    My question is about, making this a multi-threaded process. meaning that i want to download ~5 files simultaneously.
    will it be enough to change the channel Q
    PHP Code:
    <int:channel id="fileList"> <int:queue capacity="5"/></int:channel
    and this?
    PHP Code:
    <int:channel id="channel13"> <int:queue capacity="5"/></int:channel
    or do i need to use custom task-execution (an example would be very helpful)?

    thanks!

  • #2
    It's probably better to use an ExecutorChannel to control the concurrency.

    http://static.springsource.org/sprin...ecutor-channel

    http://static.springsource.org/sprin...xecutorchannel

    You also might want to consider using (s)ftp instead of http, to avoid having to store the downloaded file in memory.

    If you have to use HTTP, you might want to open an improvement JIRA issue to allow the content to be written directly to disk...

    https://jira.springsource.org/browse/INT

    ... or, even better - implement the feature and submit a pull request...

    https://github.com/SpringSource/spri...ONTRIBUTING.md

    Comment

    Working...
    X