Announcement Announcement Module
Collapse
No announcement yet.
Possible to slow a batch for web request tasks? Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Possible to slow a batch for web request tasks?

    I need to import a lot of data from a remote service that only lets me connect once per second. If I connect too often, they take away access. I'm looking for a way to throttle my single-threaded batch down to 1 request per second. I suppose I could add a sleep step as the last one, but would hope there's a more elegant solution.

    Thanks!

  • #2
    You could do that if you use spring integration too in your batch processing. You could use a pollable channel that polls every second in the item reader. Please check out the spring integration reference documentaion.

    http://static.springsource.org/sprin...lling-consumer

    You could do the following:
    1. Introduce an inbound gateway based on the ItemReader,
    2. Connect item reader to a pollable channel with the inbound gateway. Configure the poller to the desired throttling frequency so that a call to the itemreader is delayed before calling the next step.
    3. connect the poller channel to a web service outbound gateway which would be called only based on the polling frequency.

    Regards,
    Anoop
    Last edited by anoop2811; Mar 3rd, 2013, 01:47 PM. Reason: Mistook use case for item writing

    Comment


    • #3
      Thanks!

      I'm gonna think more about your suggestions. I ended up writing a throttle queue

      I'm going to share an instance of this between my batch jobs and my servets. It's a throttled gate that only allows an opening every N milliseconds. Requests that come in quicker than that are put in a queue that's sorted by the request priority (passed in) and the date requested. This way, if a batch is running with priority 1, a user's web request with priority 2 can interrupt it without even temporarily increasing the request rate. Two batches running at the same time with the same priority will take turns, based on request date.

      I still haven't fully tested it with a real batch, but think it should work just fine.

      Thanks again
      - Blake
      Last edited by blake; Mar 3rd, 2013, 10:01 PM.

      Comment

      Working...
      X