Announcement Announcement Module
No announcement yet.
Extreme High Throughput/ Low Latency Aggregation Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Extreme High Throughput/ Low Latency Aggregation

    Hi Team,

    We have a fire hydrant of streaming quotes at 100k/sec with 17k symbols that need real-time conversion to 1 second quote bars. Oleg demonstrated an excellent usage to build quote bars using an aggregator and reaper in The throughput on Oleg's example is about 10k-20k for 100 symbols on my laptop, which is good but not enough. The latency inherent in the aggregator/ reaper is probably due to having to pass over once again all of the messages that compose the aggregation to create the quote bar instead of building it as you go. If we avoid passing over all the messages a second time we would get a boost. But let me ask an open question.

    My question is: how would one scale given this use case with Spring Integration? What have you found with putting a router in front of the aggregators and then allowing different horizontal nodes handle the aggregation? And then if a channel fails?

    Thank you