Announcement Announcement Module
No announcement yet.
How do we size our thread pool? Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • How do we size our thread pool?


    Silly question perhaps, but I'm really looking for tips from the experts.

    We have a few million incoming files to process over the next few months using a very basic Spring Integration setup (see recent posts). There's a single directory poller, dispatching to a pool of service activator worker threads. Processing a single file is potentially long running (say a minute or more for example), and the process involves several database queries, inserts, and some downstream communication of some sort (currently undefined, could be a web services call).

    We expect to have brand new dedicated hardware (not a virtual machine) with something like 6 or 12 cores (depending on what they finally buy) and 24 GB of RAM, running linux.

    Obviously I must test, test, test, but are there any rules of thumb? I know there's the one-thread-per-core rule, but it strikes me that that's perhaps weighted toward GUI apps and that may not apply to bulk processing.

    And HOW should I test? Do I just crank up the size of the pool until something falls over? Or is there a more methodical approach?

    If something falls over, how will it fall over? Obviously we can monitor memory usage and thread counts, but how do we know when and how it will fail?

    Much obliged,