Announcement Announcement Module
No announcement yet.
Design for Extension and Dynamic Routing in SI Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Design for Extension and Dynamic Routing in SI

    Apologies if this is rambles on a bit - it's a relatively complex issue and I'm wondering if anybody has come across it before.

    We've been thinking a lot about how to deal with extension via configuration in our current Spring Integration project and we were wondering if anybody had any thoughts on the best approach or suggestions.

    For background we are using SI as the core of a high-volume transaction processing platform - there is a fairly well defined existing transactional integration pipeline (i.e. : accept message, perform basic validation, get context, check config, process message, store results, construct response , reply (with the usual exceptional cases etc)) , conceptually to us there is a "well trodden path of integration" which we'd like to preserve as best as possible. What we are keen to understand is to what extent we could "extend" the functionality of this at load time.

    In the past We've used SI as a platform/framework for arbitrarily extending basic service functionality (e.g. new API calls) by sitting new features on filtering channels (e.g. adding a "Foo" call on a service API message channel) which has worked very well. But now we're considering if or how we could do the same for AOP-style cross-cutting of an existing (but well defined) pipeline.

    We have a number of features which we know we could integrate by extending the current pipeline in various places - a review of these features has led us to the conclusion that almost all of them can be implemented using one or more of the following patterns inserted at one or more places in the existing pipeline:

    * Header enrichment (e.g. early on in the pipeline validating building and storing the context of one feature)
    * Later content enrichment from those headers (preserving the existing payload types e.g. consolidating early results into a response)
    * Re-routing /"bailing out" out of the existing "well trodden" pipeline to an egress point (there is no need to "re-join" the pipeline) (e.g. something like a pluggable validator which diverted a message into a validation errors channel and on to response in the simplest case)

    We've identified a set of extension points in the existing pipeline that everything we're currently planning would fit into, all of the features are essentially isolated (barring one or two clear dependencies (e.g. A, A +B but not B) ) ordering of individual extension is also not likely to be a big issue (simple integer ordering would suffice where needed) .

    If possible we'd like to be able to add features by say just including the relevant feature's spring config files in the app context load (which is what we've done before and works pretty well, we think) .

    Each feature can currently be expressed as its own spring context (sub to the application context) + set of SI chains "extensions", each of which currently integrates with a single "extension point" on the main pipeline (so one feature might introduce several cooperating extensions) , each extension either passes the message through (possibly with transformation and elaboration) to the next handler , or diverts it out to it's own channel, aborting the main message flow.

    The case for us being able to do this is compelling as it looks like it offers the flexibility we think we need, while preserving some or all of the rigidity and understanding of the core pipeline as well as allowing each feature to be tested end-to-end in context without the others. It's conceptually close to AOP in the normal java world but with a limited set of cut points (we are all-too-well aware the pitfalls and risks of trying to use AOP to deliver core functionality) .

    The real question is - has anybody tried to do this sort of AOP-like extension to a pipeline before, and what worked?

    What I think would work is a sort of "Pluggable" SI primitive which sat at each of these extension points and did the following:
    * Was configured for the extension chains/channels relevant to that point
    * Could dispatch the message down each of those channels such that:
    ** If that chain did not "divert" the message the outcome of the chain message was dispatched down the next chain until all chains were visited
    ** If it did divert the message then processing would stop from the point of view of the remaining chains (and ongoing pipeline)

    We've tried a few off-the-shelf compositions of pub/sub and existing routing primitives but nothing has spring out as the right solution.

  • #2
    There are a couple of things I can point you to...

    1. There is a 'dynamic-ftp' sample in the advanced samples here

    This shows how you can create a dynamic channel resolving router. In that particular case, it resolves to a channel in a newly-created (or cached) application context that is separate from the context in which the resolver itself resides. A new context is created for each destination FTP server. But, there's really no limit to what you can do. For example, route to the child context under some condition, or to the "normal' channel under other conditions (where you might simply inject the "normal" channel into the resolver).

    <bean id="channelResolver" class="org.springframework.integration.samples.ftp.DynamicFtpChannelResolver" />
    <int:channel id="toDynRouter" />
    <int:router input-channel="toDynRouter"
    2. David Turanski has been working on a <flow/> namespace; this allows you to define encapsulated subflows, which are themselves contained in separate application contexts, and can be configured using placeholders, with the Spring 3.1 environment support. This allows multiple instances of the same subflow to be included, with different sets of properties. More information on this can be found here These flows may return messages to the main flow, or they can be a one-way street and consume the message.

    It is still a work-in-process, but I am aware of a number of implementations that are nearing production (in system test).

    Now, in terms of being able to add extension points, one technique would be to define <bridge/> elements at strategic points...

    <bridge id="extensionPoint.1" input-channel="previous" output-channel="next" />
    You can then do conditional imports of elements that would override these bridges. You can use Spring profiles to override these beans, or do conditional <import/>s that might contain elements to override, such as

    <router id="extensionPoint.1" input-channel="previous" output-channel="next" ... />

    <service-activator id="extensionPoint.1" input-channel="previous" output-channel="next" ... />
    etc, etc.

    Hope that helps.