Announcement Announcement Module
Collapse
No announcement yet.
DataSource/Database Channel Adapter Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • DataSource/Database Channel Adapter

    Following up on the ideas raised in http://forum.springsource.org/showthread.php?t=64887, I started working on a data source backed channel adapter. So far things are going well and I hope to have my application ported from JMS to Spring Integration in the next day or so.

    The basic approach includes an implementation of an adapter that uses a message DAO to save serialized messages to the DB. A direct channel is used from the message sender to the adapter to ensure the same transaction context. This allows me to prevent message delivery if the business operation fails (desired behavior that required distributed transactions with JMS).

    On the other side, an input channel adapter is used to read the messages from the DB using the same DAO. The poller starts a transaction and the adapter reads the message and immediately deletes it using the DAO. The message is sent on a direct channel to a handler to maintain the transaction context. If the message processing succeeds, the TX is committed and the deletion is applied. If the TX is rolled back, the message is left in the DB for a retry.

    I've also implemented support for adding an error channel header to the message which will deliver failed messages to a handler that will record the failure and delete the message (no retries). Using some Oracle row locking magic I can also support multiple consumers without duplicating messages and in theory (yet untested) I can support parallel deployments of my application with each instance grabbing unique messages as they are available (and at the same time being fault tolerant).

    So far so good. All in all I only wrote 2 classes (the DAO and the adapter). Thanks for a great framework.

    Once I get this done I'll look into getting an article together covering my approach.

    -mike

  • #2
    Sweet music

    If you pull this off (and it sure sounds like you're going to) we will be very interested obviously. The oracle row locking magic worries me a bit, but we might find parts of this that we can find abstractions in the framework for.

    Thanks for pioneering!

    Comment


    • #3
      Did you get any way on this? I would be interested in the results.

      I have started something similar but still a bit different. I'm just figuring out Spring Integration so there's probably lots that could be done better, but have a look at http://github.com/anderssv/si-hibern...er/tree/master .

      My implementation builds on the following concepts:
      - Objects to be put on the queue needs to be persisted with Hibernate
      - Because it is persisted (ie serialized), you only need to store the Id and Class in the queue
      - Because the queue is a table in the database there is no need for JTA etc

      Like I said. This is a early, early implementation. Have a look and let me know if you think it's interesting. And contribute if you can.

      Comment


      • #4
        It's great to see these developments. From my perspective, this fits quite well with what we want to do in Spring Integration 2.0 to implement the EIP "Claim Check" pattern. Essentially, you "check-in" a Message and you get back a "ticket" (the ID). Later you can check it out. As I see it, the two main uses of this are:
        1) persist the Message within the same TX boundary as it was received (e.g. from a JMS queue/topic).
        2) reduce the footprint of the Message being passed around.

        For addressing #2, I think its important to recognize that an ID by itself is not going to be very useful. For example, if routing, etc. is necessary. So, what I would like to do in the claim-check is persist the payload but return a new Message that includes the original Message's headers along with either a "claim ticket" or an "empty" proxy as the payload.

        In this case, the actual persistence mechanism would be determined by the provided implementation of a strategy interface to the ClaimCheck component. Implementation choices would include JDBC, JPA, in-memory Map, or any custom solution.

        Does this seem relevant to anyone other than me?

        Comment


        • #5
          I did get a persistent channel adapter working and wrote an article about it with one of my coworkers. It works really well for our application and we've had it in production for about 4 months now.

          http://www.devx.com/Java/Article/40813

          -mike

          Comment


          • #6
            Mark,

            As far as implementing the "Claim Check" goes, I think it would help with reducing the amount of data moved around but in reality I think most of the messages are simply passed around in in-memory queues so I'm not sure that pulling the data out into a central storage will save much. It also introduces complexity if the 'empty proxy' message moves across some boundary where the data store is no longer accessible.

            I'd just want to make sure that the entire message is recoverable in the event that the receiver fails to process it (within the fetch TX boundary). So you would need a way to find the messages (and not just the data) that have been queued if the claim check ticket is lost. This is normally what a poller is used for so in my mind it makes more sense to have some type of persistent channel and separate the claim check out into a different pattern. I'm not convinced that killing two birds with one stone (persistence and reduced message traffic size) would be a good idea hear because the use cases seem different enough to me.

            -mike

            Comment


            • #7
              Mike,

              Very interesting solution, but why don`t just use JMS chanell adaters and out-of-the-box solution such as ActiveMQ or OpenMQ to achive the same reliable messaging effect?

              Comment


              • #8
                We tested ActiveMQ originally but ran into a bunch of problems/bugs where large blocks of messages would get stuck in the queue or lost entirely. We ended up using nightly builds to get anything working and just lost confidence in the product pretty quickly.

                We did run with OpenMQ for a while but we found that we were spending a lot of time maintaining the deployment, start up, and shutdown scripts and getting them to sync up with our app server (i.e. starting/stopping one and waiting until things were ready before starting/stopping the other). We also had a lot of trouble using distributed transactions (XA) so the JMS messages would be in the same TX as the business logic. Oracle has some limitations on what you can do with the XA driver vs the standard JDBC driver. Once we started to look into multiple parallel deployments (master/slave configuration) it became a pain. It did work, but it was more maintenance than we wanted.

                We really just needed a simple in process, persistent message system. We don't do any distributed messaging (which is where I think JMS can shine). For a single application in a single app server using Spring Integration embedded in our WAR was much simpler. The persistent channel adapter was only about a week of work and it saved us that much time plus trying to configure and maintain a whole JMS solution.

                I'm sure we could have gotten JMS to do what we needed, but I didn't think it was worth it. JMS seems very much like a YAGNI technology in 95% of the cases. A robust ESB gets us everything we need and a much lower config/maintenance overhead. We debated between ESBs but went with SI just because it tied into our existing Spring configuration better.

                Hope that helps,
                -mike

                Comment


                • #9
                  I should have also mentioned that I was comfortable going with SI because we have the option of dropping out our custom persistent channel adapter and plugging in the JMS adapter with little or no code change. So I felt like we got the simple solution with the ability to scale up if needed. However we've been running operationally for a few months now and no one has mentioned going back to JMS

                  -mike

                  Comment


                  • #10
                    Mike,

                    Thanks for the detailed comments. This is a very useful account for us to consider as we move on to 2.0 development in Spring Integration.

                    I do agree with your comment about the Claim Check. Even though we will likely implement that solution in 2.0, we will provide a separate DB adapter. We will also be providing a JMS Channel (e.g. <jms:channel id="c" queue="q"/>) rather than requiring separate inbound/outbound adapters.

                    Regards,
                    Mark

                    Comment


                    • #11
                      Originally posted by mpilone View Post
                      I did get a persistent channel adapter working and wrote an article about it with one of my coworkers. It works really well for our application and we've had it in production for about 4 months now.

                      http://www.devx.com/Java/Article/40813

                      -mike
                      Doh, I should have seen from the username that it was that article I based my implementation on. Thanks for all the details here and in the article.

                      A,

                      Comment


                      • #12
                        Originally posted by Mark Fisher View Post
                        I do agree with your comment about the Claim Check. Even though we will likely implement that solution in 2.0, we will provide a separate DB adapter. We will also be providing a JMS Channel (e.g. <jms:channel id="c" queue="q"/>) rather than requiring separate inbound/outbound adapters.
                        Excellent. If I get to move forward with the Hibernate adapter I probably won't include the Claim Check pattern. For now I need to keep it as simple as possible. Like Mike points out, one of the benefits of going for SI is that you can try an alternative solution, and always know that you will have a fallback to something like JMS.

                        As for others considering not to use JMS; I've had similar experiences like those of Mike. It works, after a lot of hassle. There should be simpler solutions for most cases.


                        Anders,

                        Comment


                        • #13
                          Originally posted by Mark Fisher View Post
                          Mike,

                          Thanks for the detailed comments. This is a very useful account for us to consider as we move on to 2.0 development in Spring Integration.

                          I do agree with your comment about the Claim Check. Even though we will likely implement that solution in 2.0, we will provide a separate DB adapter. We will also be providing a JMS Channel (e.g. <jms:channel id="c" queue="q"/>) rather than requiring separate inbound/outbound adapters.

                          Regards,
                          Mark
                          With regards to the DB adapter I too have implemented a solution based on Mike's article. A key consideration I found was around having multiple consumers of the database "queue" and parallel deployments, we havnt all got Oracle row locking magic :-) I ended up with the inbound adapter having to pessimistcaly lock the message it was about to work on. Maybe there's a beter approach for this??

                          Also, one of the constraints I had to work with was that rather than serialiaze the message payload to the database I had to unmarshall the xml and then persist. So some support around this would be handy for my situation.

                          Comment

                          Working...
                          X