Announcement Announcement Module
Collapse
No announcement yet.
Contribute some new adapter code to Spring integration. Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Contribute some new adapter code to Spring integration.

    Hello,
    I have got some Amazon AWS spring integration adapters developed for
    Amazon S3 (inbound and outbound), Amazon SES (outbound) and Amazon SQS (inbound and outbound). These adapters are developed with respective namespace support and successfully run for some basic test cases i have developed.

    I have already signed this contract and have sent a scanned signed copy to the mentioned email id (I am yet to send the hard copy of the signed agreement though).

    Can anybody tell me whom should I get in touch with or anything i am expected to do?

  • #2
    Can any of the moderators please throw some light on this?

    Comment


    • #3
      We are working on setting up an extensions repository, and I will keep you posted on that.

      In the meantime, is your code in github, and if so, would you mind sharing the URL?

      Comment


      • #4
        Thanks Mark
        I have it in my svn repository currently, i'll put it up on github and post the link here in few hours.

        Comment


        • #5
          Hi Mark,
          I have added the code to the github and it can be found here

          There is a small getting started doc i have added to the docs folder (also attaching the .doc format )

          Comment


          • #6
            Hi Mark,
            Have you got a chance to have look at this one?
            Also tentatively when would the extension repository be setup?

            Comment


            • #7
              Amol,

              Thanks for the reminder. I've been a bit overwhelmed getting 2.1 ready recently, but I will finally take a look (and will forward it to some other interested parties).

              Keep an eye out for the extensions updates...

              Thanks,
              Mark

              Comment


              • #8
                Thanks Mark , let me know your views/suggestions for it.

                Comment


                • #9
                  Hello Mark,
                  Apologies for digging out the old thread again. but did you get a chance to have a look, would like your feedback and suggestions to make some more changes to improve its usability.
                  Also, any news about the extension adapters repository?

                  Comment


                  • #10
                    Hi Amol

                    Just wanted to leave some feedback on your AWS library. It's very handy and easy to add into our configuration, but ultimately we were unable to use it for a number of reasons.
                    • For various reasons, we cannot upgrade to Spring 3.1, but for some reason, your SQS adapter uses a ConversionService implementation only available in 3.1. I don't know why your SQS adapter needs to handle any conversion at all. Isn't that the job of Message Transformers?
                    • While the XML configuration offered by your XSD makes dropping in the functionality very simple, it does not allow for overriding various components. It would be great if I could just point it to a reference of my sqsClient, which might be a slightly different implementation rather than the default (add encryption, etc.). Also, you use a default Json message transformer, but don't have the configuration in place to use a different message transformer, or even customize the configuration of the Json transformer. We have some Jackson modules that are necessary to serialize/deserialize our data, but we can't add that in with the XSD configuration.
                    • I'm not sure why you need to use your own interface to describe the AwsCredentials. AwsCredentials is already an interface for which there are multiple implementations. We have one that retrieves our credentials from a secure store, but can't use it with your library because it doesn't implement your interface.

                    Ultimately, I think your components need to focus on doing fewer things. Ideally, the SQS adapters should just focus on dealing with the SQS data and delegating to any configurable transformers to manage serializing/deserializing the data. Some configuration like the following would be a great improvement:

                    Code:
                        <!-- only one of credentialsRef or sqsClient is necessary -->
                        <aws-sqs:inbound-channel-adapter
                            credentialsRef="myCredentialsImpl" 
                            sqsClient="myClientImpl"
                            channel="inboundChannel"
                            sqs-queue="https://queue.amazonaws.com/myworkqueue"
                            transactional="false"
                            max-redelivery-attempts="3">
                                <int:poller fixed-rate="500" receive-timeout="3000" />
                                <!-- optional transformer or transformation could be handled in the delegate channel chain -->
                                <int:json-to-object-transformer type="com.foo.MyDataClass" object-mapper="objectMapper"/>
                        </aws-sqs:inbound-channel-adapter>
                    Please keep working on the library, as I think it would be a great extension to SpringIntegration. The transaction support is nice and the option to use SES is also really cool. SNS would be a nice addition to round out the AWS messaging services available.

                    Thanks,

                    Mike

                    Comment


                    • #11
                      Hello Mike, thanks a lot for your detailed feedback. Sorry to hear you are unable to use it for the reasons you have mentioned and i will work on those to sort the issues out.
                      Regarding your concerns, please find my answer to them below.

                      For various reasons, we cannot upgrade to Spring 3.1, but for some reason, your SQS adapter uses a ConversionService implementation only available in 3.1. I don't know why your SQS adapter needs to handle any conversion at all. Isn't that the job of Message Transformers?
                      The messages sent to SQS can only contain string payload, where as, we cannot restrict the Spring integration message sent to the outbound adapter to have a payload of type String only. It can contain any type as long as the conversion service can convert the object to string and vice versa. This is where the conversion service comes into picture. In its absence, the user would have to convert the user provided object to a type string say using transformers as you suggested (which should also contain the original class type so that it can be converted back to the object ) and vice versa upon receiving the string message from the SQS queue. Also, when I developed this library, i had kept in mind spring integration version 2.1.x which makes use of conversion services as well.
                      I will however go though this thing again and rethink on it to eliminate the use of conversion services.

                      BTW, what version of spring and spring integration are you using ?

                      While the XML configuration offered by your XSD makes dropping in the functionality very simple, it does not allow for overriding various components. It would be great if I could just point it to a reference of my sqsClient, which might be a slightly different implementation rather than the default (add encryption, etc.). Also, you use a default Json message transformer, but don't have the configuration in place to use a different message transformer, or even customize the configuration of the Json transformer. We have some Jackson modules that are necessary to serialize/deserialize our data, but we can't add that in with the XSD configuration.
                      This shouldn't be too difficult to allow accepting an instance of sqs client and the transformer to use. There is an interface org.springframework.integration.aws.sqs.core.Amazo nSQSOperations which has methods to send, receive and delete messages from the SQS queue. Providing an implementation of this interface should override the default implementation that is used to send and receive messages.
                      Allowing to accept a user provided (de)serializer is possible as well which should override the default JSON one. We will be accepting an implementation of the interface org.springframework.integration.aws.sqs.core.Amazo nSQSMessageTransformer

                      The class org.springframework.integration.aws.sqs.core.Amazo nSQSMessage is a container for the various headers,payload and the original type of the payload received by the adapter. This instance needs to be serialized into a string value that is sent as the payload of the SQS queue and then deserialize it back to the AmazonSQSMessage instance.

                      I'm not sure why you need to use your own interface to describe the AwsCredentials. AwsCredentials is already an interface for which there are multiple implementations. We have one that retrieves our credentials from a secure store, but can't use it with your library because it doesn't implement your interface
                      The reason why AWS credentials are represented using the interface org.springframework.integration.aws.core.AmazonWSC redentials is because i didn't want any dependency for the end user on AWS SDK classes. For example, if we start allowing the user to provide an implementation of the AmazonSQSOperations, the implementation can very well be implemented using some propitiatory implementation of perhaps using the REST API but not using AWS SDK. In that case, just for the AWS credentials class we'll have to include the SDK classes. Can you give me some more details about the issue you are facing on this one. I am sure we can find some solution to it which may possibly need a new/custom implementation of the
                      org.springframework.integration.aws.core.AmazonWSC redentials


                      So to summarize,
                      I can address your concern 2, on providing me with some details your concern 3 can also be addressed ( i'm hoping to )
                      For the 1st concern, its a tough call. Eliminating the conversion service means effectively restricting the adapter to accept messages with string payload only. However, if that is the case, the adapter implementation gets simplified to a great extent eliminating the need of a conversion service and thus addressing your concern one. The problem is to decide a draw a line on what needs to be provided, or, is the responsibility of the end user and what needs to be handled by the adapter.

                      Can you perhaps briefly give your use case that can help provide a better adapter implementation?
                      Mark, Oleg, Gary, Gunnar
                      Can you throw some light on this and share your thoughts/views on it? I'll need you'll help for sorting the issue faced by using the conversion service

                      Comment


                      • #12
                        Hi Amol

                        I did see the various interfaces and their default implementations. It's nice to be able to provide alternate implementations of those to plug in our own custom logic, but I don't see the point in providing interfaces to implement when there are already common interfaces that represent the various components. For instance, the Transformer interface already handles converting the payload from one type to another. But there are many implementations of the Transformer interface and other libraries are coded to deal with that interface, but not with your AmazonSQSMessageTransformer. Additionally, implementing your AmazonSQSMessageTransformer means if we want to plug in someone else's SNS library to route some messages to that channel, we have to implement the transformation logic twice. Meanwhile, there's a perfectly suitable JsonToObjectTransformer that is configurable and implements a common interface that all Spring Integration libraries will be able to deal with and can be plugged in anywhere. Similarly, we can implement your AmazonSQSOperations and add the logic to use the AmazonSQS client, but why bother implementing your interface when we could just implement the MessageHandler interface and reduce our dependencies? And you're correct that the ConversionService is common in the SpringIntegration framework, but everywhere, the conversion is invisible, rather than explicit. So my message endpoint can implement a method, processMessage(MyDataType data) and just registering a ConversionService that knows how to go from String to MyDataType is enough to make it all happen.

                        I'll give you a brief overview of our use case- we're processing a number of feeds from various feed sources in the cloud. During feed validation and processing we enqueue messages to SQS for the processors/validators to pick up and process. We use an encrypting SQS client so that our data isn't transmitted in plain text and we use a custom implementation of the AWSCredentials interface that retrieves our credentials from a secure store (so we don't have to store our credentials in code/configuration and credential rotation can happen without deployment). Our implementations are also wrapped in retrying decorators so short network blips don't short circuit the entire processing. In the future we want to be able to route messages for different feeds to different channels, whether different SQS queues, SNS notifications or e-mails via SES so each of our customers can be notified of their own feeds in whatever manner is appropriate so their systems can, for instance, read messages from their queue or subscribe to an SNS channel for a feed they depend on, etc.

                        Your library is very appealing as it provides two of the notification mechanisms and SNS seems like an obvious choice for a future extension. However, in your effort to decouple your library from the Spring abstractions and the AWS library, you're forcing users to couple themselves to you instead. Most of us are perfectly happy binding our systems to Spring code (we use @Component and @Autowired throughout our codebase knowing that Spring is ubiquitous within our systems). But coupling our components to your library isolates those components from the rest of the system, resulting in a lot of duplication. Spring is about simplification and pluggability. Don't force someone to implement interfaces when they can wire together loosely coupled components declaratively instead.

                        Best of luck with your library.

                        Mike

                        Comment


                        • #13
                          the Transformer interface already handles converting the payload from one type to another. But there are many implementations of the Transformer interface and other libraries are coded to deal with that interface, but not with your AmazonSQSMessageTransformer.
                          The transformer here is used by the AmazonSQSOperationsImpl to transform the AmazonSQSMessage instance to a JSON string and back to the AmazonSQSMessage instance. Trouble with using the spring's transformer interface it has the following method

                          Message<?> transform(Message<?> message);


                          If i do need to use this transformer, then the transformation needs to be done before the message is being given to the AmazonSQSService. I will have a look at this one though and see if spring integration's transformers can be used eliminating the use of conversion services and avoid implementing the transformer interface of the framework. Will need to rework on the design a bit, will update you on that

                          If you choose to provide your own implementation of AmazonSQSOperations, you might not even need to have a transformer as that implementation might be doing the conversion by itself

                          Similarly, we can implement your AmazonSQSOperations and add the logic to use the AmazonSQS client, but why bother implementing your interface when we could just implement the MessageHandler interface and reduce our dependencies?
                          If you have the MessageHandler handler implementation already in place, then the framework has nothing left in it. That is the reason why i feel providing a generic interface makes sense. Also, MessageHandler is used for outbound adapter, what for inbound channel adapter? The user then will have to also provide an implementation of MessageSource.

                          And you're correct that the ConversionService is common in the SpringIntegration framework, but everywhere, the conversion is invisible, rather than explicit. So my message endpoint can implement a method, processMessage(MyDataType data) and just registering a ConversionService that knows how to go from String to MyDataType is enough to make it all happen.
                          Isn't it transparent here? Nowhere, you need to explicitly use the conversion service. If an appropriate converter is found, your payload will be converted to string and vice versa transparently.

                          Your library is very appealing as it provides two of the notification mechanisms and SNS seems like an obvious choice for a future extension. However, in your effort to decouple your library from the Spring abstractions and the AWS library, you're forcing users to couple themselves to you instead.
                          Your application's other components do not depend on this interface. If you already have an implementation of the class that does these SQS operations for you, you just need to be implementing this interface and delegate calls to your implementation. You would then need to just define this in your config and pass the adapter a reference to it. If you choose not to use the library all you need you to do is delete this class and the corresponding configurations. This is IMHO

                          SNS is definitely on the radar and am planning to include support for it, but i am deferring it till the existing adapter implementations become relatively stable.
                          Last edited by Amol Nayak; Apr 5th, 2012, 11:41 AM. Reason: Added missing details

                          Comment


                          • #14
                            Hi Amol

                            If you have the MessageHandler handler implementation already in place, then the framework has nothing left in it. That is the reason why i feel providing a generic interface makes sense.
                            Here, I think, is where the discrepancy lies. It seems like you want to provide a framework for implementing Amazon SQS messages. But we don't want a framework. We've already chosen a framework with the appropriate abstractions: Spring Integration itself satisfies all of the abstractions and common interfaces. What we need are simple adapters to make SQS, SNS and SES available channels to send/receive messages using existing AWS APIs. Additionally, we need common logic to be shared between all channels, such as type transformations, filtering, etc. By adding another abstraction layer beneath Spring Integration for SQS message transformations, you're making it necessary to implement that transformation for every adapter we want to include.
                            If you choose to provide your own implementation of AmazonSQSOperations, you might not even need to have a transformer as that implementation might be doing the conversion by itself
                            Again, going this route, our Amazon SNS implementation would then have to implement the same conversion logic.
                            Your application's other components do not depend on this interface. If you already have an implementation of the class that does these SQS operations for you, you just need to be implementing this interface and delegate calls to your implementation. You would then need to just define this in your config and pass the adapter a reference to it.
                            Again, this is the exact same work that would be required to implement the MessageHandler and MessageSource interfaces. The difference being that the MessageHander and MessageSource interfaces are common interfaces in the Spring Integration framework, therefore plug in directly, giving us more flexibility and making the components cooperative with other components in the framework.

                            Here is the configuration we would like to have in place.
                            Code:
                                <!-- inbound -->
                                <aws-sqs:inbound-channel-adapter
                                    sqsClient="myClientImpl"
                                    channel="inboundChannel"
                                    sqs-queue="https://queue.amazonaws.com/myworkqueue"
                                    transactional="false"
                                    max-redelivery-attempts="3">
                                    <int:json-to-object-transformer type="com.foo.MyDataClass" object-mapper="objectMapper"/>
                                    <int:transformer expression="payload.dataField" />
                                    <int:filter ref="myMessageFilterBean" discard-channel="notReadyChannel" />
                                    
                                    <int:service-activator ref="dataProcessor" method="processData" />
                                </aws-sqs:inbound-channel-adapter>
                            
                                <!-- outbound -->
                                <int:chain output-channel="myCustomerOutput" id="customerOutputChannel">
                                    <int:transformer ref="myCustomersCustomDataTransformer" />
                                    <aws-sqs:outbound-channel-adapter
                            		sqsClient="myClientImpl"
                            		channel="inboundChannel"
                            		sqs-queue="https://queue.amazonaws.com/myCustomersQueue" />
                                </int:chain>
                            Here we have the SQS messaging set up as two message chains, including filters and transformers. Note that the input channel uses a different transformer from the output channel. The difference is that we produce the input messages being consumed here, therefore always use the same message format. Our customers, however, have different serialization formats- some use JSON, but other use a binary serialization format. The goal is to make it easy to route the right messages to the right customer queue using the right transformer. As mentioned, some messages will go out via SQS, but others will use SNS or SES. Message transformers are common among several output channels, so it does not make sense to implement the conversion in the SQS Operations impl, then implement it again in SNS Operations Impl. Additionally, any new message solutions that might be added in the future (e.g., AMQP, etc.) should also be able to utilize the existing tools.

                            Hope this helps to clarify the issues for you.

                            Mike

                            Comment


                            • #15
                              Hi Amol,

                              We are not able to use the sqs adapter for queues which are in different endpoint than queue.amazonaws.com
                              The reason being amazon sqs client sets as default endpoint queue.amazonaws.com and then ignores the endping information from the queue urls.

                              May be adding an attribute to adapter xml for specifying the endpoint would be useful.

                              This is very useful library, thanks a lot
                              Bilgin

                              Comment

                              Working...
                              X