Announcement Announcement Module
Collapse
No announcement yet.
Batch message consuming with Spring AMQP & Spring Mail? Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Batch message consuming with Spring AMQP & Spring Mail?

    In my email send app, I use Spring AMQP storing email message that come from upper apps(exposing a REST to them). then fetch email message from queue and sending it to SMTP Server through Spring Mail.

    Now i have a tricky problem need to solve:
    For every email message that fetched from MQ, i open a Socket Connection(under the JavaMail Transport) between our app and SMTP Server for it, and close the connection after sending is complete. But, our apps is in China and SMTP Server is in Europe(a commercial SMTP service provider), the Socket Connection is expensive, on an average it taking around 5 seconds to open it(As a comparison, the actually sending just taking around 2 seconds).

    For better performance, I think i must reuse the Socket Connection for emails sending. In other words, sending them in batch(bulk? whatever). But i noticed that in Spring AMQP's SimpleMessageListenerContainer, the message is always consume by actually listener one by one, even though i can fetch them in batch by specifying prefetchCount to a larger number:
    Code:
    	private boolean doReceiveAndExecute(BlockingQueueConsumer consumer) throws Throwable {
    		Channel channel = consumer.getChannel();
    		for (int i = 0; i < txSize; i++) {
    			logger.trace("Waiting for message from consumer.");
    			Message message = consumer.nextMessage(receiveTimeout);
    			if (message == null) {
    				break;
    			}
    			try {
    				executeListener(channel, message);
    			} catch (ImmediateAcknowledgeAmqpException e) {
    				break;
    			} catch (Throwable ex) {
    				consumer.rollbackOnExceptionIfNecessary(ex);
    				throw ex;
    			}
    
    		}
    		return consumer.commitIfNecessary(isChannelLocallyTransacted(channel));
    	}
    Use prefetchCount of SimpleMessageListenerContainer, i can decreasing the interaction between our Mail App and AMQP Broker, but I also need decreasing the interaction between our Mail App and SMTP Server, sending email messages in batch. And, feedback mechanism of batch processing is tricky too(thus, which part in a batch is sending successful, which is not).


    Do you guys have any suggestions? All response is greatly appreciated.

    -Jason
    Last edited by Wuaner; Feb 20th, 2013, 09:22 PM. Reason: correcting spell

  • #2
    Perhaps subclass JavaMailSenderImpl, overriding getTransport() - you can bind the transport to the container thread and reuse it; you would need to invalidate it after a send failure.

    Comment

    Working...
    X