Announcement Announcement Module
Collapse
No announcement yet.
Redis store inbound adapters OOM on large datasets Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Redis store inbound adapters OOM on large datasets

    Hi Gary,
    Back again .This time request some help on the redis adapter. My need is to poll a particular ZSET in regular intervals and to get all the items in that ZSET whose score is within a range. The score is actually a system time in current milliseconds and we poll to consume all elements who lie in the next poll range. i.e if we are polling at time t0, we expect to get all items who have a score within t0 and t0+ poll time. That said, I was trying to do some performance test by adding about a million items in a single poll period and used a message store inbound adapter to consume them and log. However I am getting an OOM understandably when java tries to deserialize 1000000 items. Is there any methodology by which I could sort of restrict the number of items received (maybe paginate the set somehow to reduce load)? This is probably a redis question, but thought of running it through you guys since you guys are the most responsive of all the forums and kudos to that ! my configuration (with hard coded values) are as below:
    Code:
    	<int-redis:store-inbound-channel-adapter
    		id="yearlyBucketAdapter" redis-template="redisTemplate"
    		key="bucket:yearly" channel="yearlyChannel" auto-startup="false"
    		key-serializer="stringRedisSerializer" value-serializer="objectRedisSerializer"
    		hash-key-serializer="stringRedisSerializer" hash-value-serializer="objectRedisSerializer"
    		collection-type="ZSET" >
    		<int:poller fixed-rate="360000" time-unit="SECONDS"
    			max-messages-per-poll="2" />
    	</int-redis:store-inbound-channel-adapter>
    
    	<int:channel id="yearlyChannel"></int:channel>
    	<int:chain input-channel="yearlyChannel" output-channel="log">
    		<int:header-enricher>
    			<int:header name="redisStore" expression="payload" />
    		 <int:header name="startScore" expression="T(java.lang.System).currentTimeMillis()" />
    			<int:header name="endScore"
    				expression="T(java.lang.System).currentTimeMillis() + 86400000" /> 
    
    		</int:header-enricher>
    		<int:transformer
    			expression="payload.rangeByScore(headers.startScore,headers.endScore)" />
    		<int:transformer ref="redisTransformer" method="transform" />
    		<int:transformer
    			expression="headers.redisStore.removeByScore(headers.startScore,headers.endScore)" />
    	</int:chain>
    	<int:channel id="log" />
    	<int:logging-channel-adapter channel="log" />
    	<bean id="redisConnectionFactory"
    		class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory"
    		p:host-name="127.0.0.1" p:port="6379" />
    	<bean id="redisTransformer"
    		class="com.monitise.mvp.scheduler.redis.service.RedisTransformer" />
    	<bean id="stringRedisSerializer"
    		class="org.springframework.data.redis.serializer.StringRedisSerializer" />
    
    	<bean id="objectRedisSerializer"
    		class="org.springframework.data.redis.serializer.JdkSerializationRedisSerializer" />
    
    	<bean id="redisTemplate" class="org.springframework.data.redis.core.RedisTemplate">
    		<property name="connectionFactory" ref="redisConnectionFactory" />
    		<property name="keySerializer" ref="stringRedisSerializer"/>
    		<property name="hashKeySerializer" ref="stringRedisSerializer"/>
    		<property name="valueSerializer" ref="objectRedisSerializer"/>
    		<property name="hashValueSerializer" ref="objectRedisSerializer"/>
    			
    	</bean>

    The redis transformer just iterates and logs each set item.

    Thanks ,
    Anoop
    Last edited by anoop2811; Aug 22nd, 2013, 07:34 PM.
Working...
X