Announcement Announcement Module
Collapse
No announcement yet.
Transaction performance and how to lighten transaction manager ? Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Transaction performance and how to lighten transaction manager ?

    Hi folks !

    I'm working about a identity, access and role management with SDN 2.1.0-RC4 without AspectJ.
    This system aims to manage around 120 000 users and data is extracted from legacy and HR applications.
    I have from 500 to 30000 updates/day, so performance can be a touchy subject
    I ran some workbenches with default configuration.

    I used a very simple and stupid way.
    A simple class
    Code:
    @TypeAlias("event")
    @NodeEntity
    public class Event {
    
    	@GraphId
    	private Long graphId;
    	
    	@Indexed
    	private Long eventId;
    	
    	@RelatedTo(type="PREVIOUS_EVENT", direction=Direction.OUTGOING)
    	private Event previous;
    	
    	private String description;
    
            /.../
    }
    and I insert data one by one
    Code:
            /.../
    	/**
    	 * 
    	 */
    	public void loadAllData() {
    		
    		Event root = new Event();
    		root.setEventId(0L);
    		root.setDescription("ROOT");
    		
    		repository.save(root);
    		Event curr = root;
    		
    		
    		for(int i = 0; i< SIZE; i ++) {
    			curr = insertData(curr, i );
    		}
    	}
    	
    
    	/**
    	 * @param curr
    	 * @param i
    	 * @return
    	 */
    	public Event insertData(Event curr, int i) {
    		long lastTime = System.currentTimeMillis();
    		Event evt = new Event();
    		evt.setEventId(curr.getEventId()+1);
    		evt.setPrevious(curr);
    		evt.setDescription("event #"+i);
    		repository.save(evt);
    		curr = evt;
    		delais[i] = System.currentTimeMillis() - lastTime;
    		return curr;
    		
    	}
    /.../
    I test several by overloading theses methods

    1) use of @Transactional
    Code:
    	@Override
    	@Transactional
    	public Event insertData(Event curr, int i) {
    		return super.insertData(curr, i);
    	}
    2) Use of neo4j transaction
    Code:
     	@Override
    	public void loadAllData() {
    		gds = getContext().getBean(GraphDatabaseService.class);
    		super.loadAllData();
    	}       
    
    	@Override
    	public Event insertData(Event curr, int i) {                
    		Transaction tx = gds.beginTx();
    		try {
    			curr = super.insertData(curr, i);
    			tx.success();
    		} catch(Exception e) {
    			tx.failure();
    		} finally {
    			tx.finish();
    		}
    		return curr;
    			
    	}
    I know benchmark are a trolling subject and it's not what I want. So take theses values as they are: a stupid test

    Results
    JtaTransactionManager
    Mean 47,20 ms, Min 21,00 ms, Max 425,00 ms

    GraphDatabaseService
    Mean 0,90 ms, Min 0,00 ms, Max 3,00 ms

    Ok, JtaTransactionManager is very slow compared to native neo4j server, but JtaTransactionManager is a global and ambitious TransactionManager.

    My purpose is: how to lighten ou create a custom transactionManager with less ambition, a smaller scope but still using the @Transactional annotation ? May be I've missed something ?

    Thanks you for any potential help or advice.

    PS: my configuration

    Code:
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <beans xmlns="http://www.springframework.org/schema/beans"
           xmlns:context="http://www.springframework.org/schema/context"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xmlns:neo4j="http://www.springframework.org/schema/data/neo4j"
           xmlns:tx="http://www.springframework.org/schema/tx"
           xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
    		http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
    		http://www.springframework.org/schema/data/neo4j http://www.springframework.org/schema/data/neo4j/spring-neo4j-2.0.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.5.xsd">
    
        <context:annotation-config/>
        <context:spring-configured/>
        
        <context:component-scan base-package="benchmark"/>
    
        <neo4j:repositories base-package="benchmark.repository"/>
        <neo4j:config graphDatabaseService="graphDatabaseService"/>
        
        
        <bean id="graphDatabaseService" class="org.neo4j.kernel.EmbeddedGraphDatabase" 
          destroy-method="shutdown">
             <constructor-arg index="0" value="data/benchmark.db" />
        </bean>
                        
    
    </beans>
    Marc DeXeT

  • #2
    #1 you should create bigger transactions (up to and around 10k items large) around your inserts not item by item tx, there the tx overhead will kill your performance
    #2 for creating the relationship you should rather use template.createRelationshipBetween(..., allowDuplicates=true) which is much faster as it delegates directly to the neo4j method w/o delta detection and

    Comment


    • #3
      Originally posted by MichaelHunger View Post
      #1 you should create bigger transactions (up to and around 10k items large) around your inserts not item by item tx, there the tx overhead will kill your performance
      #2 for creating the relationship you should rather use template.createRelationshipBetween(..., allowDuplicates=true) which is much faster as it delegates directly to the neo4j method w/o delta detection and
      and


      You're right, It's a successful strategy. It's speed up the process by a 3 factor for just 200 items large transaction.
      My concern is that modification flow is an event-driven, not a old classical nightly batch flow. So I can have one insert per minute or a big load of 50k modifications that come charging along from sources.

      May be must I have to create a specific transaction manager with AbstractPlatformTransactionManager that triggers commit following a size indicator for big flow and a timeout for small changes...
      Last edited by marc dexet; Oct 22nd, 2012, 08:16 AM.

      Comment


      • #4
        If you already have a messaging system this should be quite easy,
        just configure the endpoint listener to take let's say 10k large message packs from the queue, or wait at most 500ms before consuming messages. Then you should either have enough for a large transaction, or you have reacted within a reasonable time (latency).

        HTH

        Michael

        Comment

        Working...
        X