Announcement Announcement Module
Collapse
No announcement yet.
Spring/Hibernate/JPA Issue - EntityManager's session not properly closed/cleaned Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Spring/Hibernate/JPA Issue - EntityManager's session not properly closed/cleaned

    Spring: 2.5.5
    Hibernate/EntityManager: 3.3.0.SP1/3.4.0.GA
    WebSphere: 6.1.0.27
    DB2/zOS: v8

    We've encountered a Spring/JPA/Hibernate issue in production and it seems to surface only under load. The issue appears to be triggered by an insert failing with a ConstraintViolationException. In some instances of this exception, the Hibernate EntityManager associated with the request does not appear to be closed/cleaned properly. The next unlucky user to get this thread from the thread pool ends up receiving the same constraint violation (which is triggered by a flush at the end of whatever it was they were doing).

    It appears that the original JPA managed object is still hanging around waiting to be saved and, due to the nature of the error, it never will be. From that point on, the thread is unusable for some period of time (until WebSphere appears to purge it from the pool?). Local testing has not been able to recreate the issue -- we've even tried reducing the thread pool size without success.

    We've also done some initial research looking at the Spring and Hibernate code and nothing immediately jumps out. Our code uses Spring's JpaTransactionManager around POJO services, so we don't do anything special with the EntityManager's session. Our best guess is that somehow during the EntityManager close, the registerSynchronization() is failing to execute the afterCompletion() properly… but with all of the ThreadLocals Spring is using it could be anywhere. We've searched both the Spring and Hibernate bug repositories and can't find anything similar.

    Has anyone else seen this behavior and, if so, are there know fixes or a work around? Our plan is to upgrade to Spring 3.0.3.RELEASE and Hibernate 3.5.3-Final in the next release, but it would be nice to know conclusively that this issue will go away as end users are starting to complain about reliability of the application.

    - Shawn

  • #2
    As a follow-up to this comment, we discovered and fixed the root cause -

    In one instance we used programmatic transaction handling instead of @Transactional (it was needed at the time because we were integrating with a legacy subsystem that was not Spring aware). This custom transaction logic failed to handle a rare unchecked exception that was thrown by the subsystem and it left the transaction "open". Because of this logic failure, the doCleanupAfterCompletion() was never called and thus, the TransactionSynchronizationManager did not unbind the EntityManager from ThreadLocal.

    Interestingly, this improper transactional code does not cause the problem alone. The next user of that specific thread can do commit and rollback as normal; it is only when that user also encounters a UniqueConstraintViolation that a dirty object remains in the EntityManager's managed object cache. As this duplicate object can never be saved properly, it leaves the EntityManager in an invalid state for the rest of the life of that thread -- any user unlucky enough to get this thread receives the original UniqueConstraintViolation as it is triggered by an autoFlush. Under typical production volume, it can take some time for the corrupted thread to time out and, during this time, it creates what appeared to be random failures to the (unhappy) users of the system.

    Since the sequence of events that lead to this problem are relatively rare, it took us some time to track it down and recreate it locally. I suspect the reason that this happens is because the original transaction is the only one that has the isNewManagerHolder set to true (as the following transactions are all created in the context of the EntityManagerHolder left pinned to ThreadLocal). Since the cleanup logic is guarded to only be performed when isNewManagerHolder is true, we were left with a dirty ThreadLocal that would never be cleaned up.

    I wouldn't say this is a bug of any sort with the Spring code per se, but a very unfortunate situation created by our code's misuse of Spring transactional behavior (and the difficulty we encountered in debugging issues related to ThreadLocal).

    - Shawn
    Last edited by Shawn Sherwood; Jan 18th, 2011, 09:55 AM.

    Comment

    Working...
    X