Announcement Announcement Module
Collapse
No announcement yet.
Spring Data Infinite Loop in HashMap in PreferredConstructor and CustomConversions Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Spring Data Infinite Loop in HashMap in PreferredConstructor and CustomConversions

    Spring Data Infinite Loops in Production

    This post is related to an issue we have started seeing in production at the company where I work. I am trying to figure out where to take this next. Should I proceed to filing a bug report, or do I need to try and collect more information?

    It started happening a few weeks ago. We would see a box CPU suddenly go to 100% indefinitely until the app was redeployed. Fortunately, we did take thread dumps, and in one case a heap dump. We were able to see that there is a recurring infinite loop in two places in Spring Data.

    This has happened a total of 3 times now. Given the number of nodes we run in PROD, INT and QA (some number > 30), and the length of time we have been in production on this stack, we would characterize it as a rare occurrence. But it worries us.

    One could ask "what changed in the application to make this start happening?". Ack. We are in our busy season right now, so a lot of commits went into our application for the weeks prior to the first occurrence. And given the rare nature of the problem, it would be difficult to really identify even the time range of commits that we would need to evaluate. Root Cause

    In all 3 cases, the infinite loop is coming from concurrent updates to the thread-unsafe HashMap class. We see it happening within two Spring Data classes:
    • org.springframework.data.mapping.PreferredConstruc tor
    • org.springframework.data.mongodb.core.convert.Cust omConversions
    I could explain why HashMap is getting stuck in infinite loops, but several others have already done a great job doing it, so I refer you there:
    Stack Trace Snippets

    These stack traces come from our thread dumps. Each stack is seen multiple times when the problem occurs.

    PreferredConstructor
    "http-bio-8080-exec-235" daemon prio=10 tid=0x00007fb19c0a9800 nid=0x5337 runnable [0x00007fb228953000]
    java.lang.Thread.State: RUNNABLE
    at java.util.HashMap.getEntry(HashMap.java:469)
    at java.util.HashMap.get(HashMap.java:421)
    at org.springframework.data.mapping.PreferredConstruc tor.isConstructorParameter(PreferredConstructor.ja va:120)
    at org.springframework.data.mapping.model.BasicPersis tentEntity.isConstructorArgument(BasicPersistentEn tity.java:98)
    at org.springframework.data.mongodb.core.convert.Mapp ingMongoConverter$1.doWithPersistentProperty(Mappi ngMongoConverter.java:252)
    at org.springframework.data.mongodb.core.convert.Mapp ingMongoConverter$1.doWithPersistentProperty(Mappi ngMongoConverter.java:249)
    at org.springframework.data.mapping.model.BasicPersis tentEntity.doWithProperties(BasicPersistentEntity. java:257)
    at

    CustomConversions

    "http-bio-8080-exec-75" daemon prio=10 tid=0x00007f3688018000 nid=0x6c2e runnable [0x00007f376e9ef000]
    java.lang.Thread.State: RUNNABLE
    at java.util.HashMap.getEntry(HashMap.java:469)
    at java.util.HashMap.get(HashMap.java:421)
    at org.springframework.data.mongodb.core.convert.Cust omConversions.getCustomReadTarget(CustomConversion s.java:317)
    at org.springframework.data.mongodb.core.convert.Cust omConversions.hasCustomReadTarget(CustomConversion s.java:281)
    at org.springframework.data.mongodb.core.convert.Mapp ingMongoConverter.read(MappingMongoConverter.java: 200)
    at org.springframework.data.mongodb.core.convert.Mapp ingMongoConverter.readCollectionOrArray(MappingMon goConverter.java:791)
    at


    More Information

    I posted a lot more information about this issue here:

    http://peterlairdtech.blogspot.com/2...ashmap-in.html

    Please let me know if there are things I can do to help diagnose this further. Thanks!


  • #2
    This is tracked via: https://jira.spring.io/browse/DATACMNS-518

    Please, move your question to the StackOverflow - we are going to close this forum soon and rely on SO.
    We need to clean this forum (old unanswered question) before close it.

    Thanks for understanding

    Comment

    Working...
    X