Announcement Announcement Module
Collapse
No announcement yet.
Occasional deadlocks on MapJobRegistry.getJob(...) Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Occasional deadlocks on MapJobRegistry.getJob(...)

    I am using Spring Batch 2.1.0 for a large number of data feeds that run as often as every 15 seconds. Things will run fine for a few weeks or so, but then a deadlock issue occurs (randomly of course) that tends to wreck my day. The deadlock occurs within a synchronized block in MapJobRegistry.getJob() line 78.

    I have a bounded thread pool (from Quartz) and eventually all the threads lock up on this method, and so the system grinds to a halt and has to be restarted. I confirmed this by dumping all the thread stacktraces using VisualVM. I can also see the blocked threads in the VisualVM GUI.

    It looks like there is no easy way to make this call unsynchronized. In my case, I'd much rather deal with the consequences of a race condition than have the whole system lock up on the getJob() call.


    A sample stacktrace from one of the threads is below. There are many such threads and all are BLOCKED on this single method at line 78 in MapJobRegistry.

    "job_fetch_data" prio=6 tid=0x00a08000 nid=0x13a4 waiting for monitor entry [0x2947f000]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at org.springframework.batch.core.configuration.suppo rt.MapJobRegistry.getJob(MapJobRegistry.java:78)
    - waiting to lock <0x0ed65f78> (a java.util.HashMap)


    Perhaps it is not a deadlock, but some thread is not releasing the lock for another reason. The end result is my entire system is hosed because of a single ill-behaved thread. I am still investigating and will report my findings.

    -Trey

  • #2
    Found the problem

    I found the thread which is holding up the show:

    If you look at this line:
    org.springframework.batch.core.configuration.suppo rt.ClassPathXmlApplicationContextFactory$ResourceX mlApplicationContext.<init>(ClassPathXmlApplicatio nContextFactory.java:83)

    There is a call to refresh(), which loads a new Spring application context. However, due to a networking issue the loading of the XML context hangs when trying to do the schema validation.




    "jobDetail_foo-78" prio=6 tid=0x00a05c00 nid=0x165c runnable [0x284ce000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream. java:129)
    at java.io.BufferedInputStream.read1(BufferedInputStr eam.java:256)
    at java.io.BufferedInputStream.read(BufferedInputStre am.java:317)
    - locked <0x10ba82d8> (a java.io.BufferedInputStream)
    at sun.net.www.MeteredStream.read(MeteredStream.java:116)
    - locked <0x10baa848> (a sun.net.www.MeteredStream)
    at java.io.FilterInputStream.read(FilterInputStream.j ava:116)
    at sun.net.www.protocol.http.HttpURLConnection$HttpIn putStream.read(HttpURLConnection.java:2512)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityM anager$RewindableInputStream.read(XMLEntityManager .java:2961)
    at com.sun.org.apache.xerces.internal.impl.io.UTF8Rea der.read(UTF8Reader.java:299)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityS canner.load(XMLEntityScanner.java:1742)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityS canner.skipSpaces(XMLEntityScanner.java:1492)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocum entScannerImpl.scanAttribute(XMLNSDocumentScannerI mpl.java:442)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocum entScannerImpl.scanStartElement(XMLNSDocumentScann erImpl.java:277)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumen tFragmentScannerImpl$FragmentContentDriver.next(XM LDocumentFragmentScannerImpl.java:2755)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumen tScannerImpl.next(XMLDocumentScannerImpl.java:648)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocum entScannerImpl.next(XMLNSDocumentScannerImpl.java: 140)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumen tFragmentScannerImpl.scanDocument(XMLDocumentFragm entScannerImpl.java:511)
    at com.sun.org.apache.xerces.internal.impl.xs.opti.Sc hemaParsingConfig.parse(SchemaParsingConfig.java:4 35)
    at com.sun.org.apache.xerces.internal.impl.xs.opti.Sc hemaParsingConfig.parse(SchemaParsingConfig.java:4 91)
    at com.sun.org.apache.xerces.internal.impl.xs.opti.Sc hemaDOMParser.parse(SchemaDOMParser.java:510)
    at com.sun.org.apache.xerces.internal.impl.xs.travers ers.XSDHandler.getSchemaDocument(XSDHandler.java:1 802)
    at com.sun.org.apache.xerces.internal.impl.xs.travers ers.XSDHandler.parseSchema(XSDHandler.java:531)
    at com.sun.org.apache.xerces.internal.impl.xs.XMLSche maLoader.loadSchema(XMLSchemaLoader.java:552)
    at com.sun.org.apache.xerces.internal.impl.xs.XMLSche maValidator.findSchemaGrammar(XMLSchemaValidator.j ava:2436)
    at com.sun.org.apache.xerces.internal.impl.xs.XMLSche maValidator.handleStartElement(XMLSchemaValidator. java:1781)
    at com.sun.org.apache.xerces.internal.impl.xs.XMLSche maValidator.startElement(XMLSchemaValidator.java:7 05)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocum entScannerImpl.scanStartElement(XMLNSDocumentScann erImpl.java:400)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocum entScannerImpl$NSContentDriver.scanRootElementHook (XMLNSDocumentScannerImpl.java:626)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumen tFragmentScannerImpl$FragmentContentDriver.next(XM LDocumentFragmentScannerImpl.java:3103)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumen tScannerImpl$PrologDriver.next(XMLDocumentScannerI mpl.java:922)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumen tScannerImpl.next(XMLDocumentScannerImpl.java:648)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocum entScannerImpl.next(XMLNSDocumentScannerImpl.java: 140)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumen tFragmentScannerImpl.scanDocument(XMLDocumentFragm entScannerImpl.java:511)
    at com.sun.org.apache.xerces.internal.parsers.XML11Co nfiguration.parse(XML11Configuration.java:808)
    at com.sun.org.apache.xerces.internal.parsers.XML11Co nfiguration.parse(XML11Configuration.java:737)
    at com.sun.org.apache.xerces.internal.parsers.XMLPars er.parse(XMLParser.java:119)
    at com.sun.org.apache.xerces.internal.parsers.DOMPars er.parse(DOMParser.java:235)
    at com.sun.org.apache.xerces.internal.jaxp.DocumentBu ilderImpl.parse(DocumentBuilderImpl.java:284)
    at org.springframework.beans.factory.xml.DefaultDocum entLoader.loadDocument(DefaultDocumentLoader.java: 75)
    at org.springframework.beans.factory.xml.XmlBeanDefin itionReader.doLoadBeanDefinitions(XmlBeanDefinitio nReader.java:388)
    at org.springframework.beans.factory.xml.XmlBeanDefin itionReader.loadBeanDefinitions(XmlBeanDefinitionR eader.java:334)
    at org.springframework.beans.factory.xml.XmlBeanDefin itionReader.loadBeanDefinitions(XmlBeanDefinitionR eader.java:302)
    at org.springframework.beans.factory.support.Abstract BeanDefinitionReader.loadBeanDefinitions(AbstractB eanDefinitionReader.java:143)
    at org.springframework.context.support.AbstractXmlApp licationContext.loadBeanDefinitions(AbstractXmlApp licationContext.java:122)
    at org.springframework.context.support.AbstractXmlApp licationContext.loadBeanDefinitions(AbstractXmlApp licationContext.java:92)
    at org.springframework.context.support.AbstractRefres hableApplicationContext.refreshBeanFactory(Abstrac tRefreshableApplicationContext.java:130)
    at org.springframework.context.support.AbstractApplic ationContext.obtainFreshBeanFactory(AbstractApplic ationContext.java:465)
    at org.springframework.context.support.AbstractApplic ationContext.refresh(AbstractApplicationContext.ja va:395)
    - locked <0x10b7fbf0> (a java.lang.Object)
    at org.springframework.batch.core.configuration.suppo rt.ClassPathXmlApplicationContextFactory$ResourceX mlApplicationContext.<init>(ClassPathXmlApplicatio nContextFactory.java:83)
    at org.springframework.batch.core.configuration.suppo rt.ClassPathXmlApplicationContextFactory$ResourceX mlApplicationContext.<init>(ClassPathXmlApplicatio nContextFactory.java:77)
    at org.springframework.batch.core.configuration.suppo rt.ClassPathXmlApplicationContextFactory.createApp licationContext(ClassPathXmlApplicationContextFact ory.java:70)
    at org.springframework.batch.core.configuration.suppo rt.ApplicationContextJobFactory.createJob(Applicat ionContextJobFactory.java:55)
    at org.springframework.batch.core.configuration.suppo rt.MapJobRegistry.getJob(MapJobRegistry.java:81)
    - locked <0x0ed65f78> (a java.util.HashMap)
    at org.springframework.batch.core.configuration.suppo rt.ClassPathXmlJobRegistry.getJob(ClassPathXmlJobR egistry.java:63)
    at com.foo.etl.platform.JobLauncherDetails.executeInt ernal(JobLauncherDetails.java:93)
    at org.springframework.scheduling.quartz.QuartzJobBea n.execute(QuartzJobBean.java:86)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:2 02)
    at org.springframework.core.task.SimpleAsyncTaskExecu tor$ConcurrencyThrottlingRunnable.run(SimpleAsyncT askExecutor.java:229)
    at java.lang.Thread.run(Thread.java:619)

    Locked ownable synchronizers:
    - None

    Comment

    Working...
    X