Announcement Announcement Module
Collapse
No announcement yet.
ToolTasklet classpath issue when running Hadoop jar Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • ToolTasklet classpath issue when running Hadoop jar

    2.2.0RC1.

    A weird ToolTasklet classpath issue we are dealing with:

    We have a Tool class that uses JCascalog.

    What is odd is I can run the Hadoop Tool class fine when I do it outsde of Spring, but when I jar it all up and run it as a ToolTasklet and call tasklet.setJar(myHadoopUberJar), it fails. When it executes the Clojure RT class doesn't like one enty on the classpath.

    As best I can discern, when JCascalog is called, Cascading initializes, and this causes the Clojure class RT to be initialized. RT apparently then scans through the classpath. It gets through a few hundred classes, then it gets to one Spring Batch class (StepExecution.class) that it throws an Exception on. (See stack trace below) This class is in a jar on the classpath, and in fact many other classes from the same jar and the same package were already read successfully.

    Looking in HadoopExecutor.runCode(), it seems if a jar is provided a new classloader based on the Hadoop jar is substituted on the execution thread as you would expect. What I can't figure out why one single class is failing.


    java.lang.ExceptionInInitializerError
    at clojure.core__init.__init0(Unknown Source)
    at clojure.core__init.<clinit>(Unknown Source)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:266)
    at clojure.lang.RT.loadClassForName(RT.java:2098)
    at clojure.lang.RT.load(RT.java:430)
    at clojure.lang.RT.load(RT.java:411)
    at clojure.lang.RT.doInit(RT.java:447)
    at clojure.lang.RT.<clinit>(RT.java:329)
    at cascalog.Util.<clinit>(Util.java:29)
    at jcascalog.Api.setApplicationConf(Api.java:99)
    at com.test.DataShredder.run(DataShredder.java:113)

  • #2
    This is indeed due to an interesting peculiarity of Clojure's runtime implementation. For whatever reason, RT class will fail to initialize if it was loaded by a classloader different from the context class loader of the executing thread. It may sound somewhat strange in terms of how this can actually happen in practice? But it is quite possible if the actual context classloader chooses "parent first" style of loading classes - thus delegating to its parent classloaders (or just the system classloader) first.

    This has been discussed on the following thread in this group in some detail: https://groups.google.com/forum/?hl=...er/Bh_YmI6e-wY

    Spring Hadoop chooses an interesting style of classloading delegation when it submits the Hadoop "uberjar" using its ParentLastURLClassLoader implementation. It actually delegates to the system classloader first, then tries to load classes from the uberjar, and only then goes to the parent classloader if it is available. So, if you're running a simple Java process (i.e. not inside an application server or web container), as can be the case
    in many situations including launching a simple unit test from inside an IDE, this ParentLastURLClassLoader effectively becomes a "parent first" classloader.

    Thus it creates an interesting situation: if clojure.jar is present on the classpath outside of the "uberjar" (not an unreasonable expectation), the RT class will be loaded by the system classloader when the context classloader is set to ParentLastURLClassLoader classloader. I have created an RFE in Spring Hadoop Jira to allow making classloading delegation configurable (i.e. allowing true "parent last" delegation as an option): https://jira.springsource.org/browse/SHDP-135.

    The way to work around these issues is to either ensure that no clojure.jar is present on the main classpath when submitting "uberjars" to Hadoop from inside a running live JVM process or replacing a context classloader with a custom "true parent last" classloader (assuming the SHDP-135 is not addressed) in a try/finally block.

    At this point it is unclear why Clojure RT class behaves the way it does. It is certainly somewhat "unconventional". It would be hugely beneficial and very interesting to know why the actual implementation of the context classloader should matter to the startup of Clojure runtime. I think the whole community would benefit greatly from this understanding.

    Comment

    Working...
    X