Announcement Announcement Module
Collapse
No announcement yet.
Spring Batch with HiveTasklet giving error Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Spring Batch with HiveTasklet giving error

    I am trying to execute the apachelog hive example with hiveTasklet.

    The hiveTasklet is launched but it throws error :

    16:44:49.246 [pool-2-thread-1] INFO org.apache.hadoop.hive.ql.exec.Task - set mapred.reduce.tasks=<number>
    16:44:49.256 [pool-2-thread-1] WARN o.a.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    16:44:49.256 [pool-2-thread-1] INFO o.a.hadoop.hive.ql.exec.ExecDriver - Generating plan file file:/tmp/impadmin/hive_2013-04-26_16-44-48_418_979559096410501356/-local-10002/plan.xml
    16:44:49.433 [pool-2-thread-1] INFO o.a.hadoop.hive.ql.exec.ExecDriver - Executing: /home/impadmin/hadoop-1.0.1/bin/hadoop jar /home/impadmin/hadoopBook/spring-data-book-master/hadoop/batch-wordcount-hive/target/appassembler/repo/hive-service-0.8.1.jar org.apache.hadoop.hive.ql.exec.ExecDriver -libjars file:///home/impadmin/hadoopBook/spring-data-book-master/hadoop/batch-wordcount-hive/target/appassembler/repo/hive-builtins-0.8.1.jar -plan file:/tmp/impadmin/hive_2013-04-26_16-44-48_418_979559096410501356/-local-10002/plan.xml ...............


    Exception in thread "main" java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.exec.ExecDriver
    at java.net.URLClassLoader$1.run(URLClassLoader.java: 202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.j ava:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:3 06)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:2 47)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:149 )
    Execution failed with exit status: 1
    16:44:49.781 [pool-2-thread-1] ERROR org.apache.hadoop.hive.ql.exec.Task - Execution failed with exit status: 1



    I could figure out that its because of change in hadoop program jar. It should pick hive-exec but it is picking hive-service.

    Executing: /home/impadmin/hadoop-1.0.1/bin/hadoop jar /home/impadmin/hadoopBook/spring-data-book-master/hadoop/batch-wordcount-hive/target/appassembler/repo/hive-service-0.8.1.jar org.apache.hadoop.hive.ql.exec.ExecDriver

    Could someone please help me to set the correct HiveJar.

    Thanks

  • #2
    Hi,

    I guess you modified the book example 'batch-wordcount' to include a hive tasklet? The dependency on hive can be found in the sample directory 'hive'. There are quite a number of dependencies for hive, the ExecDriver class is in hive-exec-0.8.1.jar.

    We are running into this issue ourselves recently, so expect to be able to provide a more updated example, one we sort it out ourselves. If you get it working, posting a gist would be appreciated.

    Cheers,
    Mark

    Comment

    Working...
    X