Announcement Announcement Module
Collapse
No announcement yet.
How to execute a Hadoop Project with the "Run" Dialog in STS (from Shell it works) Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • How to execute a Hadoop Project with the "Run" Dialog in STS (from Shell it works)

    Hi,
    I created 2 Maven Projects:
    • com.ponyCircus
      - application Context and everything else (like public static void main...)
    • com.ponyCircus.mr
      - ExampleMapper
      - ExampleReducer

    In the pom.xml of com.ponyCircus i configured a dependency to com.ponyCircus.mr.


    Scenario A (working):
    I build the stuff with maven and execute the com.ponyCircus.Main class.
    With this everything is ok. (That means the configuration is fine).

    Scenario B (working):
    I checkout the com.ponyCircus example, configure it as Maven project. Then m2e will get the com.ponyCircus.mr.jar from the repository and link it as Jar-File.

    If i lunch it with the Run dialog (F11) everything is fine .

    This screenshot shows that it links the JAR File:
    http://s2server.de/success.png

    Scenario C (There i get the problem...):
    I checkout both projects and configure it as Maven Projects. Now m2e will Link the com.ponyCircus.mr Project (not the Jar File).

    If I lunch it, i will get a
    Code:
    java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.ponyCircus.mr.wordCount.WordMapper not found
    	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1571)
    	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)
    	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605)
    	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
    	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at javax.security.auth.Subject.doAs(Subject.java:396)
    	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    	at org.apache.hadoop.mapred.Child.main(Child.java:262)
    Caused by: java.lang.ClassNotFoundException: Class com.ponyCircus.mr.wordCount.WordMapper not found
    	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1477)
    	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1569)
    Exception because it was not able to identify and ship the correct jar file.

    This screenshot shows the problems (i guess):
    http://s2server.de/error1.png

    Is there a way to make it running with STS?

    P.S.: I don't know whether this is a STS or Hadoop Question. Please move if you feel it's not correct .
    Last edited by d0x; Feb 19th, 2013, 09:48 AM.

  • #2
    Have you tried jar-by-class (and jar) attributes?

    <hdp:job id="flattener" input-path="#{inputPath}"
    output-path="#{outputPath}"
    mapper="x.y.Importer$ImporterMapper"
    jar-by-class="x.y.Importer$ImporterMapper"
    jar="file:///E:/workspace-sts3/proj/target/proj.jar"/>

    Comment


    • #3
      Hi Saurabhjain,

      i haven't specified the "jar=" property. I can't put an absolute path there. And maybe also a relative path is not good .

      But i gave the jar-by-class property.

      Like this:
      Code:
      <?xml version="1.0" encoding="UTF-8"?>
      <beans xmlns="http://www.springframework.org/schema/beans"
      	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      	xmlns:hdp="http://www.springframework.org/schema/hadoop"
      	xmlns:p="http://www.springframework.org/schema/p"
      	xmlns:context="http://www.springframework.org/schema/context"
      	xsi:schemaLocation="
      		http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
      		http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
      		http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">
      
      
      	<context:component-scan base-package="com.ponyCircus" />
      
      	<hdp:configuration id="hadoopConfiguration">
      		fs.defaultFS=hdfs://carolin....net:8020
      		mapred.job.tracker=michaela.....net:8021
      	</hdp:configuration>
      	
      	<hdp:job id="wordCountJob"
      		mapper= "com.ponyCircus.mr.wordCount.WordMapper"
      		reducer="com.ponyCircus.mr.wordCount.WordReducer"
      		jar-by-class= "com.ponyCircus.mr.wordCount.WordMapper"
      		input-path="ponyCircusExample/input"
      		output-path="ponyCircusExample/output" />
      		
      </beans>

      Comment


      • #4
        As workaround we are using now the "External Tool Chain"-Configuration and the exec-maven-plugin.

        But that's a ugly workaround.

        With this we can just click the External Tool Chain button to test our code in the IDE. But the Run-Dialog and Debugging doesn't work.

        http://s2server.de/workaround.png

        Can someone suggest a more elegant way?

        Comment

        Working...
        X