Announcement Announcement Module
Collapse
No announcement yet.
Job not appearing on JobTracker user interface Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Job not appearing on JobTracker user interface

    While trying to use Spring Hadoop to run map-reduce using the maven-assembly approach outlined @ http://www.petrikainulainen.net/prog...apache-hadoop/

    The job runs successfully and generated output but does not appear on the Job Tracker user interface.

    Also, the logs show the warning: WARN [org.apache.hadoop.util.NativeCodeLoader] - <Unable to load native-hadoop library for your platform... using builtin-java classes where applicable>

    I am not sure if these two are inter-related. Any explanation of this behavior will be very helpful.

    Thanks,
    Saurabh

  • #2
    I tried running the wordcount example that comes bundled with spring-data-hadoop-0.9.0.RELEASE. Observation was same with this example: the Job Tracker user interface didn't track this job.

    Is this a issue with Spring Hadoop?

    Comment


    • #3
      Did you set 'mapred.job.tracker' property in <hdp:configuration>..</..> like $HADOOP_HOME/conf/mapred-site.xml?

      My spring-hadoop.xml is look like (spring-hadoop-1.0.0RC1)
      <hdp:configuration>
      fs.default.name=${hd.fs:hdfs://localhost:9000}
      mapred.job.tracker=${mapred.job.tracker:master:900 1}
      </hdp:configuration>
      ...
      There is no problem.

      Comment


      • #4
        Thanks shekhina. I was missing the "mapred.job.tracker" property in the hdp configuration. Adding that fixed the issue and I am now able to see the job tracked on the Job Tracker UI.

        Comment


        • #5
          shekhina is right (thanks for helping out).
          The underlying problem is that without setting up the job tracker, Hadoop will use the default one which can be a local tracker - meaning only one map/reduce task is executed on the local machine instead of the actual Hadoop cluster.
          This is a somewhat not-obvious side-effect that catches many (including us) by surprise.

          Comment

          Working...
          X