Announcement Announcement Module
Collapse
No announcement yet.
How to set number of reduce tasks in my Job with a Partitioner Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • How to set number of reduce tasks in my Job with a Partitioner

    Hi!, I'm new in this forum and I'm starting with spring-hadoop. I have some problems that I would like if someone could help me.

    I'm trying to test the Partitioner and I don't get to set up the reduce tasks number that I need for my Partitioner example in my pseudo-distributed environment.

    I've tried to do it in two ways
    Firstly, I've setup the task number at the configuration like that:

    Code:
    <hdp:configuration>
       	fs.default.name=hdfs://localhost:8020
    	mapreduce.job.tracker=localhost:8021
    	mapred.reduce.tasks=2
     </hdp:configuration>

    And later, I've tried to setup the task number in my job:

    Code:
    <hdp:job id="testPartitioner" reducer="MyReducer" mapper = "MyMapper" partitioner="MyPartitioner"
        	input-path="myInputPath" output-path="myOutputPath">
    
        	mapred.reduce.tasks=2
     </hdp:job>
    Both setup doesn't work and gives me the error:
    java.io.IOException: Illegal partition for 01-01-2013 (1).

    How can I do this configuration correctly?

    Thanks!!
Working...
X