Announcement Announcement Module
Collapse
No announcement yet.
Specifying a JobJar in the Tool Tasklet. Page Title Module
Move Remove Collapse
This topic is closed
X
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • #46
    1. Yes you can. I've just committed a fix for this in master (https://jira.springsource.org/browse/SHDP-79). This applies for files, archives and libs and you can specify even patterns - files="dev/props.properties, cfg/*.properties"
    2. M2 is going to be released this week. Watch the forum for the release announcement.

    Comment


    • #47
      Sounds good, thanks.

      Comment


      • #48
        1.0.0.M2 was just released. See http://forum.springsource.org/showth...-0-M2-Released

        Comment


        • #49
          Thanks Costin,

          I just noticed some duplications in the dependencies. For example:
          Code:
            Found in:
              commons-beanutils:commons-beanutils:jar:1.7.0:compile
              commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
          I would just advice to use Maven Enforcer plugin:

          Code:
          <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-enforcer-plugin</artifactId>
           <version>1.1</version>
          
           <dependency>
               <groupId>org.codehaus.mojo</groupId>
               <artifactId>extra-enforcer-rules</artifactId>
               <version>1.0-alpha-3</version>
           </dependency>

          Comment


          • #50
            Those are not from our project - probably are transitive dependencies from different dependencies. And we are not using Maven but Gradle.
            We could try to find out who pulls them in but even then excluding them will most likely cause issue - we could exclude 1.8 and leave 1.7 but still... It will end up as a messy situation trying to 'fix' something that we don't control...

            Comment


            • #51
              Ahh, I see. We are using Maven Enforcer, since having duplicate classes in the classpath can potentially cause unknown issues and that has been the case several times when we had transitive dependencies that would interfere with each other.
              I have excluded 1.8 from our project, but agree with you that it's not worth spending too much effort on that, unless it is causing some weird behavior.

              Sincerely,
              David

              Comment


              • #52
                Hi Costin,

                Is there a way to output Job Counters of the executing Hadoop Tool Tasklet?


                Sincerely,
                David

                Comment


                • #53
                  Not out of the box. Care to give a code example of what you are looking for?

                  Cheers.

                  Comment


                  • #54
                    We are using org.apache.hadoop.mapred.Reporter for doing some reporting when running hadoop jobs.
                    We have enumeration with the counters and we do reporter.incrCounter(Enum, 1), and after a specific job is finished, all of the Job Counter statistics will be shown, something like:

                    Counter Map Reduce Total
                    Value1 0 932,644 932,644
                    Value2 0 46,125,154 46,125,154
                    Value3 0 932,644 932,644

                    As you can see, same counter can be incremented both on the mapper and reducer sides, if needed.

                    Let me know if you need more information.

                    P.S: Btw, tried to send a message to you through this forum, but seems that your inbox is full

                    Comment


                    • #55
                      Right but I'm still missing on what type of work SHDP can do? As far as I can tell you don't need any specific configuration for this to work? Or am I missing something?

                      P.S. Yeah, my inbox gets full every months or so, and I ended up cleaning it some years ago...

                      Comment


                      • #56
                        Ok, so usually when I am running the job from console after the job ends, all these statistics are printed out, so I can crawl the logs and extract important information for reporting purposes, such as sending email after each job run.
                        If SHDP can print out all these statistics after it executes the jobs, that would be great, what do you think?

                        Comment


                        • #57
                          I think I know what you're issue it - the job tasklet runs the job in a non-verbose manner. I can make that configurable so the information shows up. Out of curiosity how are you using this data - nobody really reads the logs.

                          Comment


                          • #58
                            Suppose that each job calculates some stats during execution, for example:
                            • TOTAL_USERS
                            • ACTIVE_USERS
                            • SUBSCRIBER_USERS
                              ...

                            These are incremented in the reducer, as it finds more Users of a specific type. Please note that these numbers might be different depending on when it is run and for which geographic location, so we need to know these numbers to understand for example, how did our Marketing Campaign X impact user growth, or decline, etc...

                            This is just a simple example, since the counters are of different types and can be used for many other purposes.

                            Comment


                            • #59
                              Right. But the counters are currently incremented right - you just can't see their output in the console, is that right?

                              Comment


                              • #60
                                When I am running the job from console, the counter values are printed after the job is done.
                                From the SHDP perspective, you are right, they are actually incremented, but I can't see them in console.
                                Last edited by davidgevorkyan; Jun 27th, 2012, 05:12 PM.

                                Comment

                                Working...
                                X