Announcement Announcement Module
Collapse
No announcement yet.
Specifying a JobJar in the Tool Tasklet. Page Title Module
Move Remove Collapse
This topic is closed
X
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • #31
    Great. Could you share more on how you pass the properties to your jobs and the use of Spring Batch?
    Do the jobs pass information between each other - and if so, how? Anything that you think it's missing?

    Cheers,

    Comment


    • #32
      Hi Costin,

      I did extensive testing with property files and here are my findings:

      Below is my tasklet configuration:
      Code:
      <hdp:tool-tasklet id="hadoopTasklet" scope="step" configuration-ref="hadoop-configuration"
                    tool-class=MyJob" jar="my_job.jar" properties-location="myProp1.properties" files="myProp1.properties">
        ...
      </hdp:tool-tasklet>
      Here is one of the spring configs in my_job.jar:
      Code:
          <bean id="propertyPlaceholderConfigurer"
              class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
              <property name="ignoreUnresolvablePlaceholders" value="true" />
              <property name="ignoreResourceNotFound" value="true" />
              <property name="locations">
                  <list>
                      <value>classpath:myProp1.properties</value>
                  </list>
              </property>
          </bean>
      I can tell you for sure that properties-location property is working, since all the values from the property file were loaded and can be seen in the Job Configuration page in Hadoop.

      At the same time the files property doesn't work, since I am getting the following errors:

      Code:
      attempt_201107061330_12011_r_000321_0: 2012-05-11 16:59:31,477 WARN  [PropertyPlaceholderConfigurer]
      Could not load properties from class path resource [myProp1.properties]: class path resource
      [myProp1.properties] cannot be opened because it does not exist
      ...
      I tried to change it to files="classpath:myProp1.properties" but it didn't solve the issue.

      When invoking from command line we were using -files fullPath/myProp1.properties


      Please let me know if I need to add something else.


      Sincerely,
      David

      Comment


      • #33
        It looks like you ran into a bug [1]
        The namespace parser currently doesn't properly parse the files/archives/libs properties (even though you did specify them).
        I'll try to address that shortly - unfortunately I'm travelling (again) as we speak and will be in transit until next week; in the worse case scenario I'll have an update by next weekend.

        Thanks for your feedback and patience.

        [1] https://jira.springsource.org/browse/SHDP-74

        Cheers,

        Comment


        • #34
          Hi Costin,

          Please let me know as soon as you have an update on this.


          Sincerely,
          David

          Comment


          • #35
            Haven't forgot this. Due to some hectic problem (and some bad food) I didn't get a chance to take a look at this - but I plan to do this shortly this weekend or early next week as I'm currently on my way home.

            Comment


            • #36
              Hi David,

              The issue has been fixed and the update pushed upstream and a nightly build published.

              P.S. The namespace was actually working but the tool wasn't configured properly - which has been now been addressed.

              Feedback is welcome!

              Comment


              • #37
                Hi Costin,

                Thanks for the update, I haven't been able to test the changes yet, since the cluster under a complete rebuild.
                I will do and let you know as soon as possible.

                Sincerely,
                David

                Comment


                • #38
                  Hi David,

                  Any update?

                  Comment


                  • #39
                    Hi Costin,

                    Sorry for the late reply, the cluster was rebuilt on Friday and I only got a chance to test it today.
                    I just run an end to end test of the project against spring-data-hadoop-1.0.0.BUILD-20120603.231511-119 and everything is working well. I will run another test tomorrow, will let you know if I find something else.


                    Sincerely,
                    David

                    Comment


                    • #40
                      That's good to hear. By the way, the latest snapshot also supports security (the docs should get in there by the end of the day) meaning you can now specify a "user" as an impersonation for running Tool#run.
                      For what it's worth, this is supported across the board by all MR components (streaming, job & tool) and Pig as well.

                      Comment


                      • #41
                        Hi Costin,

                        For the impersonation, can you elaborate a little what are you talking about? Are you using some property in hadoop, such as "hadoop.job.ugi"?

                        Sincerely,
                        David

                        Comment


                        • #42
                          See the docs at [1] - code can be executed using using a different entity (assuming the cluster has been configured accordingly).

                          [1] http://repo.springsource.org/libs-sn...PSHOT-docs.zip

                          Comment


                          • #43
                            By the way, another feature available in the latest nightly build is the fallback to mainClass for tool namespace, if a jar is mentioned and no tool* is specified. Basically if your main class is the Tool (applies to most cases I've seen so far), simply point the tool to the jar and everything will be scanned automatically.

                            Comment


                            • #44
                              By the way, the snapshot docs are now published only in browsing format as well:
                              http://static.springsource.org/sprin.../security.html

                              Comment


                              • #45
                                That's really cool, especially locating mainClass and considering that as Tool.
                                I have 2 questions:

                                1) Can I specify more than 1 property file in the files argument (are the file names comma separated)?
                                2) Is it possible to cut a milestone release with the current functionality, since we have been waiting for that and are very excited to use it in one of our applications?

                                Sincerely,
                                David

                                Comment

                                Working...
                                X