Announcement Announcement Module
Collapse
No announcement yet.
spring hadoop, job properties questions Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • spring hadoop, job properties questions

    Hi,

    I have a working hadoop process.

    I would like to migrate it to spring framework using spring hadoop.

    I have a few questions.


    1. How does hadoop child procesess instatiate spring application context, Is the xml uploaded to distributed
    cache?

    2. configureReducerTypesIfPossible() at JobFactoryBean is not implemented, how does reducerKey and
    value types determined?

    3. How can I determine job properties like org.apache.hadoop.mapreduce.JobContext.setNumReduc eTasks() ,
    If they are not in spring-hadoop.xsd ?




    4.I would like to determine parameter 'input-path' dynamically at runtime, What is the recommened
    approcah?

  • #2
    1. Spring Hadoop is not used to instantiate spring application context in Hadoop child process, it is used to manage hadoop jobs from Spring. The xml which is used to define beans, jobs will not be uploaded to distributed cache. The xml is used by main application or spring batch to instantiate beans, the beans will call hadoop client API.
    2.Hadoop can determine type dynamically. Anyway, configureReducerTypesIfPossible() should be implemented later for safe.
    3.After getting the "Job" fro "JobFactoryBean", you can set properties on the job with Job api.
    4.You can use SpEL. to input-path property.
    Last edited by leejianwei; Mar 27th, 2012, 10:45 PM.

    Comment


    • #3
      How do I access the the job before IOC submits the job?

      Comment


      • #4
        Thanks,
        The input path is determined dynamically based on many conditions.
        How can I make sure I access the job before it is submitted to processing.

        If I subclass JobFactoryBean and implement afterPropertiesset(), how do I use my implementation with the hadoop namespace ?

        Comment


        • #5
          vito, if you want to stay entirely on the declarative side of things then using SpEL is the way to go - not only you can evaluate things directly through the expression but you can also call methods on other beans which, in effect, behave as factories.
          Subclassing JobFactoryBean means using the programmatic approach which means you can't use the namespaces as these work only against JobFactoryBean - you can however still configure the JobFactoryBean through 'classic' beans XML - after all, a namespace, is just a predefined, concise declaration (basically an XML template).
          Hope this helps,

          Comment


          • #6
            Thank you,
            I have not had a chance to use spring in a few years, Last version I used was 2.0. Seems spring has added a feature or two since
            Can you link me with examples of using SpEL with that approach , that is call methods on other beans.
            Regarding the job namespace, how can I access/inject the job created ?

            Comment


            • #7
              There are plenty of posts on SpEL: http://lmgtfy.com/?q=spring+spel
              The reference docs can be found here: http://j.mp/Hnsclq

              As for the job - if you are using the namespace or not, you are still assigning a bean to the job - so simply inject that job/bean and start using it.

              Comment

              Working...
              X