Announcement Announcement Module
Collapse
No announcement yet.
Setting Executor on hdb:job-runner Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Setting Executor on hdb:job-runner

    I am trying to invoke several Hadoop jobs serially using Spring Hadoop, but one problem I'm having is that the main thread is not waiting for the job to complete before proceeding to the next job. The XML I'm using to define each job looks like:

    <hdp:job-runner
    id="runner"
    job-ref="my-job"
    run-at-startup="true"
    post-action="cleanup-script"
    wait-for-completion="true"
    />

    although the wait-for-completion attribute doesn't seem to have the desired effect. I've tried setting the "executor" attribute, as stated in the docs:

    As the Hadoop job submittion and execution (when wait-for-completion is true) is blocking, JobRunner uses a JDK Executor to start (or stop) a job. The default implementation, SimpleAsyncTaskExecutor creates a new Thread for each new task. Before going into production, it is recommended to double-check whether this strategy is suitable or whether a throttled or pooled implementation is better. One can customize the behaviour through executor parameter.

    but I can't get the configuration right -- it's looking for an Executor instance but I can only set executor to a String.

  • #2
    Hi,

    There was a bug in the schema (executor should have been executor-ref) which has now been fixed in master. The default executor (the async) might be changed back to the sync approach as that seems to be the behaviour expected by most people...

    Comment


    • #3
      Originally posted by Costin Leau View Post
      Hi,

      There was a bug in the schema (executor should have been executor-ref) which has now been fixed in master. The default executor (the async) might be changed back to the sync approach as that seems to be the behaviour expected by most people...
      Cool, thanks for your help

      Comment


      • #4
        Hi.

        I've pushed a nightly build with the latest changes - the executor-ref should work properly now. Also the default executor is as before, synchronous, which means jobs are executed on the calling thread.
        Could you try it out and report back please?

        Thanks

        Comment

        Working...
        X