Announcement Announcement Module
Collapse
No announcement yet.
Configuring a job for online system Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Configuring a job for online system

    I Need help figuring out how to setup a batch to use existing online objects, specifically my service layer and corresponding dao's. I create a job that ran completely external to the online application as a demo However I duplicated a lot of objects and logic. Now I would like to use my service layer, in my ItemWriter, so I can reuse the online logic instead of duplicating. The online system uses Spring and Hibernate and the DAOs and service objects use spring DI to get refernce to the sessionFactory, datasource configured in the application context. I would like to run everything from the online system (manual jobs and scheduled jobs). Right now I would like the controller to be able to launch a job ( synchronously or asynchronously). What I tried was to used imports in the psring batch config for the application context file and I tried ot kick it off like I previously did with the CommandLineJobRunner and it went into an infinit loop. I am not sure how this needs to be setup and launched, can someone give me a rundown of how to kick off the job from a servlet and how to configure the batch files so the exisitng DAOs get access to the sessionFactory and datasource? I don't care about running synchronously or asynchronously at this point.

    Thank you,
    Todd

  • #2
    Okay, I got the job to run and figured out that I had a cyclic reference in the config (xml) files imports. I added a new configuration for the sessionFactory in the launcher xml and referenced the service object and DAOs by putting their config files on the classpath and imported them in my launcher xml. Then I kicked off the job with the command line runner...

    CommandLineJobRunner.main(new String[] {"InventoryJob.xml", "inventoryDailyJob", "schedule.date(string)=" + df.format(new Date())});

    The above turned out to have a few issue, the appserver was shut down after each run and when running on tomcat I kept getting a random null pointer exception that looked like it was loosing connections?? I ended up kicking off the job with the job launcher I had configured in my launch xml like so..

    JobParameters jobParameters = new JobParametersBuilder().addDate ("schedule.date", new Date());
    JobExecution run = jobLauncher.run(job, jobParameters);

    I also added the SimpleAsyncTaskExecutor to the job launcher and now the servlet is not waiting on the job. I think I got this resolved.

    Comment


    • #3
      Originally posted by tbone21w View Post
      Okay, I got the job to run and figured out that I had a cyclic reference in the config (xml) files imports. I added a new configuration for the sessionFactory in the launcher xml and referenced the service object and DAOs by putting their config files on the classpath and imported them in my launcher xml. Then I kicked off the job with the command line runner...

      CommandLineJobRunner.main(new String[] {"InventoryJob.xml", "inventoryDailyJob", "schedule.date(string)=" + df.format(new Date())});

      The above turned out to have a few issue, the appserver was shut down after each run and when running on tomcat I kept getting a random null pointer exception that looked like it was loosing connections?? I ended up kicking off the job with the job launcher I had configured in my launch xml like so..

      JobParameters jobParameters = new JobParametersBuilder().addDate ("schedule.date", new Date());
      JobExecution run = jobLauncher.run(job, jobParameters);

      I also added the SimpleAsyncTaskExecutor to the job launcher and now the servlet is not waiting on the job. I think I got this resolved.
      You don't want to use a commandline runner to launch a batch job from within your web application.

      I'm using the following:

      Code:
      <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
      		<property name="jobRepository" ref="jobRepository" />
      		<property name="taskExecutor">
      		  <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor" />
      		</property>
      	</bean>
      
      	<bean id="jobRepository" class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
      	  <property name="databaseType" value="sqlserver"/>
      	  <property name="dataSource" ref="dataSource"/>
      	  <property name="transactionManager" ref="transactionManager"/>
      	</bean>
      Then I launch the job from within a helper class called by my controller:

      Code:
                  Job batchJob = jobFactory.createJob();
                  // You would want to add any parameters for your job here
                  JobParameters parameters = new JobParameters(map, new HashMap<String,Long>(), new HashMap<String,Double>(), new HashMap<String,Date>());
                 
                  // Now, kick off the job
                  jobLauncher.run(batchJob, parameters);
      Note that many of the beans that you will interact with in a batch job are stateful so you cannot define them as spring singleton beans. What I did was wire up all of my job beans and their associated readers/writers in a separate spring bean config that is not loaded into my app context at startup. Then, I use this implementation for my job factory so that everytime that I create a job to run, it is created in it's own sub context using the parent app context of the web app and the prototype beans in the separate bean config:

      Code:
      public class ContextAwareJobFactory implements JobFactory, ApplicationContextAware, InitializingBean
      {
          private ClassPathXmlApplicationContextJobFactory delegate;
          
          /* The parent application context */
          private ApplicationContext applicationContext;
          /* The job bean name */
          private String beanName;
          /* resource path to subcontext spring config */
          private String subcontextPath;
      
          public void setBeanName(String beanName)
          {
              this.beanName = beanName;
          }
      
          public void setSubcontextPath(String subcontextPath)
          {
              this.subcontextPath = subcontextPath;
          }
      
          /* (non-Javadoc)
           * @see org.springframework.batch.core.configuration.JobFactory#createJob()
           */
          @Override
          public Job createJob()
          {
              return delegate.createJob();
          }
      
          /* (non-Javadoc)
           * @see org.springframework.batch.core.configuration.JobFactory#getJobName()
           */
          @Override
          public String getJobName()
          {
              return delegate.getJobName();
          }
      
          /* (non-Javadoc)
           * @see org.springframework.context.ApplicationContextAware#setApplicationContext(org.springframework.context.ApplicationContext)
           */
          @Override
          public void setApplicationContext(ApplicationContext context) throws BeansException
          {
              this.applicationContext = context;
          }
      
          /* (non-Javadoc)
           * @see org.springframework.beans.factory.InitializingBean#afterPropertiesSet()
           */
          @Override
          public void afterPropertiesSet() throws Exception
          {
              delegate = new ClassPathXmlApplicationContextJobFactory(this.beanName, this.subcontextPath, this.applicationContext);
          }
      }
      The job factory is declared as such:

      Code:
      <bean id="jobFactory" class="com.foobar.ContextAwareJobFactory">
            <property name="beanName" value="provisioningBatchUploadJob"/>
            <property name="subcontextPath" value="classpath:spring/fubar-project-batch-processing-prototype-beans.xml"/>
          </bean>
      The code in my helper class above is invoked directly from within my spring mvc controller. In my case, I'm taking an uploaded file from the user and using that as the source of the batch input. After futzing around with it for a few days, I finally got all the issues sorted out and now it is working like a champ. I've even verified that the restart behavior works correctly, allowing an admin to restart a failed batch by fixing the input file and restarting from the web interface.

      Comment


      • #4
        chudak thank you for the configuration, this is exactly what I have been trying to do. I moved all my job beans to a new config file and declared them as prototype. I also created the ContextAwareJobFactory and wired it up in the application context. In my helper class that kicks of the job I have access to the JobFactory since it is part of the application context (and I have it configured/wired in the application context). However, I am having trouble getting reference to my job launcher that is configured in the job bean config file. How did you wire up the helper class to the launcher?

        Thank you for the help,
        Todd
        Last edited by tbone21w; Sep 28th, 2008, 10:32 AM.

        Comment


        • #5
          Originally posted by tbone21w View Post
          chudak thank you for the configuration, this is exactly what I have been trying to do. I moved all my job beans to a new config file and declared them as prototype. I also created the ContextAwareJobFactory and wired it up in the application context. In my helper class that kicks of the job I have access to the JobFactory since it is part of the application context (and I have it configured/wired in the application context). However, I am having trouble getting reference to my job launcher that is configured in the job bean config file. How did you wire up the helper class to the launcher?

          Thank you for the help,
          Todd
          You don't need to declare the beans as prototypes if you move them to a separate bean file and use the job factory as I've indicated (in fact it can cause problems). The 'prototype'ness is provided by the job factory.

          The job launcher is not stateful so it should be in your parent app context. Only stateful beans (or beans that are injected with stateful beans) should be in the sub context bean file.

          Comment


          • #6
            I moved the JobLauncher and JobRepository to the application context and made sure the sub context beans were not declared as prototype and everything seams to be working. I was using an earlier version of Spring Batch and moved to 1.1.2 version. In doing so I had to change import references on a coulple of classes and I had to rebuild the batch tables based on the new schema. Thanks for all the help.

            Comment


            • #7
              Originally posted by chudak View Post

              Code:
              <bean id="jobFactory" class="com.foobar.ContextAwareJobFactory">
                    <property name="beanName" value="provisioningBatchUploadJob"/>
                    <property name="subcontextPath" value="classpath:spring/fubar-project-batch-processing-prototype-beans.xml"/>
                  </bean>
              .
              Chudak, Thanks for posting this example. Even i'm thinking of using this code in my application. But i've several jobs in my web applications and it gets invoked from the user through User Interface. So, please suggest how i can avoid hardcoding JOBNAME in the above configuration

              Comment


              • #8
                Originally posted by Jonathan_r View Post
                Chudak, Thanks for posting this example. Even i'm thinking of using this code in my application. But i've several jobs in my web applications and it gets invoked from the user through User Interface. So, please suggest how i can avoid hardcoding JOBNAME in the above configuration
                In the new spring batch, there is a class ClassPathXmlJobRegistry that ostensibly does what the above does but allows you to register multiple jobs. You still have to name them (the name refers to a configured job spring bean).

                Comment


                • #9
                  Chudak,

                  Thanks for this excellent posting.

                  Can you please explain in your framework how you are capturing the errors which is happening during file processing and providing that errors back to the user in the UI.

                  Your help on this is much appreciated.

                  Thanks,
                  vbforums

                  Comment


                  • #10
                    Originally posted by vbforums View Post
                    Chudak,

                    Thanks for this excellent posting.

                    Can you please explain in your framework how you are capturing the errors which is happening during file processing and providing that errors back to the user in the UI.

                    Your help on this is much appreciated.

                    Thanks,
                    vbforums
                    I'm not capturing errors and reporting them to the user in the UI.

                    (My) Batch processing is asynchronous and, in the case of tens of thousands of records, can take upwards of a half hour to process. I only return an error to the browser user if there is a synchronous error/failure in STARTING the job processing.

                    What I AM doing is configuring a job execution listener that sends out an email in case of failure to a mailing list so that an administrator can take the appropriate action to fix and restart the batch job.

                    Code:
                    public class ProvisioningBatchJobErrorExecutionListener extends JobExecutionListenerSupport
                    {
                        private MailSender mailSender;
                        private String[] recipients;
                        private String sender;
                        private String subject;
                        private String message;
                        
                        /**
                         * Set the mail sender.
                         * 
                         * @param mailSender the sender
                         */
                        public void setMailSender(MailSender mailSender)
                        {
                            this.mailSender = mailSender;
                        }
                    
                        /**
                         * Set the recipients for the message.
                         * 
                         * @param recipients the recipients
                         */
                        public void setRecipients(String[] recipients)
                        {
                            this.recipients = recipients;
                        }
                    
                        /**
                         * Set the sender for the email.
                         * 
                         * @param sender the sender
                         */
                        public void setSender(String sender)
                        {
                            this.sender = sender;
                        }
                    
                        /**
                         * Set the subject to be sent for the email.
                         * 
                         * @param subject the subject string
                         */
                        public void setSubject(String subject)
                        {
                            this.subject = subject;
                        }
                    
                        /**
                         * Set the message format style message to be sent
                         * in the email body. The batch guid, batch filename and
                         * stacktrace (as a string) will be used as arguments, in that order.
                         * 
                         * @param message the message format style string
                         */
                        public void setMessage(String message)
                        {
                            this.message = message;
                        }
                    
                        @Override
                        public void onError(JobExecution jobexecution, Throwable throwable)
                        {
                            String batchGuid = jobexecution.getJobInstance().getJobParameters().getString(Constants.GUID);
                            String batchFile = jobexecution.getJobInstance().getJobParameters().getString(Constants.UPLOAD_FILENAME);
                            
                            String formattedMessage = MessageFormat.format(this.message, batchGuid, batchFile, ExceptionUtils.getStackTrace(throwable));
                            
                            SimpleMailMessage mailMessage = new SimpleMailMessage();
                            mailMessage.setFrom(this.sender);
                            mailMessage.setTo(this.recipients);
                            mailMessage.setSubject(this.subject);
                            mailMessage.setText(formattedMessage);
                            
                            mailSender.send(mailMessage);
                        }
                    }
                    Configured as a bean:

                    Code:
                    <bean id="provisioningBatchErrorListener" 
                              class="com.(elided)provisioning.batch.ProvisioningBatchJobErrorExecutionListener">
                          <property name="mailSender" ref="mailSender"/>
                          <property name="sender" ref="mailReplyTo"/>
                          <property name="recipients" ref="mailRecipients"/>
                          <property name="subject" value="${provisioning.inbound.edf.notification.host}: Provisioning Batch Upload Processing Error"/>
                          <property name="message">
                              <value>
                    Batch Guid: {0}
                    Batch File: {1}
                    Host: ${provisioning.inbound.edf.notification.host}
                    
                    The processing of this file has failed. Please fix the error and restart the batch using the above guid.
                    
                    Details:
                    
                    {2}
                              </value>
                          </property>
                        </bean>
                    Then this is registered with the job:
                    Code:
                        <!-- Prototype job bean -->
                        <bean id="simpleJob" class="org.springframework.batch.core.job.SimpleJob" abstract="true">
                    		<property name="jobRepository" ref="jobRepository" />
                    		<property name="restartable" value="true" />
                    		<property name="jobExecutionListeners" ref="provisioningBatchErrorListener"/>
                    	</bean>

                    Comment


                    • #11
                      Originally posted by chudak View Post

                      Code:
                      <bean id="jobFactory" class="com.foobar.ContextAwareJobFactory">
                            <property name="beanName" value="provisioningBatchUploadJob"/>
                            <property name="subcontextPath" value="classpath:spring/fubar-project-batch-processing-prototype-beans.xml"/>
                          </bean>
                      .

                      Thanks for your response chudak. Can you please let me know if you have the definitions of spring batch beans like jobLauncher, jobRepository etc. in your fubar-project-batch-processing-prototype-beans.xml? or loading in parent context?

                      Comment


                      • #12
                        Originally posted by Jonathan_r View Post
                        Thanks for your response chudak. Can you please let me know if you have the definitions of spring batch beans like jobLauncher, jobRepository etc. in your fubar-project-batch-processing-prototype-beans.xml? or loading in parent context?
                        Anything that is stateless or thread safe goes in the parent context. Anything that is stateful or wired up to something that is stateful goes in the prototype context.

                        For example, the following beans types are in my parent context:
                        • Tasklet
                        • JobExecutionListener
                        • JobFactory
                        • FieldSetMapper
                        • Tokenizer
                        • Transformer
                        • JobLauncher

                        These bean types are in my prototype context:
                        • Job
                        • Step
                        • ItemWriter
                        • ItemReader

                        Comment


                        • #13
                          Chudak, Thanks for the reply.

                          You had mentioned that we need not explicitly specify "prototype" scope in the bean definition as we are reloading the context. Just wanted to know will there be any problem due to this as it will keep loading these objects again and again for every job. Will they get garbage collected once the job is run?

                          Comment


                          • #14
                            Originally posted by Jonathan_r View Post
                            Will they get garbage collected once the job is run?
                            Yes, because the job factory destroys the child app context when the job is finished.

                            Comment


                            • #15
                              That's excellent..

                              Comment

                              Working...
                              X