Announcement Announcement Module
Collapse
No announcement yet.
Example of using Spring Integration & Batch to poll file system? Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Example of using Spring Integration & Batch to poll file system?

    Hi,

    I need to poll the file system for a file and launch a job when a file arrives in the directory.

    From reading around it seems like using Spring Integration & Spring Batch together are the way to do this.

    Is there an example of this anywhere I could look at? There seems to be a presentation done on this at Spring One but I can't find the actual presentation anywhere.

    I've just read all the Spring Batch docs, and don't have time to read all the Spring Integration stuff too right away, so a quick start of how to use the two together would be good.

    Thanks,
    Sarah

  • #2
    I am using Quartz scheduler to constantly poll the directory and kick off processing when file arrives. The integration is quite simple. Samples provided with batch should have an example.

    Comment


    • #3
      There has to be a better way but this is what I came up with

      I've been looking at this thread hoping someone would answer, because I've been wondering how to accomplish this very task.

      I'll caveat what I did with the following:
      New to Spring Batch
      New to Spring Integration
      And as evidenced by my Spring in Action Book, I haven't used Spring in general since before 2.0.

      Integration Part
      There's a whole bunch of examples on how to set up polling so really no need to go into it here.

      Service Activator
      Consumes files from inbound-adapter
      Code:
      JobParametersBuilder builder = new JobParametersBuilder();
      builder.addString("input.file", file.getAbsolutePath());
      
      JobExecution jobExecution = this.getJobLauncher.run(this.getJob, builder.toJobPareters());

      Batch Part

      Code:
      <batch:job id="readCSVJob">
        <batch:step id="step1">
          <batch:tasklet>
            <batch:chunk comit-interval="500" reader="CSVItemReader" writer="CSVPersistToDBWriter"/>
          </batch:tasklet>
        </batch:step>
      </batch:job>
      
      
      <bean class="org.springframework.batch.item.file.FlatFileItemReader" id="CSVItemReader" scope="step">
        <property name="resource" value="file:#{jobParameters[file.input}"/>
        ... line mapper defined here
      </bean>

      Like I said I really hope there's a better way of accomplishing this, because it seems just a bit hackish. As I come to know the project a bit better I will come up with something a bit more elegant, but for the time being it works, and it allows me to move on to other things.

      Good luck!
      j

      Comment


      • #4
        That's how I'm doing it...I have a SI file activator that looks for new files, pushes to a ServiceActivator endpoint that launches the batch using the input file. There's a little more to it in my case (I'm doing some file transformations and moving them to another directory) but the basic idea is the same.

        Comment


        • #5
          Service Activator
          Consumes files from inbound-adapter
          Code:
          JobParametersBuilder builder = new JobParametersBuilder();
          builder.addString("input.file", file.getAbsolutePath());
          
          JobExecution jobExecution = this.getJobLauncher.run(this.getJob, builder.toJobPareters());

          Batch Part

          Code:
          <batch:job id="readCSVJob">
            <batch:step id="step1">
              <batch:tasklet>
                <batch:chunk comit-interval="500" reader="CSVItemReader" writer="CSVPersistToDBWriter"/>
              </batch:tasklet>
            </batch:step>
          </batch:job>
          
          
          <bean class="org.springframework.batch.item.file.FlatFileItemReader" id="CSVItemReader" scope="step">
            <property name="resource" value="file:#{jobParameters[file.input}"/>
            ... line mapper defined here
          </bean>



          jason
          How would you initialize the job/launcher details from the bean configuration inside the service activator class
          Last edited by msns3ka; Jan 31st, 2010, 03:12 AM.

          Comment


          • #6
            Init of Job Information

            If I understand your question correctly. I really didn't do anything different than normal Spring injection. Which is to say that I just injected the Job and the JobLauncher into the ServiceActivator class defined in Application Context.

            Comment


            • #7
              Originally posted by jasonnic View Post
              If I understand your question correctly. I really didn't do anything different than normal Spring injection. Which is to say that I just injected the Job and the JobLauncher into the ServiceActivator class defined in Application Context.
              Thanks Jason, I got it...[I am a newbie to spring] ....I think with Spring you have to think in terms of DI as opposed to conventional programming methods. Spring is Awesome!

              Comment


              • #8
                Thanks for all the responses, I'm going for the Spring Integration route. With quartz I think that every time the job runs and there's no file there to process, I would get an entry in the Spring job repository database.

                Attached is my prototype spring integration configuration file, in case it's useful for anyone else.

                Comment


                • #9
                  Hello,i'm new in spring batch i want an example of spring batch + quartz

                  Comment

                  Working...
                  X