Announcement Announcement Module
Collapse
No announcement yet.
Problem with Spring Batch 2.0 ... ideas? Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Problem with Spring Batch 2.0 ... ideas?

    So basically we haev a spring batch 1.0 project.

    It basically has a job and steps. Now for the steps we actually are having the reader/writers multithreaded. So you actually have 3 read/writes going at a time. Added a bit of syncrhonization as to not clober ourselves.

    In addition the same job can be run concurrently multiple times. So to accomplish that i used prototype scope for the jobs.


    ....> batch 2.0 ....>

    using the job namespace i cant set the scope so it SEEMS like its being a singleton? IS that right? I say this because i noticed the reader/writer (in the beforeStep i actually set up the data i am going to use) seem to be sharing data.

    So i said ok i'll try out this new "step" scope ... now perhaps i misunderstood its use but now it seems like everytime its accessed in ONE job its a different instance? Is that the expected behaviour?

    Just running into a bit of a roadblock and any advice would be appreciated.

    Thanks

  • #2
    Does the job work if you make it single threaded?

    The job and step that you set up in the namespace are indeed singletons. That shouldn't be a problem. You need to use step scope for components that are stateful within a step.

    One thing that occurs to me: step scope doesn't work if you create your own threads and the first access to the bean is in one of those threads. But without seeing the error or some configuration / code it would be impossible to say if that is the problem you are experiencing.

    Also there was a bug that we fixed the other day (so not in 2.0.1) where multithreaded steps would create too many instances of the step scoped beans. Can you try a snapshot and see if that helps? Again it's impossible to tell if this is the same problem.

    Comment


    • #3
      Originally posted by Dave Syer View Post
      Does the job work if you make it single threaded?

      The job and step that you set up in the namespace are indeed singletons. That shouldn't be a problem. You need to use step scope for components that are stateful within a step.

      One thing that occurs to me: step scope doesn't work if you create your own threads and the first access to the bean is in one of those threads. But without seeing the error or some configuration / code it would be impossible to say if that is the problem you are experiencing.

      Also there was a bug that we fixed the other day (so not in 2.0.1) where multithreaded steps would create too many instances of the step scoped beans. Can you try a snapshot and see if that helps? Again it's impossible to tell if this is the same problem.

      OK ... is there a problem with the snapshot repo right now?

      I am trying to download it per instructions here -

      http://static.springframework.org/sp...snapshots.html

      Downloading: http://s3.amazonaws.com/maven.spring...-core/2.0.2.CI
      SNAPSHOT/spring-batch-core-2.0.2.CI-SNAPSHOT.pom
      [INFO] Unable to find resource 'org.springframework.batch:spring-batch-coreom:2.0.2.CI-SNAPSHOT' in repository spring-s3 (
      ttp://s3.amazonaws.com/maven.springframework.org/snapshot)
      Downloading: http://repo1.maven.org/maven2/org/sp...g-batch-core-2
      0.2.CI-SNAPSHOT.pom

      Comment


      • #4
        Works for me (and I can browse the repo with S3Fox). Can you try again?

        Comment


        • #5
          Originally posted by Dave Syer View Post
          Works for me (and I can browse the repo with S3Fox). Can you try again?
          still didnt work .... i am not a maven expert but i have done it before ... figured it was simple cut and paste.

          can everyone browse with s3fox?

          Comment


          • #6
            Originally posted by jnusaira View Post
            still didnt work .... i am not a maven expert but i have done it before ... figured it was simple cut and paste.

            can everyone browse with s3fox?

            never mind not sure what i did wrong in the maven script .... used ivy and got it to download fine .... could be the firewall at work? not sure.

            Comment


            • #7
              Originally posted by Dave Syer View Post
              Also there was a bug that we fixed the other day (so not in 2.0.1) where multithreaded steps would create too many instances of the step scoped beans. Can you try a snapshot and see if that helps? Again it's impossible to tell if this is the same problem.
              Still didn't seem to work with the snapshot from friday.

              I basically have a step that is multithreaded, then i use the beforeStep to setup a global List object.

              I did an output of "this" and i get different Ids back. It does going into the beforestep only once though. So thats good. But hten the other items just dont work.

              Comment


              • #8
                I still don't really understand what you are doing. Can you provide some code / config samples?

                Comment


                • #9
                  Originally posted by Dave Syer View Post
                  I still don't really understand what you are doing. Can you provide some code / config samples?
                  Sure ... thanks for the help btw ...

                  So the job is defined as this ->
                  <job id="compoundOnlyJob">
                  <step id="processSDFCompounds" parent="parseSdfsForCompounds"/>
                  </job>

                  Step is ->
                  <bean id="parseSdfsForCompounds" parent="multiThreadedStep" scope="prototype">
                  <property name="itemReader" ref="loaderCompoundItemReader"/>
                  <property name="itemWriter" ref="sdfCompoundWriter"/>
                  </bean>

                  And the parent ->

                  <bean id="multiThreadedStep" parent="simpleStep" scope="prototype">
                  <property name="taskExecutor">
                  <bean class="org.springframework.core.task.SimpleAsyncTa skExecutor">
                  <property name="concurrencyLimit" value="${loader.threadAmount}"/>
                  </bean>
                  </property>
                  </bean>


                  (please not i also tried it without the scope on the two steps as well)

                  public void beforeStep(StepExecution stepExecution) {
                  -- this is where i set a list set on the object.

                  Anything else ya need?

                  Comment


                  • #10
                    Originally posted by jnusaira View Post
                    public void beforeStep(StepExecution stepExecution) {
                    -- this is where i set a list set on the object.
                    The scope="prototype" is certainly not helping, but it might not be hurting. I would get rid of it asap if I were you. The reader and writer (assuming they are the stateful components) are the ones that you need to thing about the scope for, and scope="step" would be best, but "singleton" should be OK in this case of a proof of concept, since there's only one step and one job. My advice: get it working in an integration test first. Did you do that?

                    What list (and why)? What's a "list set"? What object? I assume either the item reader or writer is the listener?

                    (Please use [code][/code] tags to post code and stack traces.)

                    Comment


                    • #11
                      Originally posted by Dave Syer View Post
                      The scope="prototype" is certainly not helping, but it might not be hurting. I would get rid of it asap if I were you. The reader and writer (assuming they are the stateful components) are the ones that you need to thing about the scope for, and scope="step" would be best, but "singleton" should be OK in this case of a proof of concept, since there's only one step and one job. My advice: get it working in an integration test first. Did you do that?

                      What list (and why)? What's a "list set"? What object? I assume either the item reader or writer is the listener?

                      (Please use [code][/code] tags to post code and stack traces.)
                      yeah i've gotten rid of the prototype as i said didnt hurt or help. And yeah i am using scope="step" on the reader and writer.

                      No i havent gotten working with an integration test, mainly because it is integrated into Seam so its hard to run the outof the box tests. But i think i will go that way as well.

                      By list i mean like this for the reader:

                      Code:
                      public class MolecularItemReader extends AbstractFileLoader implements ItemReader<String> {
                      
                          private static final Log log = LogFactory.getLog(MolecularItemReader.class);
                      
                          protected List<String> moleculeList = null;
                      
                          public String read() {
                              // pop from the list here but the list is null
                              return moleculeList.remove()
                          }
                          }
                      
                          @Override
                          public void beforeStep(StepExecution stepExecution) {
                              super.beforeStep(stepExecution);
                              // set up the list here.
                              moleculeList = new ArrayList();
                          }
                      That is the general jist i took a lot out because its actually a bit complicated since i do forward reading.

                      Comment


                      • #12
                        So AbstractFileLoader implements StepExecutionListener? And this reader is injected directly into a step? If the answer is "no" to either of those questions, that's probably what you need to fix.

                        Comment


                        • #13
                          Originally posted by Dave Syer View Post
                          So AbstractFileLoader implements StepExecutionListener? And this reader is injected directly into a step? If the answer is "no" to either of those questions, that's probably what you need to fix.
                          yes to the first part.

                          No to the second part. The reader isnt injected. Inside the beforeStep method is where we are actually setting the object neeeded to be looked at.

                          Basically i am injecting (well looking up cause i am using the old code) an id. And the beforeStep goes out based on id to our directory to look up all the files in there.

                          Would that not be correct?

                          Code:
                          <bean id="flatFileItemReader" scope="step"
                                  class="org.springframework.batch.item.file.FlatFileItemReader">
                              <property name="resource" value="#{jobParameters[input.file.name]}" />
                            </bean>
                          in that example code it looks like you are just injecting the name and then actually initializing the reader inside.

                          Comment


                          • #14
                            Dave

                            FWIW not sure if i had to do a clean or what have you. But i have "step" in there now and that part "works". However, if i run 2 of the same jobs at the same time it still seems to use the wrong object. Is this what you meant by injecting the reader?

                            Comment


                            • #15
                              Originally posted by jnusaira View Post
                              No to the second part. The reader isnt injected.
                              Then it needs to be registered as a listener - the Step factry beans only detect listeners that are directly injected (as reader, writer, processor or tasklet). If you put a log statement or a breakpoint in your beforeStep you should be able to confirm that it is actually being called.

                              Comment

                              Working...
                              X