Announcement Announcement Module
Collapse
No announcement yet.
Pre-processing Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Pre-processing

    Hi,

    I am new to Spring Batch and am trying to find the right components to implement the requirements below.

    These are the things i am looking forward to implement:

    1. FTP a file from remote location to the server, where Spring Batch is going to run.
    2. Check the length of this file and update the length in the database table (T1).


    My questions are:
    Can i use step level listener for actions 1 and 2 above? If not listeneres then where can we handle this?
    Can we use listeners to retrieve or insert data in the database?

    Thank you,
    SpringForever

  • #2
    You may want to look at Spring Integration. Spring Batch made a decision to stay out of file moving very early on. However, Spring Integration has many of these types of tools already.

    Comment


    • #3
      To address the issue of listeners, I would advise against adding any "business" logic to listeners - if moving a file or sending something via FTP is one of the goals of your job, it should be in the main body of a tasklet or item-oriented artifact (e.g. ItemReader, ItemWriter). The listeners should, in my opinion, be used for resource lifecycle maintenance (i.e. initializing restart, opening and closing streams and sockets, creating or deleting temporary files, etc.).

      By enforcing such a contract, you allow for such conveniences as adding AOP calls around business logic, managing restartability and more easily tracking down the source of errors (since you'll know where to look instead of realizing that your configuration registers a listener that you didn't realize was being invoked).

      Comment


      • #4
        Thank you dkaminsky and lucas for your reply....

        Where and how do i check the length of a file? We also have a requirement to update this length in the database.
        Last edited by springforever; Mar 20th, 2008, 11:22 AM. Reason: additional question

        Comment


        • #5
          If you have a java.io.File object, which you can easily create with just the path, all you have to do is call length(), which will return the length in bytes as a long.

          Comment


          • #6
            Can i get the number of records in a file, using Spring Batch API?

            Comment


            • #7
              There's no "out-of-the-box" way to do this.

              There are several ways to handle this:

              a) Add the logic to and register a job listener whose beforeJob method performs this calculation for the current run - since persisting to the database is done at the execution context level and it doesn't exist yet at this point, this would only be a viable option if your job is non-restartable

              b) Add a TaskletStep whose tasklet registers itself as an ItemStream and does the following:
              1. stores the execution context locally in its open method
              2. performs this calculation for the current run in its execute method
              3. check if this calculation was passed in as part of the execution context
              4. if so, compare the two and error out if they don't match
              5. store this calculation to the execution context in its close method

              c) Calculate this externally using system tools (e.g. "wc -l" for unix would give you the number of lines in the file) and pass it in as a JobParameter - this would ensure that it would create a completely new JobInstance if the value had changed, thus avoiding restart issues (this may, though, introduce a data duplication issue depending on what's in the file)

              d) Create an instance of InitializingBean elsewhere in your ApplicationContext whose afterPropertiesSet method calculates this - inject this same bean into your tasklet or item reader and have it do the comparison between this and prior runs as necessary

              I would personally use (b)...

              Comment


              • #8
                Originally posted by dkaminsky View Post
                There's no "out-of-the-box" way to do this.

                There are several ways to handle this:

                b) Add a TaskletStep whose tasklet registers itself as an ItemStream and does the following:
                Questions:
                1) How to register Tasklet as an ItemStream, using TaskletStep?
                2) Do i need to read a file using Java IO (in the Tasklet.execute()) and count the number of lines?
                Last edited by springforever; Apr 15th, 2008, 04:38 PM.

                Comment


                • #9
                  You can't register it as an ItemStream, use a StepExecutionListener instead. You can then call getExecutionContext.

                  Comment


                  • #10
                    Originally posted by springforever View Post
                    Can i get the number of records in a file, using Spring Batch API?
                    You can get the itemCount from the StepExecution (e.g. in a listener) after the step has completed. It will only be accurate if there are no skipped records on read with the current meta-data, but that might be what you want anyway (how many records were processed)?

                    Comment


                    • #11
                      Originally posted by Dave Syer View Post
                      You can get the itemCount from the StepExecution (e.g. in a listener) after the step has completed. It will only be accurate if there are no skipped records on read with the current meta-data, but that might be what you want anyway (how many records were processed)?
                      Actually, i want to get the total number of records in the input file. Listener apporach would work, i think.

                      I have 4 records in the input file, with no skippable items. However, i get output of 5 on using "stepExecution.getItemCount()". Why is that?

                      Also, doc for getItemCount() is "Returns the current number of tasks for this execution". What do we mean by tasks here? Is it the total number of processed items or total number records going to the database?

                      Comment


                      • #12
                        The off-by-one item count in step execution should be fixed in trunk. Item count is the number of items processed - I've tidied up the StepExecution javadoc a bit.
                        Last edited by robert.kasanicky; Apr 16th, 2008, 10:29 AM. Reason: spelling

                        Comment

                        Working...
                        X