Announcement Announcement Module
Collapse
No announcement yet.
Need suggestions\pointers. Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Need suggestions\pointers.

    Hi Guys,

    I am a novice with Spring Batch.

    I have a requirement where I have to read huge files (Excel, CSV and XML), transform and then store (the usual ETL.) I have to transform and load the data in parallel to make the whole ETL faster.

    I read through the Spring Batch documentation and could not figure out where to start from. Please point me to the right direction.

    Thanks

  • #2
    Is it one huge file, or multiple files?

    Comment


    • #3
      Its one huge file. My plan is to read the whole file, divide the raw data in batches and run them in parallel.

      Comment


      • #4
        How are you going to divide it? Are you going to dump the file into a database, then split it from there?

        Comment


        • #5
          I am planning to load the whole file into memory first, and then depending on the size of records (say 200 records) create batches and process them in parallel.

          I will appreciate any suggestion.

          Comment

          Working...
          X