Announcement Announcement Module
No announcement yet.
Performance slowdown gradually Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Performance slowdown gradually


    We are using spring batch to load couple thousand files. Each file contains various lines of data, ranging from hundreds to fifty thounsands. The issue I am facing now is the performance slowed down gradually. From precessing 12000 records/minute to 4000 records/minute within 10 minutes.

    Initial we thought it is database issue that stored procedure calls taking longer and longer time with more data. But if I restart the importer process without removing the data, speed pick up again, the inital speed is always at 12000 records/minute, pretty steady. Over the time, stored procedure does tend to take longer time within the same importer run.

    So I was wondering could it be related to Java side caching, connection pools etc. Can anyone shed any lights on this issue?

  • #2
    Maybe you are updating your DB with a unique transaction.

    If you can, commit every a number of record or try to use addBatch and batchUpdate on your connection.


    • #3
      I have set commit level to 10.

      I have let my batch running for 1 day. The processing speed drop from 12000 records/minute to 2 records/minute. Then I restared my importer app. The initial speed changed to 4000 records/minute. So that definitely tells me something not set/config correctly in the java side.

      My batch job is running on the same server as the database server.


      • #4
        Reading your post I'm thinking something about memory leak or connection issue (even if you are running on database server the connection is over the net).

        You can monitor process and OS resource?

        If you can, post your config and code, maybe we can help more with this.


        • #5
          A memory leak does sound likely, it could also be that you're not stateless somewhere. For example, if you've got a list as an instance variable that's growing in size. Eventually, it should cause a heap size error, but up until then, garbage collection is going to take longer and longer.


          • #6
            I run top command on the server box.

            This is an 8 CPU box. MySQL database is taking about 500% CPU. My importer process is using only 1%MEM, pretty steady. When the importer throughput is low, I do see %CPU drop to below 10%.

            Here is the copy of top output.

            top - 10:45:18 up 67 days, 19:44, 2 users, load average: 5.08, 5.06, 5.48
            Tasks: 157 total, 1 running, 156 sleeping, 0 stopped, 0 zombie
            Cpu(s): 63.2%us, 0.2%sy, 0.0%ni, 36.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
            Mem: 37048220k total, 36672296k used, 375924k free, 466868k buffers
            Swap: 2031608k total, 20320k used, 2011288k free, 19732780k cached

            11766 mysql 15 0 27.3g 12g 58m S 484.3 36.3 28463:31 mysqld
            16815 root 24 0 1379m 344m 9000 S 5.7 1.0 10:08.35 java

            See any issue here?

            I didn't set heap size in my JAVA_OPTS. Should I? And what would be appropriate value?


            • #7
              Another find is when my process just started, it did get higher CPU usage. like +100%CPU. But the CPU usage slowly decresed.

              When the CPU usage is high, my throughput is high. When it is low, the throughput is low. Now the question is why the CPU Usage drop for long running process?


              • #8
                Did you find out what was happening here? It reminds me of an issue someone had a while ago when trying (inadvertently it turned out) to run a Job in a single transaction. That would certainly slow you down progressively. Are you running in an MDB / EJB / other container that might wrap the Job.execute() or Step.execute() in a transaction?