Announcement Announcement Module
Collapse
No announcement yet.
JdbcCursorItemReader throw out of memory in postgresql Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • JdbcCursorItemReader throw out of memory in postgresql

    JdbcCursorItemReader throw out of memory in postgresql.

    i am using latest version(2.1.5.RELEASE).
    it seem that all the data is loaded in the client side.
    i am reading a lot of data (about 50 million rows), so increasing the memory will help in this case (not that is should work in this way...).

    does any body encounter this problem in this version ?
    how can i fix it ?

    thanks !

  • #2
    The memory consumption of the JdbcCursorItemReader really depends on the database driver (how it efficiently "streams" the data). You can consider increasing the memory of the batch process, but perhaps the PostgreSQL driver isn't efficient enough for such datasets, period. Did you consider using the JdbcPagingItemReader instead?

    Comment


    • #3
      using JdbcPagingItemReader

      Originally posted by arno View Post
      The memory consumption of the JdbcCursorItemReader really depends on the database driver (how it efficiently "streams" the data). You can consider increasing the memory of the batch process, but perhaps the PostgreSQL driver isn't efficient enough for such datasets, period. Did you consider using the JdbcPagingItemReader instead?
      thanks for your replay.

      well, performance will be problematic when reading large volumes of data.

      Comment


      • #4
        Originally posted by eranmiz133 View Post
        well, performance will be problematic when reading large volumes of data.
        that's where the JdbcCursorItemReader can help. Instead of using one query and one ResultSet (and being dependant on the driver's cursor implementation), Spring Batch will handle the paging for you. You can use pages of 100 items and you shouldn't face memory exhaustion, at the price of issuing multiple queries (which is a fair trade-off).

        Comment

        Working...
        X