Announcement Announcement Module
Collapse
No announcement yet.
Sharing connection between several JdbcCursorItemReader Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Sharing connection between several JdbcCursorItemReader

    Hello,

    I want to use several JdbcCursorItemReaders in one composite step. I initialize all of them and enable `useSharedExtendedConnection` feature.

    Step definition looks like

    Code:
    <b:step id="test-step">
        <b:tasklet transaction-manager="transactionManager">
            <b:chunk reader="itemReader" writer="itemWriter" commit-interval="3">
                <b:streams>
                    <b:stream ref="jdbcCursorItemReader1"/>
                    <b:stream ref="jdbcCursorItemReader2"/>
                    <b:stream ref="jdbcCursorItemReader3"/>
                    <b:stream ref="jdbcCursorItemReader4"/>
                </b:streams>
            </b:chunk>
        </b:tasklet>
    </b:step>
    I expect that because of extendedShared connection feature they will reuse same connection in one step, but it was not true. After some debug I've found that TransactionSyncronizationManager.initSynchronizati on should be called before getConnection operations. It is called implicitly from TaskletStep in doExecute method from lines

    Code:
    result = (RepeatStatus) new TransactionTemplate(transactionManager, transactionAttribute)
    				.execute(new ChunkTransactionCallback(chunkContext, semaphore));
    but doExecute is called after open (where streams get connection).

    Do I miss something in spring-batch or it is bug?
    Last edited by rlz; May 22nd, 2013, 03:28 PM.

  • #2
    1. Please use the code tags when posting code.
    2. Are you using the ExtendedConnectionDataSourceProxy to wrap your DataSource?
    3. I will say that this feature wasn't designed for the connection sharing you are attempting.

    I do have to ask why the need to share a single connection across all of those open cursors...

    Comment


    • #3
      Originally posted by mminella View Post
      1. Please use the code tags when posting code.
      Fixed

      Originally posted by mminella View Post
      2. Are you using the ExtendedConnectionDataSourceProxy to wrap your DataSource?
      Yes. If I do not, than I get exception about it.

      Originally posted by mminella View Post
      3. I will say that this feature wasn't designed for the connection sharing you are attempting.
      This is sad.

      Originally posted by mminella View Post
      I do have to ask why the need to share a single connection across all of those open cursors...
      Because I use pooled DataSource. Size of my pool is 8. I want to process data in 8 threads. One step is executed in one thread. Now one step use 5 readers inside, so I can have only one processing thread. If I try to use 2-3 threads I usually get deadlock on getting connection (every thread get 1-4 connections and waits for all 5 connections).

      Comment


      • #4
        My other question is whether they all need to be JdbcCursorItemReaders (why not paging)?

        Comment


        • #5
          Originally posted by mminella View Post
          My other question is whether they all need to be JdbcCursorItemReaders (why not paging)?
          Our data process pattern:

          We partition data into small portion (~5000 items), using custom partitioner.
          Then, for every partition we load our items and add additional data based on values from same SQL database. Because it is not simple calculations we can't just use simple joins. cursorItemReaders are used by composite reader to get this additional data for calculation.
          Then we store data into NoSQL database.

          Comment

          Working...
          X