Announcement Announcement Module
Collapse
No announcement yet.
Possible issue with closing connections in HBaseTemplate? Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Possible issue with closing connections in HBaseTemplate?

    Hey all,

    I'm hoping you can help me with something that is perplexing me. I'm a bit of an HBase newb (and certainly spring-data-hadoop newb).

    I have the following configuration set in my spring config:

    Code:
    <?xml version="1.0" encoding="UTF-8"?>
    <beans xmlns="http://www.springframework.org/schema/beans"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xmlns:hdp="http://www.springframework.org/schema/hadoop"
                 xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
                                     http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">
    
        <hdp:hbase-configuration id="hbaseConfig" />
        <bean id="hbaseTemplate" class="org.springframework.data.hadoop.hbase.HbaseTemplate">
            <property name="configuration" ref="hbaseConfig" />
        </bean>
    
    
    </beans>

    I am injecting the template into my own DAO class:

    Code:
    <?xml version="1.0" encoding="UTF-8"?>
    <beans xmlns="http://www.springframework.org/schema/beans"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xmlns:hdp="http://www.springframework.org/schema/hadoop"
           xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
                                     http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">
    
    
    
        <bean id="userStore" class="data.store.hbase.UserStoreHBaseImpl">
            <constructor-arg name="hbaseTemplate" ref="hbaseTemplate" />
        </bean>
    
    </beans>

    My user store DAO inherits from my own Abstract class, but at the moment it ONLY contains a reference to the template:

    AbstractHBaseStore:
    Code:
    public abstract class AbstractHBaseStore {
    
        private HbaseTemplate hbaseTemplate;
    
        protected AbstractHBaseStore(HbaseTemplate template) {
            this.hbaseTemplate = template;
        }
    
        public HbaseTemplate getHbaseTemplate() {
            return hbaseTemplate;
        }
    
        public void setHbaseTemplate(HbaseTemplate hbaseTemplate) {
            this.hbaseTemplate = hbaseTemplate;
        }
    }

    UserStoreHBaseImpl:
    Code:
    
    
    public class UserStoreHBaseImpl extends AbstractHBaseStore implements UserStore, RowMapper<User> {
    
        public static final String TABLE_NAME = "User";
        public static final String COL_FAMILY_NAME_PROFILE = "profile";
        public static final String COL_NAME_EMAIL_ADDRESS = "email_address";
        public static final String COL_NAME_GIVEN_NAME = "given_name";
        public static final String COL_NAME_SUR_NAME = "sur_name";
    
        public UserStoreHBaseImpl(HbaseTemplate hbaseTemplate) {
            super(hbaseTemplate);
        }
    
        @Override
        public User findUserByPrimaryKey(String key) {
    
            User user = getHbaseTemplate().get(TABLE_NAME, key, this);
            return(user);
    
        }
    
        @Override
        public void persistUser(final String key, final User user) {
    
            getHbaseTemplate().execute(TABLE_NAME, new TableCallback<Object>() {
                @Override
                public Object doInTable(HTable table) throws Throwable {
    
                    Put put = new Put(key.getBytes());
                    put.add(COL_FAMILY_NAME_PROFILE.getBytes(), COL_NAME_EMAIL_ADDRESS.getBytes(), user.getEmailAddress().getBytes());
                    put.add(COL_FAMILY_NAME_PROFILE.getBytes(), COL_NAME_GIVEN_NAME.getBytes(), user.getGivenName().getBytes());
                    put.add(COL_FAMILY_NAME_PROFILE.getBytes(), COL_NAME_SUR_NAME.getBytes(), user.getSurName().getBytes());
    
                    table.put(put);
    
                    return(null);
    
                }
            });
    
        }
    
        @Override
        public void delete(final String key) {
    
            getHbaseTemplate().execute(TABLE_NAME, new TableCallback<Object>() {
                @Override
                public Object doInTable(HTable hTable) throws Throwable {
    
                    Delete delete = new Delete(key.getBytes());
                    hTable.delete(delete);
    
                    return(null);
    
                }
            });
    
        }
    
        @Override
        public User mapRow(Result result, int i) throws Exception {
    
            String email = Bytes.toString(result.getValue(COL_FAMILY_NAME_PROFILE.getBytes(), COL_NAME_EMAIL_ADDRESS.getBytes()));
            String givenName = Bytes.toString(result.getValue(COL_FAMILY_NAME_PROFILE.getBytes(), COL_NAME_GIVEN_NAME.getBytes()));
            String surName = Bytes.toString(result.getValue(COL_FAMILY_NAME_PROFILE.getBytes(), COL_NAME_SUR_NAME.getBytes()));
    
            User user = new User();
            user.setEmailAddress(email);
            user.setGivenName(givenName);
            user.setSurName(surName);
    
            return(user);
    
        }
    }
    The problem that I am having is that the first call to my UserStoreHBaseImpl class (and therefore, the first call to hbaseTemplate) succeeds, but any other call after the first fails with the following errors:

    Code:
    12/07/11 14:38:12 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
    12/07/11 14:38:12 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x138779fec78000d, negotiated timeout = 40000
    12/07/11 14:38:12 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x138779fec78000d
    12/07/11 14:38:12 INFO zookeeper.ClientCnxn: EventThread shut down
    12/07/11 14:38:12 INFO zookeeper.ZooKeeper: Session: 0x138779fec78000d closed
    Inserted
    
    org.springframework.data.hadoop.hbase.HbaseSystemException: Failed after attempts=10, exceptions:
    Wed Jul 11 14:38:12 PDT 2012, org.apache.hadoop.hbase.client.HTable$5@43ce663c, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1996e136 closed
    Wed Jul 11 14:38:13 PDT 2012, org.apache.hadoop.hbase.client.HTable$5@43ce663c, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1996e136 closed
    ; nested exception is org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
    Wed Jul 11 14:38:12 PDT 2012, org.apache.hadoop.hbase.client.HTable$5@43ce663c, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1996e136 closed
    Wed Jul 11 14:38:13 PDT 2012, org.apache.hadoop.hbase.client.HTable$5@43ce663c, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1996e136 closed
    Wed Jul 11 14:38:14 PDT 2012, org.apache.hadoop.hbase.client.HTable$5@43ce663c, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1996e136 closed
    Wed Jul 11 14:38:15 PDT 2012, org.apache.hadoop.hbase.client.HTable$5@43ce663c, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1996e136 closed
    Wed Jul 11 14:38:17 PDT 2012, org.apache.hadoop.hbase.client.HTable$5@43ce663c, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1996e136 closed
    	at org.springframework.data.hadoop.hbase.HbaseUtils.convertHbaseException(HbaseUtils.java:42)
    	at org.springframework.data.hadoop.hbase.HbaseTemplate.convertHbaseAccessException(HbaseTemplate.java:111)
    	at org.springframework.data.hadoop.hbase.HbaseTemplate.execute(HbaseTemplate.java:82)
    	at org.springframework.data.hadoop.hbase.HbaseTemplate.get(HbaseTemplate.java:174)
    	at org.springframework.data.hadoop.hbase.HbaseTemplate.get(HbaseTemplate.java:164)
    	at data.store.hbase.UserStoreHBaseImpl.findUserByPrimaryKey(UserStoreHBaseImpl.java:30)
    	at data.store.hbase.UserStoreHBaseImplTest.testHBaseInsertQueryDelete(UserStoreHBaseImplTest.java:37)
    It almost looks like the first call closes the connection to HBase so that subsequent calls always fail - but that doesn't make sense, unless I am improperly using the HBaseTemplate. Am I?

    The JUnit test that tests this DAO:

    Code:
    
    @RunWith(SpringJUnit4ClassRunner.class)
    @ContextConfiguration(locations={"classpath:/spring-config/*.xml"})
    public class UserStoreHBaseImplTest {
    
        @Autowired
        private ApplicationContext applicationContext;
    
        @Test
        public void testHBaseInsertQueryDelete() throws Exception {
    
            UserStoreHBaseImpl userStore  = (UserStoreHBaseImpl) applicationContext.getBean("userStore");
    
    
    
            User user = new User();
            user.setEmailAddress("[email protected]");
            user.setGivenName("Tester");
            user.setSurName("Testable");
    
            userStore.persistUser("[email protected]", user);
            System.out.println("Inserted");
    
            // retrieve it!
            User gotUser = userStore.findUserByPrimaryKey("[email protected]");
            assertNotNull(gotUser);
            assertEquals("[email protected]", gotUser.getEmailAddress());
            assertEquals("Tester", gotUser.getGivenName());
            assertEquals("Testable", gotUser.getSurName());
            System.out.println("RETRIEVED");
    
    
    
            // delete it!
            userStore.delete("[email protected]");
            System.out.println("deleted!");
    
    
            // try to retrieve it again
            User delUser = userStore.findUserByPrimaryKey("[email protected]");
            assertNull(delUser);
    
    
    
        }
    
    
    
    
    }
    Any help would be greatly appreciated!

    Thanks.

  • #2
    Can you please raise a bug? It looks like the template is not properly disposing of tables.

    Comment


    • #3
      I've pushed some fixes into master along with a nightly build. Can you please try out and let us know whether it fixes your problems or not?

      Cheers,

      Comment


      • #4
        Hi Costin,

        Thanks for your reply. And your fix! Yes, this seems to fix the issue.

        Did you still want me to file a bug for this issue? I'd be happy to, if it helps.

        Thanks,
        -Michael

        Comment


        • #5
          No. That's fine - but in the future, go right ahead and create the issues.
          Anyway, give the project a spin and let us know what you think of it.

          Comment


          • #6
            Originally posted by Costin Leau View Post
            No. That's fine - but in the future, go right ahead and create the issues.
            Anyway, give the project a spin and let us know what you think of it.

            I am new to Hbase. I am facing the same issues as discussed here. I got the spring data library from the maven repository 1.0.0M2, please let me know where to get the fixed version of it.


            Thanks in Advance
            Sujata

            Comment


            • #7
              http://www.springsource.org/spring-data/hadoop#maven
              Use the latest build, meaning 1.0.0.BUILD-SNAPSHOT.

              Comment


              • #8
                Thank you so much Costin Leau.... I am able to proceed now

                Comment


                • #9
                  Use the following repo in your pom:

                  Code:
                          <repository>
                              <!-- Snapshots -->
                              <id>spring-snapshot</id>
                              <name>Spring Maven SNAPSHOT Repository</name>
                              <url>http://repo.springframework.org/snapshot</url>
                          </repository>
                  Then include the following dependency:

                  Code:
                          <dependency>
                              <groupId>org.springframework.data</groupId>
                              <artifactId>spring-data-hadoop</artifactId>
                              <version>1.0.0.BUILD-SNAPSHOT</version>
                          </dependency>
                  Hope that helps.

                  -Michael

                  Comment


                  • #10
                    Hi Michael,

                    Thanks for the reply. My project is not a maven project, i had to copy the latest jar from the SNAPSHOT location.
                    Now i am able to proceed.

                    Thanks
                    Sujata

                    Comment

                    Working...
                    X