Announcement Announcement Module
Collapse
No announcement yet.
Spring Data Hadoop wordcount sampe does not compile with gradlew Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Spring Data Hadoop wordcount sampe does not compile with gradlew

    Hello,

    I installed Spring Framework and Spring Data Hadoop on a Amazon Web Services EC2 machine (Amazon Linux AMI i386). Then I tried to compile wordcount sample using gradlew but I got a build exception regarding thrift 0.2.0 and jdo-api 2.3-ec.


    [ec2-user@ip-10-234-126-25 wordcount]$ ../gradlew
    :compileJava

    FAILURE: Build failed with an exception.

    * What went wrong:
    Could not resolve all dependencies for configuration ':compile'.
    > Could not find group:org.apache.thrift, module:thrift, version:0.2.0.
    Required by:
    :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hbase:hbase:0.90.5
    > Could not find group:javax.jdo, module:jdo2-api, version:2.3-ec.
    Required by:
    :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1

    * Try:
    Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

    BUILD FAILED

    Total time: 4.778 secs
    -------------------------------

    I installed thrift 0.2.0 after downloading it from http://archive.apache.org/dist/incub....0-incubating/ ( ./bootstrap.sh && ./configure; make; sudo make install) but it did not help.

    Any suggestions how to fix it?

    Thanks a lot,
    Istvan

  • #2
    Hi, you can try to change "repositories" in build.gradle as following
    Code:
    repositories {
        // Public Spring artefacts
        mavenCentral()
        maven { url "http://repo.springsource.org/libs-release" }
        maven { url "http://repo.springsource.org/libs-milestone" }
        maven { url "http://repo.springsource.org/libs-snapshot" }
        maven { url "http://www.datanucleus.org/downloads/maven2/" }
        maven { url "http://oss.sonatype.org/content/repositories/snapshots" }
        maven { url "http://people.apache.org/~rawson/repo" }
    }

    Comment


    • #3
      Thanks a lot , this helped to get the code compiled. However, at runtime I got an RCP client-server version mismatch exception:

      2012-03-24 08:54:13,057 INFO [org.springframework.beans.factory.support.DefaultL istableBeanFactory] - <Destroying s ingletons in org.springframework.beans.factory.support.DefaultL istableBeanFactory@10e3293: defining beans [org.spri ngframework.beans.factory.config.PropertyPlacehold erConfigurer#0,hadoop-configuration,wordcount-job,clean-script,ru nner]; root of factory hierarchy>
      Exception in thread "main" org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name ' wordcount-job': Invocation of init method failed; nested exception is org.apache.hadoop.ipc.RPC$VersionMismatch: Pr otocol org.apache.hadoop.hdfs.protocol.ClientProtocol version mismatch. (client = 61, server = 63)
      at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract Auto wireCapableBeanFactory.java:1422)
      at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towi reCapableBeanFactory.java:518)
      at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wire CapableBeanFactory.java:455)
      at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 93)
      at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton Bean Registry.java:222)
      at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:290 )
      at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:192)
      at org.springframework.beans.factory.support.DefaultL istableBeanFactory.preInstantiateSingletons(Defaul tLis tableBeanFactory.java:567)
      at org.springframework.context.support.AbstractApplic ationContext.finishBeanFactoryInitialization(Abstr actA pplicationContext.java:895)
      at org.springframework.context.support.AbstractApplic ationContext.refresh(AbstractApplicationContext.ja va:4 25)
      at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationCon text .java:139)
      at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationCon text .java:93)
      at org.springframework.data.hadoop.samples.wordcount. Main.main(Main.java:41)
      Caused by: org.apache.hadoop.ipc.RPC$VersionMismatch: Protocol org.apache.hadoop.hdfs.protocol.ClientProtocol version mismatch. (client = 61, server = 63)
      at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
      at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode (DFSClient.java:119)
      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient. java:238)
      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient. java:203)


      The Spring Data Hadoop requirements states that Hadoop 0.20.2 is supported
      (http://static.springsource.org/sprin...uirements.html ) and I have Cloudera CDH3 installed (hadoop-0.20.2-cdh3u3-core.jar) but apparently the client version is lower than the server side.

      How can I configure Spring Data Hadoop to use the already installed Cloudera CDH3 jars?

      Istvan

      Comment


      • #4
        You can add CDH repository in "build.gradle":
        Code:
        repositories {
            // Public Spring artefacts
            mavenCentral()
            maven { url "http://repo.springsource.org/libs-release" }
            maven { url "http://repo.springsource.org/libs-milestone" }
            maven { url "http://repo.springsource.org/libs-snapshot" }
            maven { url "http://www.datanucleus.org/downloads/maven2/" }
            maven { url "http://oss.sonatype.org/content/repositories/snapshots" }
            maven { url "http://people.apache.org/~rawson/repo" }
            maven { url "https://repository.cloudera.com/artifactory/cloudera-repos/" }
        }
        and then change hadoopVersion in "gradle.properties" to 0.20.2-cdh3u3(your CDH version).

        Comment


        • #5
          Istvan, thanks for reporting the problem - I'll take a look at fixing the missing repositories.
          By default, SHDP downloads Apache Hadoop - if you want to use a different distro then feel free to add that instead as Jarred suggested.
          Note that we plan to introduce support for webhdfs in the next release, maybe even make that the default, as it avoid the tight coupling between the client and server/cluster.

          Comment


          • #6
            Hi Friends, I am also facing some problem with Spring Data Hadoop wordcount sampe with gradlew. I don't know gradlew so well. I am trying to compile it with gradlew but it show some error :

            -----------------------------------------------
            Code:
            $ ../gradlew
            :compileJava
            
            FAILURE: Build failed with an exception.
            
            * What went wrong:
            Could not resolve all dependencies for configuration ':compile'.
            > Could not resolve group:org.apache.hadoop, module:hadoop-examples, version:1.0.0.
              Required by:
                  :wordcount:1.0.0.M1
               > Could not GET 'http://repo.springsource.org/libs-release/org/apache/hadoop/hadoop-examples/1.0.0/hadoop-examples-1.0.0.pom'. Received status code 401 from server: Unauthorized
               > Could not GET 'http://repo.springsource.org/libs-milestone/org/apache/hadoop/hadoop-examples/1.0.0/hadoop-examples-1.0.0.pom'. Received status code 401 from server: Unauthorized
               > Could not GET 'http://repo.springsource.org/libs-snapshot/org/apache/hadoop/hadoop-examples/1.0.0/hadoop-examples-1.0.0.pom'. Received status code 401 from server: Unauthorized
            > Could not find group:org.apache.hadoop, module:hadoop-core, version:0.20.1.
              Required by:
                  :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hbase:hbase:0.90.5
                  :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1
                  :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1 > org.apache.hive:hive-common:0.8.1
                  :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1 > org.apache.hive:hive-serde:0.8.1
                  :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1 > org.apache.hive:hive-shims:0.8.1
            > Could not find group:org.apache.thrift, module:thrift, version:0.2.0.
              Required by:
                  :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hbase:hbase:0.90.5
            > Could not find group:javax.jdo, module:jdo2-api, version:2.3-ec.
              Required by:
                  :wordcount:1.0.0.M1 > org.springframework.data:spring-data-hadoop:1.0.0.M1 > org.apache.hive:hive-metastore:0.8.1
            
            * Try:
            Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
            
            BUILD FAILED
            
            Total time: 4 mins 54.752 secs
            -------------------------------------------------------------
            and the content of build.gradle is :
            Code:
            description = 'Spring Hadoop Samples - WordCount'
            
            apply plugin: 'base'
            apply plugin: 'java'
            apply plugin: 'idea'
            apply plugin: 'eclipse'
            
            repositories {
                // Public Spring artefacts
                maven { url "http://repo.springsource.org/libs-release" }
                maven { url "http://repo.springsource.org/libs-milestone" }
                maven { url "http://repo.springsource.org/libs-snapshot" }
            
                maven { url "http://www.datanucleus.org/downloads/maven2/" }
                maven { url "http://oss.sonatype.org/content/repositories/snapshots" }
                maven { url "http://people.apache.org/~rawson/repo" }
            
                maven { url "https://repository.cloudera.com/artifactory/cloudera-repos/" }
            }
            
            dependencies {
                compile "org.springframework.data:spring-data-hadoop:$version"
                compile "org.apache.hadoop:hadoop-examples:$hadoopVersion"
                // see HADOOP-7461
                runtime "org.codehaus.jackson:jackson-mapper-asl:$jacksonVersion"
            
                testCompile "junit:junit:$junitVersion"
                testCompile "org.springframework:spring-test:$springVersion"
            }
            
            task run(type: JavaExec) {
                   description = 'Runs the application'
                   main = "org.springframework.data.hadoop.samples.wordcount.Main"
                   classpath = sourceSets.main.runtimeClasspath
            }
            
            defaultTasks 'run'
            ------------------------------------------------
            http://static.springsource.org/sprin...wordcount.html
            plz help to overcome this problem ......,
            Last edited by paragflume; Jun 6th, 2012, 02:47 AM.

            Comment


            • #7
              paragflume, please create a new topic instead of hijacking an existing one.

              First try running the sample w/o the extra repo. Spring for Hadoop M2 should have better depedencies so if the sample fails, add/modify the following:
              Code:
              compile ("org.springframework.data:spring-data-hadoop:$version")
                  {  dep ->
                      exclude group: 'org.apache.hive', module: 'hive-metastore'
                      exclude group: 'org.apache.hbase', module: 'hbase'
                  }
              If you still want to use CDH, then make sure you make that your first repository, before the ones from springsource.

              Comment


              • #8
                Hi Friends, I made some changes in the build.gradle file, added the following lines --
                (1) group = "org.springframework.data.hadoop"
                (2) repositories { mavenCentral() }


                The compilation process is successful but in the run process showing some errors --
                -----------------------------------------------------------------------
                Code:
                ../gradlew
                :compileJava UP-TO-DATE
                :processResources UP-TO-DATE
                :classes UP-TO-DATE
                :run
                2012-06-06 11:11:43,693 INFO [org.springframework.context.support.ClassPathXmlApplicationContext] - <Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@1a0c10f: startup date [Wed Jun 06 11:11:43 IST 2012]; root of context hierarchy>
                2012-06-06 11:11:43,791 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - <Loading XML bean definitions from class path resource [META-INF/spring/context.xml]>
                2012-06-06 11:11:44,256 INFO [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer] - <Loading properties file from class path resource [hadoop.properties]>
                2012-06-06 11:11:44,275 INFO [org.springframework.beans.factory.support.DefaultListableBeanFactory] - <Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1d85f79: defining beans [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,hadoop-configuration,wordcount-job,clean-script,runner]; root of factory hierarchy>
                2012-06-06 11:11:45,725 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s).>
                2012-06-06 11:11:46,726 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 1 time(s).>
                2012-06-06 11:11:47,727 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 2 time(s).>
                2012-06-06 11:11:48,728 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 3 time(s).>
                2012-06-06 11:11:49,768 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 4 time(s).>
                2012-06-06 11:11:50,770 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 5 time(s).>
                2012-06-06 11:11:51,771 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 6 time(s).>
                2012-06-06 11:11:52,796 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 7 time(s).>
                2012-06-06 11:11:53,797 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 8 time(s).>
                2012-06-06 11:11:54,798 INFO [org.apache.hadoop.ipc.Client] - <Retrying connect to server: localhost/127.0.0.1:9000. Already tried 9 time(s).>
                2012-06-06 11:11:54,811 INFO [org.springframework.beans.factory.support.DefaultListableBeanFactory] - <Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1d85f79: defining beans [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,hadoop-configuration,wordcount-job,clean-script,runner]; root of factory hierarchy>
                Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'wordcount-job': Invocation of init method failed; nested exception is java.net.ConnectException: Call to localhost/127.0.0.1:9000 failed on connection exception: java.net.ConnectException: Connection refused
                        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1422)
                        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:518)
                        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:455)
                        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
                        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
                        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
                        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
                        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:567)
                        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
                        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
                        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
                        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)
                        at org.springframework.data.hadoop.samples.wordcount.Main.main(Main.java:41)
                Caused by: java.net.ConnectException: Call to localhost/127.0.0.1:9000 failed on connection exception: java.net.ConnectException: Connection refused
                        at org.apache.hadoop.ipc.Client.wrapException(Client.java:1095)
                        at org.apache.hadoop.ipc.Client.call(Client.java:1071)
                        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
                        at $Proxy1.getProtocolVersion(Unknown Source)
                        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
                        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
                        at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
                        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
                        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
                        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
                        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
                        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
                        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
                        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
                        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
                        at org.springframework.data.hadoop.mapreduce.JobFactoryBean.afterPropertiesSet(JobFactoryBean.java:177)
                        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1479)
                        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1419)
                        ... 12 more
                Caused by: java.net.ConnectException: Connection refused
                        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
                        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
                        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:656)
                        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
                        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
                        at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
                        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
                        at org.apache.hadoop.ipc.Client.call(Client.ja)a:1046
                        ... 28 more
                
                FAILURE: Build failed with an exception.
                
                * What went wrong:
                Execution failed for task ':run'.
                > Command '/usr/java/jdk1.6.0_29/bin/java' finished with (non-zero) exit value 1.
                
                * Try:
                Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
                
                BUILD FAILED
                
                Total time: 19.532 secs
                -----------------------------------------------------------------------

                Friends can you plz help me to solve this problem ....
                Thank you ....
                Last edited by paragflume; Jun 6th, 2012, 02:48 AM.

                Comment


                • #9
                  You need Hadoop running locally (see the readme.txt).

                  P.S. And please use the [code] tags since your post is almost unreadable otherwise.

                  Comment


                  • #10
                    Hi Friends,
                    I am able to compile and run " Spring Data Hadoop wordcount samples with gradlew". I made change hadoopVersion in "gradle.properties" to 0.20.2-cdh3u3 and it worked wel. This thread really helped me. Thank you Costin Leau & leejianwei.

                    Comment

                    Working...
                    X