Announcement Announcement Module
Collapse
No announcement yet.
Compression codec com.hadoop.compression.lz o.LzoCodec not found. Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Compression codec com.hadoop.compression.lz o.LzoCodec not found.

    Exception in thread "main" org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name 'news_sim_script_s
    tep': Cannot resolve reference to bean 'news_sim_script_tasklet' while setting bean property 'tasklet'; nested exception is org.spri
    ngframework.beans.factory.BeanCreationException: Error creating bean with name 'news_sim_script_tasklet': Cannot resolve reference t
    o bean 'setup-script' while setting bean property 'scriptCallback'; nested exception is org.springframework.beans.factory.BeanCreati
    onException: Error creating bean with name 'setup-script': Invocation of init method failed; nested exception is java.lang.IllegalAr
    gumentException: Compression codec com.hadoop.compression.lz
    o.LzoCodec not found.
    at org.springframework.beans.factory.support.BeanDefi nitionValueResolver.resolveReference(BeanDefinitio nValueResolver.java:3
    28)
    at org.springframework.beans.factory.support.BeanDefi nitionValueResolver.resolveValueIfNecessary(BeanDe finitionValueResolver
    .java:106)
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.applyPropertyValues(Abs tractAutowireCapableB
    eanFactory.java:1360)
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.populateBean(AbstractAu towireCapableBeanFact
    ory.java:1118)
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towireCapableBeanFact
    ory.java:517)
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wireCapableBeanFactor
    y.java:456)
    at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 94)
    at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton BeanRegistry.java:225
    )
    at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:291 )
    at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:193)
    at org.springframework.beans.factory.support.DefaultL istableBeanFactory.preInstantiateSingletons(Defaul tListableBeanFactory.
    java:591)
    at org.springframework.context.support.AbstractApplic ationContext.finishBeanFactoryInitialization(Abstr actApplicationContext
    .java:918)
    at org.springframework.context.support.AbstractApplic ationContext.refresh(AbstractApplicationContext.ja va:469)
    at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationCon text.java:139)
    at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationCon text.java:93)
    at com.sohu.wap.algorithm.launch.SimLaunch.main(SimLa unch.java:31)
    Caused by: org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name 'news_sim_script_tasklet': Cannot
    resolve reference to bean 'setup-script' while setting bean property 'scriptCallback'; nested exception is org.springframework.beans
    .factory.BeanCreationException: Error creating bean with name 'setup-script': Invocation of init method failed; nested exception is
    java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lz
    o.LzoCodec not found.
    at org.springframework.beans.factory.support.BeanDefi nitionValueResolver.resolveReference(BeanDefinitio nValueResolver.java:3
    28)
    at org.springframework.beans.factory.support.BeanDefi nitionValueResolver.resolveValueIfNecessary(BeanDe finitionValueResolver
    .java:106)
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.applyPropertyValues(Abs tractAutowireCapableB
    eanFactory.java:1360)
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.populateBean(AbstractAu towireCapableBeanFact
    ory.java:1118)
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towireCapableBeanFact
    ory.java:517)
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wireCapableBeanFactor
    y.java:456)
    at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 94)
    at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton BeanRegistry.java:225
    )
    at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:291 )
    at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:193)
    at org.springframework.beans.factory.support.BeanDefi nitionValueResolver.resolveReference(BeanDefinitio nValueResolver.java:3
    22)
    ... 15 more
    Caused by: org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name 'setup-script': Invocation of init
    method failed; nested exception is java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lz
    o.LzoCodec not found.
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract AutowireCapableBeanFa
    ctory.java:1455)
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towireCapableBeanFact
    ory.java:519)
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wireCapableBeanFactor
    y.java:456)
    at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 94)
    at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton BeanRegistry.java:225
    )
    at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:291 )
    at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:193)
    at org.springframework.beans.factory.support.BeanDefi nitionValueResolver.resolveReference(BeanDefinitio nValueResolver.java:3
    22)
    ... 25 more
    Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lz
    o.LzoCodec not found.
    at org.apache.hadoop.io.compress.CompressionCodecFact ory.getCodecClasses(CompressionCodecFactory.java:9 6)
    at org.apache.hadoop.io.compress.CompressionCodecFact ory.<init>(CompressionCodecFactory.java:134)
    at org.springframework.data.hadoop.fs.HdfsResourceLoa der.<init>(HdfsResourceLoader.java:76)
    at org.springframework.data.hadoop.fs.HdfsResourceLoa der.<init>(HdfsResourceLoader.java:98)
    at org.springframework.data.hadoop.fs.HdfsResourceLoa der.<init>(HdfsResourceLoader.java:61)
    at org.springframework.data.hadoop.scripting.HdfsScri ptRunner.detectHdfsRL(HdfsScriptRunner.java:179)
    at org.springframework.data.hadoop.scripting.HdfsScri ptRunner.postProcess(HdfsScriptRunner.java:92)
    at org.springframework.data.hadoop.scripting.Jsr223Sc riptRunner.afterPropertiesSet(Jsr223ScriptRunner.j ava:91)
    at org.springframework.data.hadoop.scripting.HdfsScri ptRunner.afterPropertiesSet(HdfsScriptRunner.java: 225)
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.invokeInitMethods(Abstr actAutowireCapableBea
    nFactory.java:1514)
    at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract AutowireCapableBeanFa
    ctory.java:1452)
    ... 32 more
    Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lz
    o.LzoCodec
    at java.net.URLClassLoader$1.run(URLClassLoader.java: 200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.j ava:188)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:3 07)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launche r.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:2 52)
    at java.lang.ClassLoader.loadClassInternal(ClassLoade r.java:320)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.conf.Configuration.getClassByNam e(Configuration.java:819)
    at org.apache.hadoop.io.compress.CompressionCodecFact ory.getCodecClasses(CompressionCodecFactory.java:8 9)
    ... 42 more

    has someone encounter the same problem?

  • #2
    You need to install the Hadoop LzoCodec in the classpath (which implies the native library as well). Some of your classes refer to it and since it's not found, you get this exception. Notice that this is caused by Hadoop (as indicated by the stacktrace) not Spring.

    P.S. please use [code] tags when posting code/stacktracer - otherwise your posts are unreadable.

    Comment


    • #3
      thank your reply
      my start shell
      Code:
      nohup hadoop-news com.sohu.wap.algorithm.launch.SimLaunch &
      hadoop-new script
      Code:
      exec " -Dproc_$COMMAND $JAVA_HEAP_MAX $HADOOP_OPTS -Djava.library.path=/usr/local/goldmine/hadoop/defaul
      t/bin/../lib/native/Linux-amd64-64 -classpath /usr/local/goldmine/hadoop/default/bin/../conf:/usr/local/java/jdk/default/lib/tools.j
      ar:/usr/local/goldmine/hadoop/default/bin/..:/usr/local/goldmine/hadoop/default/bin/../hadoop-core-0.20.2-sohu.jar:/usr/local/goldmi
      ne/hadoop/default/bin/../lib/aspectjrt-1.6.5.jar:/usr/local/goldmine/hadoop/default/bin/../lib/aspectjtools-1.6.5.jar:/usr/local/gol
      dmine/hadoop/default/bin/../lib/commons-beanutils-1.7.0.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-beanutils-core-1.8
      .0.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-cli-1.2.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-codec
      -1.4.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-collections-3.2.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/c
      ommons-configuration-1.6.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-daemon-1.0.1.jar:/usr/local/goldmine/hadoop/defau
      lt/bin/../lib/commons-digester-1.8.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-el-1.0.jar:/usr/local/goldmine/hadoop/d
      efault/bin/../lib/commons-httpclient-3.0.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-lang-2.4.jar:/usr/local/goldmin
      e/hadoop/default/bin/../lib/commons-logging-1.1.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-logging-api-1.0.4.jar:/u
      sr/local/goldmine/hadoop/default/bin/../lib/commons-math-2.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/commons-net-1.4.1.jar
      :/usr/local/goldmine/hadoop/default/bin/../lib/core-3.1.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/hadoop-fairscheduler-0.2
      0.2-sohu.jar:/usr/local/goldmine/hadoop/default/bin/../lib/hadoop-lzo-0.4.8.jar:/usr/local/goldmine/hadoop/default/bin/../lib/hsqldb
      -1.8.0.10.jar:/usr/local/goldmine/hadoop/default/bin/../lib/hue-plugins-1.2.0.jar:/usr/local/goldmine/hadoop/default/bin/../lib/jack
      son-core-asl-1.0.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/jackson-mapper-asl-1.0.1.jar:/usr/local/goldmine/hadoop/default
      /bin/../lib/jasper-compiler-5.5.12.jar:/usr/local/goldmine/hadoop/default/bin/../lib/jasper-runtime-5.5.12.jar:/usr/local/goldmine/h
      adoop/default/bin/../lib/jets3t-0.6.1.jar:/usr/local/goldmine/hadoop/default/bin/../lib/jetty-6.1.26.jar:/usr/local/goldmine/hadoop/
      default/bin/../lib/jetty-util-6.1.26.jar:/usr/local/goldmine/hadoop/default/bin/../lib/jsch-0.1.42.jar:/usr/local/goldmine/hadoop/de
      fault/bin/../lib/junit-4.5.jar:/usr/local/goldmine/hadoop/default/bin/../lib/kfs-0.2.2.jar:/usr/local/goldmine/hadoop/default/bin/..
      /lib/log4j-1.2.15.jar:/usr/local/goldmine/hadoop/default/bin/../lib/mockito-all-1.8.5.jar:/usr/local/goldmine/hadoop/default/bin/../
      lib/oro-2.0.8.jar:/usr/local/goldmine/hadoop/default/bin/../lib/servlet-api-2.5-20081211.jar:/usr/local/goldmine/hadoop/default/bin/
      ../lib/slf4j-api-1.4.3.jar:/usr/local/goldmine/hadoop/default/bin/../lib/slf4j-log4j12-1.4.3.jar:/usr/local/goldmine/hadoop/default/
      bin/../lib/xmlenc-0.52.jar:/usr/local/goldmine/hadoop/default/bin/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/goldmine/hadoop/default/bin/
      ../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/goldmine/hadoop/default/lib/hadoop-lzo-0.4.8.jar -Djava.ext.dirs=lib  -Dproduction.mode=DE
      V -Dlog.output=/data/DATA/smc/whftest/newrecom/logs $CLASS "$@"
      as above,the lib hadoop-lzo-0.4.8.jar has already installed and native libraby "/usr/local/goldmine/hadoop/defaul
      t/bin/../lib/native/Linux-amd64-64" has already defined. i wonder why it report error anyway?please

      Comment


      • #4
        That's a huge classpath -notice that you declare hadoop-lzo-0.4.8.jar multiple times - you can use * since JDK 1.6 I believe instead of specifiying each jar.
        I'm unclear whether this is the command you use to start the cluster or the spring hadoop app? I ask since the latter is usually used to submit tasks to the hadoop cluster (which is already started). If that's the case make sure the LzoCodec is available for both apps.

        As an alternative you could try using another codec just to double check the Codec initialization for Hadoop and whether they are properly loaded.

        I suggest also googling around since this is a Hadoop setup problem and I'm sure plenty of people bumped into it.

        Comment

        Working...
        X