when use spark-submit command in cloudera yarn environment, got kind of exception:
java.lang.noclassdeffounderror: kafka/common/topicandpartition @ java.lang.class.getdeclaredmethods0(native method) @ java.lang.class.privategetdeclaredmethods(class.java:2701) @ java.lang.class.getdeclaredmethods(class.java:1975) @ com.fasterxml.jackson.module.scala.introspect.beanintrospector$.com$fasterxml$jackson$module$scala$introspect$beanintrospector$$listmethods$1(beanintrospector.scala:93) @ com.fasterxml.jackson.module.scala.introspect.beanintrospector$.findmethod$1(beanintrospector.scala:99) @ com.fasterxml.jackson.module.scala.introspect.beanintrospector$.com$fasterxml$jackson$module$scala$introspect$beanintrospector$$findgetter$1(beanintrospector.scala:124) @ com.fasterxml.jackson.module.scala.introspect.beanintrospector$$anonfun$3$$anonfun$apply$5.apply(beanintrospector.scala:177) @ com.fasterxml.jackson.module.scala.introspect.beanintrospector$$anonfun$3$$anonfun$apply$5.apply(beanintrospector.scala:173) @ scala.collection.traversablelike$withfilter$$anonfun$map$2.apply(traversablelike.scala:722) @ scala.collection.indexedseqoptimized$class.foreach(indexedseqoptimized.scala:33) @ scala.collection.mutable.arrayops$ofref.foreach(arrayops.scala:108) @ scala.collection.traversablelike$withfilter.map(traversablelike.scala:721) @ com.fasterxml.jackson.module.scala.introspect.beanintrospector$$anonfun$3.apply(beanintrospector.scala:173) @ com.fasterxml.jackson.module.scala.introspect.beanintrospector$$anonfun$3.apply(beanintrospector.scala:172) @ scala.collection.traversablelike$$anonfun$flatmap$1.apply(traversablelike.scala:251) @ scala.collection.traversablelike$$anonfun$flatmap$1.apply(traversablelike.scala:251) @ scala.collection.immutable.list.foreach(list.scala:318) ... the spark-submit command like:
spark-submit --master yarn-cluster \ --num-executors $2 \ --executor-cores $3 \ --class "app" \ --deploy-mode cluster \ --properties-file $1 \ --files $hdfs_path/log4j.properties,$hdfs_path/metrics.properties \ --conf spark.metrics.conf=metrics.properties \ app.jar note that, topicandpartition.class in shaded app.jar.
please try adding kafka jar using --jars option shown in example below:
spark-submit --master yarn-cluster \ --num-executors $2 \ --executor-cores $3 \ --class "app" \ --deploy-mode cluster \ --properties-file $1 \ --jars /path/to/kafka.jar --files $hdfs_path/log4j.properties,$hdfs_path/metrics.properties \ --conf spark.metrics.conf=metrics.properties \ app.jar 
Comments
Post a Comment