compiles, but fails to run on spark 1.0.0

[julianpeeters]

Hi, thanks for this example.

I copy/pasted code from this project into your simplesparkapp, and it worked for me (well, parallelize seemed to corrupt my avros but that's another story).

However this project fails to run for me on a Cloudera quickstart VM 5.0.

I run with:
source /etc/spark/conf/spark-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera

# system jars:
CLASSPATH=/etc/hadoop/conf
CLASSPATH=$CLASSPATH:$HADOOP_HOME/*:$HADOOP_HOME/lib/*
CLASSPATH=$CLASSPATH:$HADOOP_HOME/../hadoop-mapreduce/*:$HADOOP_HOME/../hadoop-mapreduce/lib/*
CLASSPATH=$CLASSPATH:$HADOOP_HOME/../hadoop-yarn/*:$HADOOP_HOME/../hadoop-yarn/lib/*
CLASSPATH=$CLASSPATH:$HADOOP_HOME/../hadoop-hdfs/*:$HADOOP_HOME/../hadoop-hdfs/lib/*
CLASSPATH=$CLASSPATH:$SPARK_HOME/assembly/lib/*

# app jar:
CLASSPATH=$CLASSPATH:target/sparkavroapp-0.0.1-SNAPSHOT-jar-with-dependencies.jar

CONFIG_OPTS="-Dspark.master=local -Dspark.jars=target/sparkavroapp-0.0.1-SNAPSHOT-jar-with-dependencies.jar"

$JAVA_HOME/bin/java -cp $CLASSPATH $CONFIG_OPTS com.cloudera.sparkavro.SparkAvroWriter 

And get the error:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.SparkContext.(Lorg/apache/spark/SparkConf;)V
at com.cloudera.sparkavro.SparkAvroWriter$.main(SparkAvroWriter.scala:39)
at com.cloudera.sparkavro.SparkAvroWriter.main(SparkAvroWriter.scala)

What am I missing to get Spark and Cloudera to play well together?

Thanks,
Julian

[sryza]

Hey Julian,

I think the issue is probably binary incompatibility between what you're compiling against and what you're running against. CDH5.0 includes Spark 0.9, and it sounds like you're compiling against Spark 1.0. Can you try replacing the Spark dependency version in the pom with 0.9.0-cdh5.0.0?

[julianpeeters]

Wonderful, that worked.

However that gets me here:

    Caused by:         org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):         Permission denied: user=cloudera, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x

Using something like sudo -u hdfs results in Error: Could not find or load main class com.cloudera.sparkavro.SparkAvroWriter. Any insight here?