Error seen in IntelliJ
@Override is not allowed when implementing interface
Diamond types are not supported at this language level
File --> Project Structure --> Modules --> Sources
Change the Language level
to 8 - Lambdas
Incorrect JDK version for the project
Add this in the pom.xml
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
Spark submit giving Class not found error
This will create a jar file of the project.
cd proj_dir
mvn package
Spark submit giving Class not found error Create one jar with all dependencies
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass></mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
How to submit spark job
/bin/spark-submit --class org.cse.JavaKafkaWordCount --master local[2] /Users/drehman/IdeaProjects/FbPageStream/target/fbpage-stream-1.0-SNAPSHOT-jar-with-dependencies.jar
call(String) and call(T) clash
Change
public Iterator<String> call(String x) {
return Arrays.asList(SPACE.split(x)).iterator();
}
to
public Iterable<String> call(String x) {
return Arrays.asList(SPACE.split(x));
}
Turn off INFO level logging
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
Logger.getLogger("org").setLevel(Level.OFF);
Logger.getLogger("akka").setLevel(Level.OFF);
Java Spark Cassandra connector
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
<version>1.5.1</version>
</dependency>
Spark connector version difference between what is mentioned in the pom.xml and what is in the spark.conf
6/07/23 02:02:46 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 10.0.0.235): java.io.InvalidClassException: com.datastax.spark.connector.rdd.partitioner.CassandraPartition; local class incompatible: stream classdesc serialVersionUID = 147531139326522345, local class serialVersionUID = 7247106480529291035