Skip to content

Instantly share code, notes, and snippets.

@isaacsanders
Created January 21, 2016 16:07
Show Gist options
  • Save isaacsanders/660f480810fbc07d4df2 to your computer and use it in GitHub Desktop.
Save isaacsanders/660f480810fbc07d4df2 to your computer and use it in GitHub Desktop.
Spark Script
#!/bin/sh
spark-submit \
--class org.alitouka.spark.dbscan.exploratoryAnalysis.DistanceToNearestNeighborDriver \
--master yarn \
--deploy-mode cluster \
--driver-cores 4 \
--num-executors 10 \
--executor-memory 8g \
--executor-cores 4 \
--conf spark.yarn.executor.memoryOverhead=1024 \
--conf spark.scheduler.mode=FAIR \
spark_dbscan-assembly-0.0.5-SNAPSHOT.jar \
--ds-master "yarn-cluster" \
--ds-jar hdfs:///user/isaac/spark_dbscan-assembly-0.0.5-SNAPSHOT.jar \
--ds-input hdfs:///data/isaac/dbscan-parameter-tuning \
--ds-output hdfs:///data/isaac/gdelt-dbscan-dnn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment