Skip to content

Instantly share code, notes, and snippets.

@jasonrig

jasonrig/spark-2.4.3-hadoop2.7.tcl

Created August 9, 2019 03:45
Show Gist options
  • Save jasonrig/4acc349b1639d64b8b91d781cdeea391 to your computer and use it in GitHub Desktop.
Save jasonrig/4acc349b1639d64b8b91d781cdeea391 to your computer and use it in GitHub Desktop.
Download ZIP
Example modulefile for Spark on HPC clusters
Raw
spark-2.4.3-hadoop2.7.tcl
#%Module1.0
## Spark 2.4.3 modulefile
##
proc ModulesHelp { } {
puts stderr "Provides the Spark environment for large-scale data processing"
puts stderr "This version has been prebuilt with Scala 2.11."
}
module-whatis "Apache Spark™ is a unified analytics engine for large-scale data processing."
set sparkHome /home/jrigby/spark/spark-2.4.3-bin-hadoop2.7
module load java
prepend-path PATH "$sparkHome/bin"
prepend-path PATH "$sparkHome/sbin"
setenv SPARK_HOME "$sparkHome"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment