These variables are used accross the guide
- $HOME = Home root folder
- $SPACE_NAME = Your squad/unique_id to create a new branch project space
- $TMP_FILE_NAME = Auto generated .tmp file to be used on deploy
- $SPARK_MASTER_POD_NAME = Pod name that is running the Spark Master instance
This process is intented to make a build of epm project and run it on Spark master cluster pod.
- Go to the project's folder (
cd /vagrant/scala/com/ibm/epm/magichat
) - Enter your space folder (
cd /$SPACE_NAME
) - Run bazel build command (
bazel build //scala/com/ibm/epm/magichat/$SPACE_NAME:$SPACE_NAME_deploy.jar
)- This will generate a .exp file under $HOME/.cedp/creds.exp
- and a .jar file under $HOME/vagrant/bazel-bin/scala/com/ibm/epm/magichat/$SPACE_NAME/$SPACE_NAME_deploy.jar
** Be careful to add the
_deploy
suffix after the $SPACE_NAME on run command
- Go machine's to root folder (
cd
) - Go to tools folder (
cd /vagrant/tools
) - execute the cedp_credentials.sh file with the cache param (
./cedp_credentials.sh cache
)- This will generate a .tmp file with a random name under /tmp folder
- Go back to root folder (
cd
) - Go to tmp folder (
cd tmp
) - Locate the .tmp file (
ls *.tmp
) - Copy this file name ($TMP_FILE_NAME)
- Go back to machine's root folder (
cd
) - Run kubectl command to list available pods (
kubectl get pods
) - Copy the spark-master pod name ($SPARK_MASTER_POD_NAME)
- Go to root folder (
cd
) - Move the generated .tmp file to Spark master pod: (
kubectl cp $HOME/tmp/$TMP_FILE_NAME.tmp $SPARK_MASTER_POD_NAME:/spark/Spark/TMP_FILE_NAME.tmp
) - Move the generated .exp file to Spark Master pod: (
kubectl cp $HOME/vagrant/.cedp/creds.exp $SPARK_MASTER_POD_NAME:/root/.cedp/creds.exp
) - Move the generated .jar file to Spark Master pod (
kubectl cp $HOME/vagrant/bazel-bin/scala/com/ibm/epm/magichat/$SPACE_NAME/SPACE_NAME_deploy.jar $SPARK_MASTER_POD_NAME:spark/Spark/jars/$SPACE_NAME_deploy.jar
)
- Enter Spark master pod terminal (
kubectl exec -it $SPARK_MASTER_POD_NAME /bin/bash
)
- Run spark submit command
(
spark-submit --conf spark.driver.extraClassPath=/spark/Spark/jars/$SPACE_NAME_deploy.jar /spark/Spark/jars/$SPACE_NAME_deploy.jar --cf=/spark/Spark/secrets/$TMP_FILE_NAME.tmp
)
Syntom:
WARNING:root:could not open file '/etc/apt/sources.list.d/apt_dockerproject_org_repo.list'
Solution: You are running spark-submit command from within vagrant/local machine
Syntom:
Error: Cannot load main class from JAR file:/dcerag_deploy.jar
Solution: Set the absolute path to .jar file