- Add the following lines to your spark-submit command
--num-executors 1 \
--executor-cores 1 \
--conf "spark.executor.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=n,address=localhost:5005,suspend=n" \
--driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005 \
- Configure Run/Debug Configuration in Intellij: Remote -> default settings for localhost
- Run the spark-submit and run the configuration as Debug (mind that app is waiting until debug plugs in)
Add in build.sbt
testOptions in Test += Tests.Argument("-oD")
- Download YourKit
- Evaluate free licence...
- unzip to $YJP_HOME, wherever it is
- Add when calling spark-submit:
--conf "spark.executor.extraJavaOptions=-agentpath:$YJP_HOME/bin/linux-x86-64/libyjpagent.so=sampling" \
--driver-java-options -agentpath:$YJP_HOME/bin/linux-x86-64/libyjpagent.so=sampling \
- run the app
- run $YJP_HOME/bin/yjp.sh
- Dig into SparkSubmit