Created
March 4, 2020 13:19
-
-
Save mostafa-asg/68f5f29c7b73c419610ceafcf726379d to your computer and use it in GitHub Desktop.
Spark: Example of command which launches multiple workers on each slave node:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ``` | |
| SPARK_WORKER_INSTANCES=3 SPARK_WORKER_CORES=2 ./sbin/start-slaves.sh | |
| ``` | |
| This will launch three worker instances on each node. Each worker instance will use two cores. | |
| Also it is possible to manually start workers and connect to Spark’s master node. To do this use command: | |
| ``` | |
| ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://IP:PORT | |
| ``` |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment