Last active
May 28, 2020 04:35
-
-
Save jtzero/a0114a640524bceeee4a2c3e554a259f to your computer and use it in GitHub Desktop.
Jupyter command `jupyter-/home/jtzero/.local/share/virtualenvs/ml-.../bin/find_spark_home.py` not found.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
``` | |
(email_ml) jtzero:~/Documents/ml$ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark | |
Traceback (most recent call last): | |
File "/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/jupyter", line 8, in <module> | |
sys.exit(main()) | |
File "/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/jupyter_core/command.py", line 247, in main | |
command = _jupyter_abspath(subcommand) | |
File "/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/jupyter_core/command.py", line 134, in _jupyter_abspath | |
'Jupyter command `{}` not found.'.format(jupyter_subcommand) | |
Exception: Jupyter command `jupyter-/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/find_spark_home.py` not found. | |
/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/pyspark: line 24: /bin/load-spark-env.sh: No such file or directory | |
/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/pyspark: line 77: /bin/spark-submit: No such file or directory | |
``` | |
REASON: https://github.com/apache/spark/pull/28256 | |
workaround: | |
set SPARK_HOME | |
`SPARK_HOME=/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/pyspark PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark` | |
finding spark home: `"$(pip show pyspark | grep 'Location: ' | sed 's/Location: //g')/pyspark"` |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment