Skip to content

Instantly share code, notes, and snippets.

@thaibui

thaibui/useful_spark_optimization_options.json

Created May 10, 2016 20:41
Show Gist options
  • Save thaibui/a364b49124db7627f0699809eab607a1 to your computer and use it in GitHub Desktop.
Save thaibui/a364b49124db7627f0699809eab607a1 to your computer and use it in GitHub Desktop.
Download ZIP
Useful Spark optimization options
Raw
useful_spark_optimization_options.json
"-Dspark.shuffle.blockTransferService": "netty",
"-Dspark.shuffle.io.numConnectionsPerPeer": 10,
"-Dspark.shuffle.consolidateFiles": true,
"-Dspark.shuffle.compress": true,
"-Dspark.shuffle.file.buffer.kb": 256,
"-Dspark.shuffle.manager": "sort",
"-Dspark.sql.shuffle.partitions": 1000,
"-Dspark.io.compression.codec": "snappy",
"-Dspark.io.compression.snappy.blockSize": 2048,
"-Dspark.shuffle.memoryFraction": 0.8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment