Skip to content

Instantly share code, notes, and snippets.

@paulrobu
Last active November 18, 2020 16:20
Show Gist options
  • Save paulrobu/cbd05800fdc5e98d6f619a00399a2e4a to your computer and use it in GitHub Desktop.
Save paulrobu/cbd05800fdc5e98d6f619a00399a2e4a to your computer and use it in GitHub Desktop.
Apache Kafka Performance Testing for SSL Clients

Prerequisites:

  1. Config file with SSL settings for producers and consumers (ssl-perf-test.properties in these examples).
  2. A topic to produce and consume from (ssl-perf-test in these examples).

Test the Producer Performance

This examples produces 3 million messages of 1 KiB each.
The -1 value for --throughput means that messages are produced as quickly as possible, with no throttling limit.
Kafka producer related configuration properties like acks and bootstrap.servers are mentioned as part of --producer-props argument.
The SSL config file it’s being referred as value for the --producer-config argument:
$ bin/kafka-producer-perf-test.sh --topic ssl-perf-test --throughput -1 --num-records 3000000 --record-size 1024 --producer-props acks=all bootstrap.servers=broker0:9093,broker1:9093,broker2:9093 --producer.config /path/to/ssl-perf-test.properties

Test the Consumer Performance

This example will read 3 million messages from ssl-perf-test topic, via SSL port 9093.
For better output readability, use jq to transpose the rows and columns:
$ bin/kafka-consumer-perf-test.sh --topic ssl-perf-test --broker-list broker0:9093,broker1:9093,broker2:9093 --messages 3000000 --consumer.config /path/to/ssl-perf-test.properties | jq -R .|jq -sr 'map(./",")|transpose|map(join(": "))[]'

Test the End-To-End Latency

This example will produce and consume 10000 messages of 1 KiB each, with acks value set to 1 (leader acks) and encrypted data transfer via SSL port 9093:
$ bin/kafka-run-class.sh kafka.tools.EndToEndLatency broker0:9093,broker1:9093,broker2:9093 ssl-perf-test 10000 1 1024 /path/to/ssl-perf-test.properties

More info:

https://medium.com/metrosystemsro/apache-kafka-how-to-test-performance-for-clients-configured-with-ssl-encryption-3356d3a0d52b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment