Mattermost Load Testing - The Complete Instructions

Problem

You want to see how many users a free tier AWS instance can support

Solution

You can use mattermost-load-test-ng to run load tests of tens of thousands of users, but it can be difficult to set up. This is a step

You Will Need

A Mattermost instance with Performance Metrics Enabled
A Prometheus server set up to monitor the Mattermost instances
One or more systems to run the load test agents
A system to run the coordinator

Setting up the Mattermost Server

Use a server that you can reset easily, i.e. not your production system. The load test will create users as needed and increase and decrease activity based on the metrics from the Prometheus server.

Also, to set up the load test users your server will need to have Signup > Enable Open Server set to True and enable server metrics.

Next, configure the access details in load-test-ng's config.json.

Setting up the agents

The easiest way to run the agents is to use a systemd service to run the ltapi, which the coordinator will talk to to start the tests. The script ltagent_setup.sh goes through this setup for Ubuntu using these steps:

Install Golang
Clone the mattermost-load-test-ng repository
Create the load test agent systemd service

Setting up Prometheus

The coordinator relies on the Prometheus server to determine how many users your server can support. This Prometheus instance should scrape your Mattermost app servers on port :8067 and the load test agents on port :4000. Here is an example:

Note: mattermost-load-test-ng requires jobs named mattermost and loadtest, with the Mattermost app servers and load test agents in their respective jobs.

# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.

  external_labels:
      monitor: 'mattermost-load-test-monitoring'

scrape_configs:
  - job_name: 'mattermost'
    static_configs:
      - targets: ['mattermost-app-1:8067', 'mattermost-app-2:8067']
  - job_name: 'loadtest'
    static_configs:
      - targets: ['load-test-agent-1:4000', 'load-test-agent-2:4000']

Configuring the Coordinator

When you set up the coordinator, make sure it can connect to your agents on the configured port (default 4000).

First, you create the test in the coordinator, either locally or via the HTTP API. If you're running the coordinator locally, you set the values in coordinator.json.

Agents: Load test agents are configured in ClusterConfig.Agents:

{
  "ClusterConfig": {
    "Agents": [
      {
        "Id": "ltagent1",
        "ApiURL": "http://ltagent1:4000"
      },
      {
        "Id": "ltagent2",
        "ApiURL": "http://ltagent2:4000"
      }
    ],
    "MaxActiveUsers": 1000
  }
  // ...
}

Configuring the Load Test

All load test configuration happens on the coordinator machine. The files that you should have are:

config.json - This is where you set the Mattermost server and user settings
coordinator.json - This is where you specify the load test agents and the prometheus server, as well as the queries to use to monitor the performance of the load test

Running the Load Test

Before you run the load test you need to create the users and teams. From the coordinator, run this command:

Note: This command must be run from inside the mattermost-load-test-ng directory.

go run ./cmd/ltagent init

Next, run this command to start the load test:

go run ./cmd/ltcoordinator

This will load the configuration into the load test agents and start the load test. To stop, press Ctrl + C

icelander/loadtest.md