Managing backend servers with Supervisor

This post will describe how we can run our different backends without any worry, mostly from crashing and then recovering. We will build on top of our already configurated Vagrant guest machine running Apache with reverse proxy [1, 2, 3]. Until now we are running our backend servers (SimpleHTTPServer and simple-websocket-server) directly from our provisioning script as a background daemon. This means that if we halt the machine and bring it back up, our servers are not running and need to be started again. It also means that if one of our backends experiences a crash (believe me, it happens) it does not recover to a running state. This is where Supervisor comes in. It is a background daemon that can manage, start, stop and restart services and programs as we configure. This guide will cover

• How to install and configure Supervisor
• How to configure our provisioning script to run our manager
• How to configure vagrant to run supervisor on boot

Step 1

If pip is not already provisioned on the system, download get-pip.py to vagrantfolder and install. Then install supervisor using pip.

$ python /vagrant/get-pip.py
$ pip install supervisor

Create a supervisor configuration file using the example-configuration.

$ echo_supervisord_conf > /vagrant/supervisord.conf

Step 2

Make sure to kill any of our backends if they are already running.

$ ps ax | grep [S]impleHTTPServer 
$ ps ax | grep [S]impleExampleServer
$ kill _PID1_ _PID2_

Step 3

Configure supervisord.conf to manage our backends.

[...]
[program:SimpleHttpServer]
command=python -m SimpleHTTPServer 8080
directory=/vagrant/www/html
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/simplehttpserver.out.log

[program:SimpleWebSocketServer]
command=python SimpleExampleServer.py --example echo
directory=/vagrant/simple-websocket-server-master/SimpleWebSocketServer
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/simplewebsocketserver.out.log

This manages our two backends, making sure they start when supervisor starts and that they will restart whenever they fail. Run our managed supervisor services.

$ supervisord -c /vagrant/supervisord.conf

Check on our services using the CLI.

$ supervisorctl -c /vagrant/supervisord.conf status
SimpleHttpServer RUNNING pid 3377, uptime 0:06:18
SimpleWebSocketServer RUNNING pid 3376, uptime 0:06:18

Navigate again to http://127.0.0.1:4567/websocket.html to make sure everything is still working.

Note it is possible to check whether our manager works by killing one of our backends and running the status-command again to see it is rebooted with a new PID.

Step 4

Configure our Vagrant provisioning-script bootstrap.sh to include our changes.

#!/usr/bin/env bash
apt-get update
apt-get install -y apache2
if ! [ -L /var/www ]; then
 rm -rf /var/www
 ln -fs /vagrant/www /var/www
fi
cp /vagrant/001-mysite.conf /etc/apache2/sites-available/
a2dissite 000-default.conf
a2ensite 001-mysite.conf
a2enmod proxy
a2enmod proxy_http
a2enmod proxy_wstunnel
service apache2 restart
cp /vagrant/simple-websocket-server-master/SimpleWebSocketServer/websocket.html /vagrant/www/html
python /vagrant/get-pip.py
pip install supervisor
cp /vagrant/supervisord.conf /etc/supervisord.conf
supervisord

Step 5

Ensure supervisor is started every time we bring the system up. Normally this is done by /etc/init.d/-scripts, but as we are running Vagrant we can use its vagrantfile to load supervisor.

config.vm.provision "shell", inline: "supervisord", run: "always"

Conclusion

We have succeded in configuring a guest machine that will launch our backends on boot and also monitor them in case of failure.