MISP Docker Dev Env

This is a short guide on how to set up a development environment for MISP using MISP/misp-docker docker images.

Prerequisites

1. Install docker and docker-compose.
2. Clone MISP project repo: git clone --recurse-submodules git@github.com:MISP/MISP.git
3. Clone MISP/misp-docker repo: git clone https://github.com/MISP/misp-docker.git
4. Clone MISP/PyMISP repo: git clone https://github.com/MISP/PyMISP.git

From now on we assume that projects are cloned in the /home/myuser directory.

Docker dev env setup

By default MISP/misp-docker uses a local clone of the codebase inside the misp-core docker container. By using a docker-compose.override.yml file we can make it map the MISP repo we checked out in our host machine instead.

This way we can make local changes on our host machine and see the changes reflected in the container.

Configuration

• Copy the template.env to .env
• Customize .env based on your needs (optional step)

IMPORTANT: Set up the ADMIN_KEY setting in the .env file.

Mapping our local MISP repo

Create docker-compose.override.yml file in /home/myuser/misp-docker/ directory.

docker-compose.override.yml

version: '3'
services:
  misp:
    volumes:
      - "/home/myuser/MISP/app/composer.json:/var/www/MISP/app/composer.json"
      - "/home/myuser/MISP/app/phinx.php:/var/www/MISP/app/phinx.php"
      - "/home/myuser/MISP/app/Console:/var/www/MISP/app/Console"
      - "/home/myuser/MISP/app/Controller:/var/www/MISP/app/Controller"
      - "/home/myuser/MISP/app/View:/var/www/MISP/app/View"
      - "/home/myuser/MISP/app/Model:/var/www/MISP/app/Model"
      - "/home/myuser/MISP/app/webroot:/var/www/MISP/app/webroot"
      - "/home/myuser/MISP/app/Locale:/var/www/MISP/app/Locale"
      - "/home/myuser/MISP/app/Lib/Dashboard:/var/www/MISP/app/Lib/Dashboard"
      - "/home/myuser/MISP/app/Lib/EventReport:/var/www/MISP/app/Lib/EventReport"
      - "/home/myuser/MISP/app/Lib/Export:/var/www/MISP/app/Lib/Export"
      - "/home/myuser/MISP/app/Lib/Tools:/var/www/MISP/app/Lib/Tools"
      - "/home/myuser/PyMISP:/var/www/MISP/app/PyMISP"
      - "./core/files/enable_xdebug.sh:/custom/files/customize_misp.sh"
    extra_hosts:
      - "host.docker.internal:host-gateway"

Share MISP directory group ownership with your host user:

sudo usermod -a -G www-data righel
chgrp righel /var/www
chmod g+rwxs /var/www

XDebug and Docker

A few tweaks are required if you want to debug the php-fpm process running inside the misp docker container.

1. Add a new entrypoint to the examples folder: core/files/enable_xdebug.sh

     #!/bin/bash
   
     apt-get update
     apt-get install php-xdebug
     cat >/etc/php/8.2/fpm/conf.d/20-xdebug.ini <<EOL
     zend_extension=xdebug
     xdebug.mode = debug
     xdebug.client_port = 9999
     xdebug.client_host = host.docker.internal
     xdebug.idekey = VSCODE
     xdebug.xdebug.start_with_request = yes
     xdebug.discover_client_host = 1
     EOL
     /etc/init.d/php8.2-fpm restart
   

2. Add a new volume to docker-compose.override.yml and the extra_host key:

     volumes:
       - "/home/myuser/MISP/app/:/var/www/MISP/app/"
       - "./examples/enable_xdebug.sh:/custom-entrypoint.sh"
     extra_hosts:
       - "host.docker.internal:host-gateway"

3. Configure your IDE to listen incoming XDebug connections on the port 9999.

   Example for Visual Studio Code:

   launch.json

     {
         "version": "0.2.0",
         "configurations": [
             {
                 "name": "Listen for Xdebug",
                 "type": "php",
                 "request": "launch",
                 "port": 9999,
                 "pathMappings": {
                     "/var/www/MISP/app": "${workspaceRoot}/app",
                 },
             },
         ]
     }

CLI Debug

For debugging MISP shell.

$ export XDEBUG_MODE=debug XDEBUG_SESSION=1

misp-modules debugging

TODO

Running MISP dockerized

Start dockerized MISP in deattached mode:

cd /home/myuser/misp-docker
docker-compose build misp-core
docker-compose up -d
docker-compose exec -T --user www-data misp-core bash -c "app/Console/cake Live 1"

After docker-compose spins up the containers you should be able to browse your local dockerized MISP instance here:

https://localhost

Update MISP

On the host machine (/home/myuser/MISP)

git submodule update --init --recursive

docker-compose exec -T --user www-data misp-core bash -c "app/Console/cake Admin updateMISP"

Run MISP tests

In the host machine:

cd /home/myuser/MISP
export HOST=localhost
export AUTH="The ADMIN_KEY you defined in /home/myuser/misp-docker/.env"
python tests/testlive_comprehensive_local.py -v
python tests/testlive_sync.py -v
python tests/testlive_security.py -v
...

Run PyMISP tests

In the host machine, create the keys.py in the /home/myuser/PyMISP/tests/

With the following content:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

url = "https://localhost"
key = "The ADMIN_KEY you defined in /home/myuser/misp-docker/.env"
verifycert = False

Run the tests:

# from the host machine
docker-compose exec misp-core bash
cd app/PyMISP
# inside the misp-docker docker container
# create virtual env and install python dependencies
apt install python3-virtualenv
python3.11 -m virtualenv -p python3 ./venv
. ./venv/bin/activate
export PYTHONPATH=$PYTHONPATH:./app/files/scripts
pip install -r requirements.txt
pip install -r requirements-dev.txt

# run the tests
python -m pytest -v --durations=0 tests/testlive_comprehensive.py
...

Check MISP logs

Application logs are accesible in your host in /home/myuser/misp-docker/logs.

For MISP container logs:

cd /home/myuser/misp-docker
docker-compose logs -f misp

Applying code changes

First, read the MISP project contributing guide.

1. Fork MISP repository via GitHub UI.
2. Go to your local clone of MISP project and create a new branch for your fix/feature:

   cd /home/myuser/MISP
   git remote add myuser git@github.com:myuser/MISP.git
   git checkout develop
   git checkout -b fix-annoying-bug
   

3. Do your code changes.
4. Keep track of the files you modified using git status and git diff, only add the files you modified to the commits.
5. Commit your changes, example: git commit -m "fix: remove typo in user view email label"
6. Push changes to your remote: git push myuser
7. Check everything is working as expected and then create the pull request via GitHub UI. Always use the develop branch as target for the merge. Add a good description of why you want to get this merged, what issue solves or how to use the feature you want to add providing use-cases if possible.