Skip to content

Instantly share code, notes, and snippets.

@l00ps

l00ps/security-onion-letsencrypt.md

Last active August 5, 2024 17:16
Show Gist options
  • Save l00ps/5753fa7fe55f6757fcc088506637a13e to your computer and use it in GitHub Desktop.
Save l00ps/5753fa7fe55f6757fcc088506637a13e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
security-onion-letsencrypt.md

Automate LetsEncrypt certificate deployment in SecurityOnion

Prerequisites

Turn on user certs in Security Onion (do this only once)

  1. Administration -> Configuration
  2. Options -> Show all configurable settings, including advanced settings
  3. Filter "cert"
  4. nginx -> ssl -> Replace Default Cert (ssl/tls in SO <= 2.4.80)
  5. Replace deafult cert: set to True
  6. Don't replace the key files yet - we're going to automate this!

Setup LetsEncrypt

Note: some of these commands may seem unnecessary, e.g. the symlinks. Don't skip them. SecurityOnion 2.4 is based on Oracle Linux, which has some quirks.

  1. Log in as your admin user and su to root.

    sudo su

  2. Install snapd.

    yum install epel-release
yum install -y snapd
ln -s /var/lib/snapd/snap /snap

    Check to ensure that the snapd service is running.

    sudo systemctl status snapd

    If it is not running, start the service.

    sudo systemctl start snapd

  3. Install certbot.

    snap install --classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot

  4. Install certbot Cloudflare DNS plugin.

    snap set certbot trust-plugin-with-root=ok
snap install certbot-dns-cloudflare

  5. Initialize the /etc/letsencrypt directory by running certbot without any arguments. Ignore the errors.

    certbot

  6. Add Cloudflare DNS API token.

    TOKEN={enter your Cloudflare API token here}
echo dns_cloudflare_api_token=$TOKEN > /etc/letsencrypt/cloudflare.ini
chmod 400 /etc/letsencrypt/cloudflare.ini

  7. Request the certificate. Replace your.fqdn.xyz with the FQDN of your SecurityOnion server.

    FQDN=your.fqdn.xyz
certbot certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini -d $FQDN

  8. Copy the cert and key to SecurityOnion's nginx salt config.

    cp -f /etc/letsencrypt/live/$FQDN/privkey.pem /opt/so/saltstack/local/salt/nginx/ssl/ssl.key
chmod 644 /opt/so/saltstack/local/salt/nginx/ssl/ssl.key
cp -f /etc/letsencrypt/live/$FQDN/fullchain.pem /opt/so/saltstack/local/salt/nginx/ssl/ssl.crt
chmod 640 /opt/so/saltstack/local/salt/nginx/ssl/ssl.crt

  9. Restart nginx.

    so-nginx-restart

    Clear your browser cache (or use an incognito window) and visit your SO admin page to ensure that the new certificate is presented.

  10. Create a post-hook to perform this each time certs are renewed.

    cat <<EOF > /etc/letsencrypt/copy_to_nginx.sh
cp -f /etc/letsencrypt/live/$FQDN/privkey.pem /opt/so/saltstack/local/salt/nginx/ssl/ssl.key
chmod 644 /opt/so/saltstack/local/salt/nginx/ssl/ssl.key
cp -f /etc/letsencrypt/live/$FQDN/fullchain.pem /opt/so/saltstack/local/salt/nginx/ssl/ssl.crt
chmod 640 /opt/so/saltstack/local/salt/nginx/ssl/ssl.crt
so-nginx-restart
EOF

chmod +x /etc/letsencrypt/copy_to_nginx.sh

  11. Test certbot renew, and create a cron job for it.

    certbot renew --dry-run --post-hook /etc/letsencrypt/copy_to_nginx.sh
crontab -e

  12. Place the following at the TOP of the file and save:

    # check certs once a week
10 4 * * 0 certbot renew --post-hook /etc/letsencrypt/copy_to_nginx.sh

Maintenance

If the steps above succeeded, there is no special maintenance required. The cron job will renew the certs automatically before they expire.

Use SecurityOnion's built-in soup utility to maintain your SecurityOnion stack.

@SignalGroup
Copy link

When I attempt step 3 - snap install --classic certbot I receive an error message "error: cannot communicate with server: Post "http://localhost/v2/snaps/certbot": dial unix /run/snapd.socket: connect: no such file or directory". Am I missing something?

@nick-lupien
Copy link

When I attempt step 3 - snap install --classic certbot I receive an error message "error: cannot communicate with server: Post "http://localhost/v2/snaps/certbot": dial unix /run/snapd.socket: connect: no such file or directory". Am I missing something?

Looks like snapd isn't running.

Pardon the ChatGPT copypasta:

The error message you're encountering typically indicates that the Snap daemon (snapd) is not running or is not installed correctly. Here are some steps to troubleshoot and resolve this issue:

  1. Check if snapd is installed:

    sudo apt update
sudo apt install snapd

  2. Ensure the snapd service is running:

    sudo systemctl enable snapd
sudo systemctl start snapd

  3. Check the status of the snapd service:

    sudo systemctl status snapd

    This command should show that the service is active and running. If it's not, you might see some error messages that can provide more insight into what's wrong.

  4. Enable and start the snapd.socket:

    sudo systemctl enable snapd.socket
sudo systemctl start snapd.socket

  5. Check if the snapd.socket is active:

    sudo systemctl status snapd.socket

  6. Reboot the system:
    Sometimes, a simple reboot can help to ensure all services start correctly.

    sudo reboot

  7. Install Certbot:
    Once snapd is running correctly, you can try installing Certbot again:

    sudo snap install --classic certbot

By following these steps, you should be able to resolve the issue and successfully install Certbot. If you encounter any specific error messages during these steps, please let me know, and I can help you troubleshoot further.

@adamphetamine
Copy link

Just a note to let you know- for v2.4.90 of SO
in the GUI at step 4 probably should be
nginx -> ssl -> Replace Default Cert (or remove 'ssl/tls' - either would be ok)
at step 2 I also had to execute
sudo systemctl start snapd

But overall this is remarkable- possibly the easiest to follow linux tutorial ever, and I've written a lot...
Thank you!

@l00ps
Copy link
Author

l00ps commented Aug 4, 2024

Just a note to let you know- for v2.4.90 of SO in the GUI at step 4 probably should be nginx -> ssl -> Replace Default Cert (or remove 'ssl/tls' - either would be ok) at step 2 I also had to execute sudo systemctl start snapd

But overall this is remarkable- possibly the easiest to follow linux tutorial ever, and I've written a lot... Thank you!

Thanks for the corrections! Glad you enjoyed it!

@SignalGroup
Copy link

When I attempt step 3 - snap install --classic certbot I receive an error message "error: cannot communicate with server: Post "http://localhost/v2/snaps/certbot": dial unix /run/snapd.socket: connect: no such file or directory". Am I missing something?

Looks like snapd isn't running.

Pardon the ChatGPT copypasta:

The error message you're encountering typically indicates that the Snap daemon (snapd) is not running or is not installed correctly. Here are some steps to troubleshoot and resolve this issue:

  1. Check if snapd is installed:

    sudo apt update
sudo apt install snapd

  2. Ensure the snapd service is running:

    sudo systemctl enable snapd
sudo systemctl start snapd

  3. Check the status of the snapd service:

    sudo systemctl status snapd

    This command should show that the service is active and running. If it's not, you might see some error messages that can provide more insight into what's wrong.

  4. Enable and start the snapd.socket:

    sudo systemctl enable snapd.socket
sudo systemctl start snapd.socket

  5. Check if the snapd.socket is active:

    sudo systemctl status snapd.socket

  6. Reboot the system:
    Sometimes, a simple reboot can help to ensure all services start correctly.

    sudo reboot

  7. Install Certbot:
    Once snapd is running correctly, you can try installing Certbot again:

    sudo snap install --classic certbot

By following these steps, you should be able to resolve the issue and successfully install Certbot. If you encounter any specific error messages during these steps, please let me know, and I can help you troubleshoot further.

That did it. Thanks! I had to start snapd twice. There was an error the first time. I waited a minute or two and tried again and it worked. Great write up!

@SignalGroup
Copy link

Everything worked out perfect. I had to remember to use sudo in front of most commands. The entries where it created files (ini & sh) I had to create them manually set chmod to 777 then add the text then reset the permissions. Most likely cause was my lack of Linux experience. Only other thing that caught me out was exiting VIM and saving. Again, Thanks, great write up.

@adamphetamine
Copy link

I've been unable to get Elastic to offer the new cert to clients.
If you go to http://FQDN:8220 and look at the SSL you can see it seems to know about the FQDN as it is listed as a SAN, but it still offers the older hostname in my case.
We could paste the new cert into the GUI, but don't want to do this every 3 months.
Probably my lack of expertise and not confident enough to play with it.
Anyone have success with this?

@l00ps
Copy link
Author

l00ps commented Aug 5, 2024 •
edited
Loading

@adamphetamine The cert used by Elastic Filebeat on port 8220 is generated separately and stored at /etc/pki/filebeat.crt /etc/pki/elasticfleet-server.crt. This one might be trickier since I think the Filebeat agent packages are shipped with this cert. You could hypothetically copy your LetsEncrypt cert here with the post-hook but I don't know the full implications of it. Worth a backup & try!

@adamphetamine
Copy link

Confession time- I wrote this for SO v2.3- but very glad you've done the work this time...
https://github.com/adamphetamine/letsencrypt-security-onion

That's great information, thank you!
I'm using macOS, and the Elastic Agent downloads from the Server aren't packaged properly- so I've been using the install instructions from Elastic.
If the Server offers a cert that is valid, we don't have to use the --insecure flag so this is definitely something I will test.
Thanks again

@l00ps
Copy link
Author

l00ps commented Aug 5, 2024

@adamphetamine Nice! One of these days I'll get the energy to build the docker image for it and put in a PR to SO.

@adamphetamine
Copy link

definitely out of my depth here. I replaced the cert at /etc/pki/elasticfleet-server.crt and rebooted the Server. It reverted to the previous, untrusted cert. Not really surprising, but I wonder if we might have better luck replacing the managerssl cert and key like in this older example- looks dangerous though-
https://github.com/adamphetamine/letsencrypt-security-onion/blob/main/move-certs.sh

Anyway I've reached the limit of my knowledge so I'll leave it to smarter people, thanks again for this writeup, much appreciated.

Just to indicate the scale of the issue-

ls /etc/pki
ca.crt                  elasticfleet-agent.p8      elasticfleet-logstash.p8     elasticfleet-server.key  filebeat.key  java            registry.crt  tls
ca.key                  elasticfleet-kafka.crt     elasticfleet-lumberjack.crt  elasticsearch.crt        filebeat.p8   managerssl.crt  registry.key
ca-trust                elasticfleet-kafka.key     elasticfleet-lumberjack.key  elasticsearch.key        influxdb.crt  managerssl.key  rpm-gpg
elasticfleet-agent.crt  elasticfleet-logstash.crt  elasticfleet-lumberjack.p8   elasticsearch.p12        influxdb.key  redis.crt       rsyslog
elasticfleet-agent.key  elasticfleet-logstash.key  elasticfleet-server.crt      filebeat.crt             issued_certs  redis.key       swid

@l00ps
Copy link
Author

l00ps commented Aug 5, 2024

I replaced the cert at /etc/pki/elasticfleet-server.crt and rebooted the Server. It reverted to the previous, untrusted cert.

Yeah, that's kind of what I was afraid of. I don't know which process manages those certs. I did notice that they are mounted by the Elastic Fleet container as read-only, so if they're getting modified, it's happening somewhere else - probably the salt config for Elastic Fleet.

A start for your search:

docker inspect so-elastic-fleet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment