We have deployed a couple of long-running Raspberry Pis equipped with cameras and sensors reporting into our network 24/7. All these are under uptime monitoring for us to keep track of the network availability.

From to time one of our Raspberry Pis freezes in the field either because of a kernel or a hardware issue. In that case there is nothing that can be done with software anymore. You can’t connect to it, can’t ping the Pi – It becomes impossible to send it a restart command in any way to bring it back to normal operation. Debugging into these events you might find indications of such as freeze in the /var/log/kernel.log file, and only manual powering down, and powering up again brought it back to live.

One little known Pi features is a builtin hardware watchdog. This little hardware service will once enabled watch the system activity and automatically power cycle the Raspberry Pi once it gets stuck.

It’s done in few steps directly on a terminal on your Pi:

1. Enable the hardware watchdog on your Pi and reboot
2. Install the watchdog system service
3. Configure the watchdog service
4. Enable the service

For detailed instructions on how to do this check out the tutorial at: https://diode.io/raspberry%20pi/running-forever-with-the-raspberry-pi-hardware-watchdog-20202/

Now next time your Raspberry Pi freezes, the hardware watchdog will restart it automatically after 15 seconds.

In some cases our Raspberry Pi Zero W would crash it’s wifi driver but not completely go down. This wouldn’t trigger the watchdog, because the device is still running - just not communicating anymore… To handle this case we added one more line to the watchdog.conf configuration file. Like this: sudo su echo 'interface = wlan0' >> /etc/watchdog.conf With this additional configuration line, the watchdog will also power cycle the Raspberry Pi when the wifi interface wlan0 gets into trouble.

Source:

• Raspberry Pi Forum Post >> https://forums.raspberrypi.com/viewtopic.php?t=328559