How to debug Linux kernel on Windows

This tutorial shows how to debug Linux kernel on a Windows operating system using another virtual machine with a Linux on it. Virtual machines will be connected by a serial port provided by VirtualBox using a named pipe on a Windows host system.

Prerequisites

• Physical computer with Windows operating system. This tutorial uses Windows 10, but any recent version of Windows (8, 8.1) should suffice. We will refer to this computer as the host system.
• Virtual machine with a Linux operating system. We will refer to this computer as the debugged system. We assume that you have correct source files for the running kernel. You can get these files either from a package manager provided by your distribution or from the official kernel page. In either case, it is important that you have access to the vmlinux file that contains binary image of the kernel.
• Virtual machine with a Linux operating system. We will refer to this machine as the debugging system. It doesn't have to be the same distribution as the debugged system, but it has to contain the source files of the debugged system kernel. Easiest way to transfer these files from the debugged system to the debugging system is by using the rsync command.

This tutorial uses VirtualBox as a virtualization tool. However, you can follow the tutorial even if you're using another virtualization software. Follow your tool's manual for more info on how to configure serial ports.

Guide

1. Start VirtualBox and go to the settings of the debugged system. Go to the section Serial ports. Enable the first port, leave it as COM1. As a port mode, choose Host pipe. Check the Connect to existing pipe/socket option. In the Path/Address field write \\.\pipe\debug (Note: debug can be any other string, but make sure you'll use the same string in the next step.)

2. Do the exact same setup for the debugging system, but leave the checkbox unchecked.

3. Start up the debugging system. VirtualBox will create a new named pipe for the serial port.

4. On the debugging system open the terminal and type:

   sudo su
   stty -F /dev/ttyS0 raw -echo
   socat -d -d /dev/ttyS0 pty &

   Take note of a virtual terminal returned by the last command.

   Explanation: Second command sets the serial port to a raw mode. That means it won't be buffered and it's very important for debugging to work. Third command creates a virtual terminal that allows duplex communication for gdb on a serial port.

5. Open a new terminal tab and change directory to where kernel source files are located. Run this command:

   sudo gdb vmlinux

   Debugging system is now ready.

6. Start up the debugged system. From the grub menu choose Advanced options for Debian GNU/Linux. You'll be presented with a list of available kernels on the system. Choose the correct one (usually the first one) and press e. Find line that begins with linux /boot/vmlinuz-4... and add this command to the end of that line:

   kgdboc=ttyS0,115200 kgdbwait
   

   Press Ctrl+X or F10 to boot. Kernel will now wait for the debugger to connect.

7. On the debugger system in the gdb console type:

   target remote /dev/pty/X
   

   Where X is number of the virtual terminal returned by the socat command earlier. Debugger will now connect to the machine and you can type c and press Enter (you're welcome to add some breakpoints before that) to continue booting the system.

Happy debugging!