Debian KVM console on a headless server

At work I use a Debian KVM with an encrypted root filesystem as a workstation (our physical workstations run Windows) running on a headless server. This means that I have to use the QEMU' VNC console to enter the password for the root filesystem very early in the boot process.

Unfortunately VNC is unsecure and anyway QEMU only binds VNC on 127.0.0.1. It would be easy to create an SSH tunnel, but this is administratively prohibited here and it is cumbersome to temporarily modify sshd_config(5) each time. So I tried a Netfilter DNAT rule as a workaround but Linux' network stack contains a very annoying line of code which checks that packets destined 127.0.0.1 comes from 127.0.0.1 as well. If you see some logs like this, you have probably been biten by it too:

Jun  2 18:14:20 srv kernel: martian destination 127.0.0.1 from 10.1.2.2, dev br0

So I gave up VNC and configured the KVM domain to use the serial port like any other headless server.

Supposedly your VM is already running so we will make the changes here first. There are three things to be told to use the serial console, which are in time-order:

• the bootloader (GRUB here);


• the kernel;


• init(8) for the login prompt.

On Debian, the first two things can be done easily through /etc/default/grub.

# Bootloader part.
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

# Kernel command-line ("quiet" has no matter in our business):
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,9600n8 quiet"

Then regen the grub.cfg:

# upgrade-grub

If you do not use Debian, here is the relevant part of the generated /boot/grub/grub.cfg:

serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1
if terminal_input serial ; then true ; else
  # For backward compatibility with versions of terminal.mod that don't
  # understand terminal_input
  terminal serial
fi
if terminal_output serial ; then true ; else
  # For backward compatibility with versions of terminal.mod that don't
  # understand terminal_output
  terminal serial
fi

menuentry "Linux 2.6.32-trunk-amd64" {
        insmod ext2
        set root='(hd0,1)'
        search --no-floppy --fs-uuid --set 9245a9e3-8ea5-4170-a19b-17d10051c107
        echo    Loading Linux 2.6.32-trunk-amd64 ...
        linux   /vmlinuz-2.6.32-trunk-amd64 root=/dev/mapper/vg0-root ro  console=tty0 console=ttyS0,9600n8 quiet
        echo    Loading initial ramdisk ...
        initrd  /initrd.img-2.6.32-trunk-amd64
}

Regarding the login prompt on serial console, edit /etc/inittab:

T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100

Now your VM is configured, let's configure your KVM domain. Dump the configuration of your vm, and change the <serial> and <console> part to use a PTY (you can choose an arbitrary PTY, /dev/pts/24 here, as it seems to be redefined each time the VM is started). Other interfaces are possible, like TCP, pipe, stdio... (see the libvirt domain XML format) but I chose PTY because it can be easily attached using screen(1) and cannot be easily snooped:

# virsh dumpxml mykvm > mykvm.xml
# vi mykvm.xml
   <serial type='pty'>
     <source path='/dev/pts/24'/>
     <target port='0'/>
   </serial>
   <console type='pty' tty='/dev/pts/24'>
     <source path='/dev/pts/24'/>
     <target port='0'/>
   </console>

Then stop your VM, redefine your KVM domain and restart it:

# virsh shutdown mykvm      # or run shutdown(8) inside the VM
# virsh undefine mykvm
# virsh define mykvm.xml
# virsh start mykvm

You can attach the console using:

# virsh console mykvm

To detach, use Ctrl + $


If you attach quickly enough after starting it, you will even see the Grub menu!

Quick HOWTO for building Qumranet's KVM

This post really deserves the name scrawl, but I thought sharing my experience in building a Qumranet's KVM snapshot on Linux could helpful for others. Indeed it was not as straightforward as it seemed at first glance and it required some grope and wandering on Google.

You will need KVM (surprising, isn't it?) and Linux kernel sources corresponding to your kernel.

FWIW, KVM snapshots can be downloaded here.

First, let's prepare the kernel tree. This step is only important for people who use packaged kernel (or IOW who don't build their kernel themselves and have extracted the kernel source tree for the sole purpose of building KVM). For others, this step could be avoided because the targets we will use are performed implicitely when building the kernel.

root# tar xzf linux-2.6.23.tar.gz
root# cd linux-2.6.23.tar.gz
root# cp /boot/config-2.6.23 .config
root# make oldconfig prepare scripts
root# cd -

Next we will build KVM. Beware your GCC version! The current major branch is GCC 4 and it is shipped with almost if not all recent Linux distributions. Unfortunately, QEMU (on which KVM's userland is heavily based, not to say they've reused QEMU with little modifications) doesn't build with GCC 4. It requires GCC 3.2 or GCC 3.4 (a thorough explanation is provided in Fabrice Bellard's paper, "QEMU, a fast and portable dynamic translator", USENIX 2005). So you need to install this version of GCC as well. If you are running Debian for instance, this is pretty straightforward :

root# aptitude install gcc-3.4

And you will have a new compiler named gcc-3.4. On other distros, you will have to either find a package of GCC 3 or build it manually as described on Gentoo Wiki.

Finally you will simply have to build KVM with a few special arguments to the configure script:

root# tar xzf kvm-snapshot-20080117.tar.gz
root# cd kvm-snapshot-20080117
root# ./configure --qemu-cc=gcc-3.4 --kerneldir=$OLDPWD/linux-2.6.23 --prefix=/opt/kvm-snapshot-20080117
root# make all install

And this should work. :-)

As you may have noticed, I installed KVM in /opt in order to avoid messing with your tidy package management system.