Wednesday, June 2, 2010

Debian KVM console on a headless server

At work I use a Debian KVM with an encrypted root filesystem as a workstation (our physical workstations run Windows) running on a headless server. This means that I have to use the QEMU' VNC console to enter the password for the root filesystem very early in the boot process.

Unfortunately VNC is unsecure and anyway QEMU only binds VNC on 127.0.0.1. It would be easy to create an SSH tunnel, but this is administratively prohibited here and it is cumbersome to temporarily modify sshd_config(5) each time. So I tried a Netfilter DNAT rule as a workaround but Linux' network stack contains a very annoying line of code which checks that packets destined 127.0.0.1 comes from 127.0.0.1 as well. If you see some logs like this, you have probably been biten by it too:
Jun  2 18:14:20 srv kernel: martian destination 127.0.0.1 from 10.1.2.2, dev br0


So I gave up VNC and configured the KVM domain to use the serial port like any other headless server.

Supposedly your VM is already running so we will make the changes here first. There are three things to be told to use the serial console, which are in time-order:

  • the bootloader (GRUB here);

  • the kernel;

  • init(8) for the login prompt.



On Debian, the first two things can be done easily through /etc/default/grub.
# Bootloader part.
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

# Kernel command-line ("quiet" has no matter in our business):
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,9600n8 quiet"


Then regen the grub.cfg:
# upgrade-grub


If you do not use Debian, here is the relevant part of the generated /boot/grub/grub.cfg:
serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1
if terminal_input serial ; then true ; else
# For backward compatibility with versions of terminal.mod that don't
# understand terminal_input
terminal serial
fi
if terminal_output serial ; then true ; else
# For backward compatibility with versions of terminal.mod that don't
# understand terminal_output
terminal serial
fi

menuentry "Linux 2.6.32-trunk-amd64" {
insmod ext2
set root='(hd0,1)'
search --no-floppy --fs-uuid --set 9245a9e3-8ea5-4170-a19b-17d10051c107
echo Loading Linux 2.6.32-trunk-amd64 ...
linux /vmlinuz-2.6.32-trunk-amd64 root=/dev/mapper/vg0-root ro console=tty0 console=ttyS0,9600n8 quiet
echo Loading initial ramdisk ...
initrd /initrd.img-2.6.32-trunk-amd64
}



Regarding the login prompt on serial console, edit /etc/inittab:
T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100



Now your VM is configured, let's configure your KVM domain. Dump the configuration of your vm, and change the <serial> and <console> part to use a PTY (you can choose an arbitrary PTY, /dev/pts/24 here, as it seems to be redefined each time the VM is started). Other interfaces are possible, like TCP, pipe, stdio... (see the libvirt domain XML format) but I chose PTY because it can be easily attached using screen(1) and cannot be easily snooped:
# virsh dumpxml mykvm > mykvm.xml
# vi mykvm.xml
<serial type='pty'>
<source path='/dev/pts/24'/>
<target port='0'/>
</serial>
<console type='pty' tty='/dev/pts/24'>
<source path='/dev/pts/24'/>
<target port='0'/>
</console>


Then stop your VM, redefine your KVM domain and restart it:
# virsh shutdown mykvm      # or run shutdown(8) inside the VM
# virsh undefine mykvm
# virsh define mykvm.xml
# virsh start mykvm


You can attach the console using:
# virsh console mykvm

To detach, use Ctrl + $


If you attach quickly enough after starting it, you will even see the Grub menu!

Monday, May 24, 2010

Quick n' Dirty Linux WPA-PSK Wireless AP

On saturday evening, there was a party at home. One of the guests poured her glass of champagne on the ADSL modem lended by my ISP. Undoubtly it wasn't champagne-proof. I have about a week to wait before getting a new one. Fortunately I have my 3G connection but only one person can use it at a given time... and we are two at home. So I have created a very quick and dirty access-point to share my 3G connection. This post has two purpose: record how I did it and show how it eventually turned out to be really easy. Ironically, it was more difficult to configure a new wireless connection on Windows XP than creating the AP.

I am assuming you are running mac80211 wireless stack, which is standard from recent kernels 2.6.30+). You will need hostapd and ISC's DHCPd.

First set up your wireless interface as you would with any other wired interface:

# ifconfig wlan0 inet 192.168.10.1 netmask 0xffffff00 up


Next, configure /etc/hostapd/hostapd.conf:

driver=nl80211
interface=wlan0
channel=13
ssid=3g2wifi
auth_algs=1
wpa=1
wpa_passphrase=XXXXXXXX


And run it:

# hostapd /etc/hostapd/hostapd.conf


From now on, you can configure a smartphone or another computer with this wireless network and see DHCP traffic when using tcpdump -ni wlan0. You may enable debugging with hostapd's -d flag if it doesn't work.

Next step is to provide connectivity to the Internet through 3G (interface ppp0). We have to masquerade the computers behind the access-point (I assume there are no filtering rules):

# iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o ppp0 -j MASQUERADE
# sysctl net.ipv4.conf.all.forwarding=1


Here you can manually configure the IPv4 layer on another computer, setting the DNS servers to the ones provided by you 3G provider, and it should work.

But the sugar on the cake would be to have a DHCP server, to minimize manual configuration. This is straightforward. Here is my /etc/dhcp3/dhcpd.conf (note that I used Google's open DNS resolvers for example purpose):

subnet 192.168.10.0 netmask 255.255.255.0 {
range 192.168.10.20 192.168.10.30;
option routers 192.168.10.1;
option domain-name-servers 4.4.4.4, 4.4.8.8;
}


Start the DHCP server:

# dhcpd3 -cf /etc/dhcp3/dhcpd.conf wlan0


And voila! Now good luck if you have to configure a Windows 7 computer to use this connection :).

Thursday, May 20, 2010

column.sh - columizator

It has been a long time since I've last written on this blog. I am indeed very busy by my work so I don't have much time to write. Nonetheless a positive aspect of this is that I happen to write some tools for me to alleviate some tasks.

I am pretty sure that many if not all of you have already been annoyed by the output format such as vmstat(8), iostat(8), ... They are great commands because they produce very valuable information but they are often very difficult to read, especially on busy servers when you need them most, because of the misalignment. We cannot blame them because it is the Unix way of doing thing: do one thing and do it well. It's not their job to pretty print the output. The "column" script exposed here will realign the output for you.

This command acts like a standard Unix command: it may be used alone or as a filter. It takes one mandatory argument, namely a keyword that will be used to recognize the «caption» line.


jlehen@warg:~$ ./column -h
Pretty-print columns, for iostat or vmstat.
OS: All.

Usage: column [-g]

Keyword is used to identify the caption line. This may be an
awk/nawk regex (thus don't forget to escape "/").
Options:
-g Change the output to what I called "giant mode".
This is useful for iostat with an awful lot of disks.


Let's take an example. Here is a classical vmstat(8) output on a busy Solaris server:

05:26:32 kthr memory page disk faults cpu
05:26:32 r b w swap free re mf pi po fr de sr m0 m1 m2 m1 in sy cs us sy id
05:26:32 0 29 0 139019936 26258128 4776 17739 38816 8 0 0 0 4 4 4 0 29969 129465 33184 21 25 54
05:26:33 0 40 0 138993232 26235120 5240 19492 47250 39 31 0 0 7 7 7 0 30665 133160 34489 23 25 51
05:26:34 0 28 0 139002648 26230848 4878 18717 36828 16 16 0 0 3 3 3 0 31368 144148 35850 22 24 54
05:26:35 1 26 0 138978136 26211456 4046 14760 30778 8 8 0 0 4 4 4 0 26295 158433 27845 20 20 61
05:26:36 0 29 0 138980472 26212968 4469 15525 33361 23 16 0 0 4 4 4 0 25826 118810 27653 21 19 60
05:26:37 0 22 0 139004424 26227824 4208 16734 29367 8 8 0 0 4 4 4 0 24879 122903 28154 23 19 58
05:26:38 0 18 0 139010608 26232864 3958 15324 27984 0 0 0 0 2 2 2 0 22422 105347 23563 17 17 66
05:26:39 0 25 0 139022192 26237584 4076 15124 27770 31 31 0 0 2 2 2 0 25046 115101 26467 17 17 65
05:26:40 0 21 0 139035416 26248720 5205 15215 38873 0 0 0 0 1 1 1 0 26961 131668 28818 22 18 60
05:26:41 0 17 0 139014056 26239552 5132 17257 31959 0 0 0 0 0 0 0 0 23780 116586 25043 18 19 63
05:26:42 0 13 0 139028280 26250352 3661 17215 17719 23 16 0 0 3 3 3 0 22048 110885 23717 15 17 67
05:26:43 0 11 0 138993472 26220968 3796 19241 16468 0 0 0 0 0 0 0 0 21640 115744 23173 18 15 66
05:26:44 0 12 0 138972904 26199192 2607 14008 15327 8 8 0 0 2 2 2 0 20428 120489 21064 18 15 67
05:26:45 0 11 0 138973008 26194376 2122 9228 21017 23 23 0 0 5 5 5 0 24543 123981 25423 20 16 64
05:26:46 2 12 0 138977112 26194960 1929 12010 23333 0 0 0 0 1 1 1 0 32145 144927 34596 26 16 59
05:26:47 1 21 0 139032632 26240256 2361 11192 32759 16 16 0 0 2 2 2 0 36018 204602 39325 27 21 51
05:26:49 1 25 0 139043840 26245952 2628 9895 35478 24 24 0 0 3 3 3 0 32645 136756 34903 26 20 54
05:26:50 0 23 0 139043552 26246304 1942 9952 21395 0 0 0 0 0 0 0 0 24948 121252 25918 21 14 66
05:26:51 0 19 0 139053752 26249176 1949 7929 26141 0 0 0 0 6 7 7 0 24490 113321 26220 19 14 67


This is difficult to read because of the large amount of memory and swap. Let's look at the same output now filtered through column (keyword used to match caption line is "swap"):

05:26:32 kthr memory page disk faults cpu
05:26:32 r b w swap free re mf pi po fr de sr m0 m1 m2 m1 in sy cs us sy id
05:26:32 0 29 0 139019936 26258128 4776 17739 38816 8 0 0 0 4 4 4 0 29969 129465 33184 21 25 54
05:26:33 0 40 0 138993232 26235120 5240 19492 47250 39 31 0 0 7 7 7 0 30665 133160 34489 23 25 51
05:26:34 0 28 0 139002648 26230848 4878 18717 36828 16 16 0 0 3 3 3 0 31368 144148 35850 22 24 54
05:26:35 1 26 0 138978136 26211456 4046 14760 30778 8 8 0 0 4 4 4 0 26295 158433 27845 20 20 61
05:26:36 0 29 0 138980472 26212968 4469 15525 33361 23 16 0 0 4 4 4 0 25826 118810 27653 21 19 60
05:26:37 0 22 0 139004424 26227824 4208 16734 29367 8 8 0 0 4 4 4 0 24879 122903 28154 23 19 58
05:26:38 0 18 0 139010608 26232864 3958 15324 27984 0 0 0 0 2 2 2 0 22422 105347 23563 17 17 66
05:26:39 0 25 0 139022192 26237584 4076 15124 27770 31 31 0 0 2 2 2 0 25046 115101 26467 17 17 65
05:26:40 0 21 0 139035416 26248720 5205 15215 38873 0 0 0 0 1 1 1 0 26961 131668 28818 22 18 60
05:26:41 0 17 0 139014056 26239552 5132 17257 31959 0 0 0 0 0 0 0 0 23780 116586 25043 18 19 63
05:26:42 0 13 0 139028280 26250352 3661 17215 17719 23 16 0 0 3 3 3 0 22048 110885 23717 15 17 67
05:26:43 0 11 0 138993472 26220968 3796 19241 16468 0 0 0 0 0 0 0 0 21640 115744 23173 18 15 66
05:26:44 0 12 0 138972904 26199192 2607 14008 15327 8 8 0 0 2 2 2 0 20428 120489 21064 18 15 67
05:26:45 0 11 0 138973008 26194376 2122 9228 21017 23 23 0 0 5 5 5 0 24543 123981 25423 20 16 64
05:26:46 2 12 0 138977112 26194960 1929 12010 23333 0 0 0 0 1 1 1 0 32145 144927 34596 26 16 59
05:26:47 1 21 0 139032632 26240256 2361 11192 32759 16 16 0 0 2 2 2 0 36018 204602 39325 27 21 51
05:26:49 1 25 0 139043840 26245952 2628 9895 35478 24 24 0 0 3 3 3 0 32645 136756 34903 26 20 54
05:26:50 0 23 0 139043552 26246304 1942 9952 21395 0 0 0 0 0 0 0 0 24948 121252 25918 21 14 66
05:26:51 0 19 0 139053752 26249176 1949 7929 26141 0 0 0 0 6 7 7 0 24490 113321 26220 19 14 67



The -g option is meant to be used when the output is giant. For instance, when a system has a lot of disks, the iostat(8) command prints so much lines that the caption line is swept out of your terminal immediately..

Let's take a sample output of iostat(8). I won't show a lot of disks because it is worthless, but keep in mind than when there are tenths or hundreds of disks, the caption line no longer appear on the screen:

extended device statistics
device r/s w/s kr/s kw/s wait actv svc_t %w %b
md0 5.0 8.1 36.3 155.5 0.0 0.1 6.7 0 6
md1 3.0 8.1 20.2 155.5 0.0 0.1 7.3 0 6
md2 2.0 8.1 16.2 155.5 0.0 0.1 5.1 0 4
md10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md11 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md12 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md50 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md51 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md52 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md60 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md61 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
md62 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ramdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
sd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
[...]


Filtering with column -g actv (using "device" would be useless as the very first line also contains it):

extended device statistics
device:md0 r/s:5.0 w/s:8.1 kr/s:36.3 kw/s:155.5 wait:0.0 actv:0.1 svc_t:6.7 %w:0 %b:6
device:md1 r/s:3.0 w/s:8.1 kr/s:20.2 kw/s:155.5 wait:0.0 actv:0.1 svc_t:7.3 %w:0 %b:6
device:md2 r/s:2.0 w/s:8.1 kr/s:16.2 kw/s:155.5 wait:0.0 actv:0.1 svc_t:5.1 %w:0 %b:4
device:md10 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md11 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md12 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md50 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md51 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md52 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md60 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md61 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:md62 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:ramdisk1 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
device:sd0 r/s:0.0 w/s:0.0 kr/s:0.0 kw/s:0.0 wait:0.0 actv:0.0 svc_t:0.0 %w:0 %b:0
[...]


This nice script is available here.
Beware that is used /bin/sh. If you want to use it on Solaris, use /bin/ksh instead. I have yet to find a shebang line that will work on *BSD, Linux and Solaris. If you know one, contact me on < jeremie le-hen org > (up to you to but the "@" and "." where you think it best fits...).