Technicals writings from JLH

Apache mod_rewrite evilness (with dynamic vhosts and .htaccess rewrite rules)

2011-11-09T17:24:00.009+01:00

At my new $job, we have a SVN repository for the websites we are maintaining. We devised a workflow to work with it: each developer has one or more branches for himself. The merge of their features is done in the trunk. Once everything seems to work, we merge into the "preproduction" branch and finally in the "production" branch.

On the developement web server, I wanted them to be able access every branch with their web browser. Initially, there was a "svn.mywebsite.com" virtual host and each branch was accessible through an URL-path within it. Unfortuntaly, for "historical" reason, the web site doesn't work correctly if set in an URL sub-directory (and we are currently writing a new version of this website, so we actually don't want to spend time fixing it). I am therefore doomed to create a virtual host for each SVN trunk/tag/branch.

Here is the relevant part of the initial configuration I wrote:


<VirtualHost *:80>
    ServerName   svn.mywebsite.com
    ServerAlias *.svn.mywebsite.com

    DocumentRoot /var/empty

    RewriteEngine on

    RewriteCond %{HTTP_HOST} ^trunk\.svn\.mywebsite\.com$
    RewriteRule $(.*)        /home/www-data/svn/project/trunk$1       [L]

    RewriteCond %{HTTP_HOST} ^([^.]+).branches\.svn\.mywebsite\.com$
    RewriteRule $(.*)        /home/www-data/svn/project/branches%1$1  [L]

    RewriteCond %{HTTP_HOST} ^([^.]+).tags\.svn\.mywebsite\.com$
    RewriteRule $(.*)        /home/www-data/svn/project/tags%1$1      [L]

</VirtualHost>

So far it's easy and it would have worked if there wasn't the following RewriteRule in the .htaccess at the root of the project:


RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !=/favicon.ico
RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]

The problem with this rule is that if we request a "virtual URL" (which does not match a physical file in the hierarchy), index.php is called with the original URL in the query string, which results in an internal redirect within Apache.

I save you from the RewriteLog, but let's say you try: http://trunk.svn.mywebsite.com/virtual-url.

Initially the ${REQUEST_URI} is "/virtual-url". The vhost rewrite rules are applied, which redirect to the filesystem path: /home/www-data/svn/project/trunk/virtual-url.

Then we reach the per-directory (.htaccess) rewrite rules which, given the file doesn't exist, redirect to /home/www-data/svn/project/trunk/index.php with the following query string q=virtual-url.

Here is the first trap: an internal redirect is done within Apache, which restarts the rewrite rules evaluation from the beginning, with ${REQUEST_URI} set to /home/www-data/svn/project/trunk/index.php, while the ${HTTP_HOST} is still the same. So the directory will be prepended twice if we do not put a safeguard: basically checking that $REQUEST_URI doesn't contain /home/www-data/svn/project/.

But the true evilness is here: we cannot rewrite to a full filesystem path in a per-directory rewrite rule. The subsequent internal redirect will invariably think that this is an URL path, that is it will try to serve a page as if you had requested: http://trunk.svn.mywebsite.com/home/www-data/svn/project/trunk/index.php. Because of the safeguard above, the dynamic vhost magic will not apply, and Apache will try to reach this file from the vhost's DocumentRoot and you will get a 404.

The workaround for this is trick Apache into thinking the content of ${REQUEST_URI} is a full filesystem path if the latter looks like a filesystem path :-). Contrary to the per-directory rewrite rules, the vhost rewrite rules are able to redirect to a full filesystem path. So just match the whole content and redirect to it.


<VirtualHost *:80>
    ServerName   svn.mywebsite.com
    ServerAlias *.svn.mywebsite.com

    DocumentRoot /var/empty

    RewriteEngine on

    RewriteRule ^(/home/www-data/svn/project/.*)       $1             [L]

    RewriteCond %{HTTP_HOST} ^trunk\.svn\.mywebsite\.com$
    RewriteRule $(.*)        /home/www-data/svn/project/trunk$1       [L]

    RewriteCond %{HTTP_HOST} ^([^.]+).branches\.svn\.mywebsite\.com$
    RewriteRule $(.*)        /home/www-data/svn/project/branches%1$1  [L]

    RewriteCond %{HTTP_HOST} ^([^.]+).tags\.svn\.mywebsite\.com$
    RewriteRule $(.*)        /home/www-data/svn/project/tags%1$1      [L]

</VirtualHost>

The first rewrite rule matches a ${REQUEST_URI} containing the a filesystem path. This is not exactly a rewrite, this is just a trick to trigger the mod_rewrite evaluation of the substitution.

Installing full-ZFS server at OVH

2011-10-08T17:32:00.008+02:00

OVH is a french web hosting service in France, that provides dedicated servers (and many other things). It is great because they offer an infrastructure which brings you low-cost but yet professional facilities for less than 20 euros a month. They provide Linux, FreeBSD and even Solaris: you can of course ask for your server to be installed with this but the real great thing is the netboot feature that will boot the same OS as the one which is installed on your server.

The FreeBSD installation is UFS based. It is nonetheless possible to migrate in on ZFS with little wizardry. It is best the do this with a fresh installation, but is should be possible to do so as long as you use less than the half of you hard drive. However, you have to move everything into the first physical half of the hard drive (it is easier when the server has just been installed, as you just have to keep the root partition).

The procedure is the following: You move all your data in a (small) transient partition at the end of the disk. Then a ZFS partition is created at the beginning of the disk, and again move your data there. You can then destroy the transient partition and create a physical swap partition in its stead. Indeed, although FreeBSD can use a ZFS vdev as swap, it cannot dump to it. Therefore this procedure creates a real partition for swap.

Your FreeBSD is booted. Go to the OVH manager and in the "Netboot" page and select "rescue-pro". Then reboot you server, wait for a while, you should receive a mail with the root password of your netbooted server.

Once connected on it, create a partition large enough to hold all your data at the end of the hard drive. We will copy them here in order to be able to install the ZFS partition at the beginning of the disk.


rescue-bsd# fdisk /dev/ad0
******* Working on device /dev/ad0 *******
[...]

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 63, size 488397105 (238475 Meg), flag 80 (active)
        beg: cyl 1/ head 0/ sector 1;
        end: cyl 655/ head 0/ sector 63
The data for partition 2 is:

The data for partition 3 is:

The data for partition 4 is:


rescue-bsd# bsdlabel /dev/ad0s1
# /dev/ad0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a: 20971520        0    4.2BSD     4096 16384    64 
  b:  2097152 20971520      swap                    
  c: 488397105        0    unused        0     0         # "raw" part, don't edit
  d: 465328433 23068672    4.2BSD        0     0     0 

rescue-bsd# mount /dev/ad0s1a /mnt/
rescue-bsd# df -k /mnt/
Filesystem 1024-blocks   Used   Avail Capacity  Mounted on
/dev/ad0s1a    10154158 495960 8845866     5%    /mnt
rescue-bsd# umount /mnt/

We have a swap partition on /dev/ad0s1b and an empty filesystem on /dev/ad0s1d. The root partition only uses 500 MB. We are going to create a partition at the end of the disk to copy the content of it. Thus this partition must be large enough. But this partition should also large enough to hold the swap partition you want on your system eventually. In this example I want 1 GB of swap.

So let's create a 1 GB partition at the end of the disk. 1 GB is 1024*1024*1024/512 = 2097152 sectors. The disk is 488397105 sectors wide, so the partition would start at 488397105 - 2097152 = 486299953. A good practice is to align the partition to a 4 KB boundary: 486299953 % 4096 = 2353, so we will use 486299953 - 2353 = 486297600 for first sector of the partition. Given the end of the disk is 488397105, the partition size will be 488397105 - 486297600 = 2099505 sectors.


rescue-bsd# bsdlabel /dev/ad0s1 > /tmp/ad0.label
rescue-bsd# vi /tmp/ad0.label
[...]

rescue-bsd# cat /tmp/ad0.label
# /dev/ad0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a: 20971520         0    4.2BSD     4096 16384    64 
  b: 2099505  486297600    4.2BSD     4096 16384    64              
  c: 488397105        0    unused        0     0         # "raw" part, don't edit

rescue-bsd# bsdlabel -R /dev/ad0s1 /tmp/ad0.label
rescue-bsd# newfs /dev/ad0s1b 
/dev/ad0s1b: 1025.1MB (2099504 sectors) block size 16384, fragment size 2048
        using 6 cylinder groups of 183.77MB, 11761 blks, 23552 inodes.
super-block backups (for fsck -b #) at:
 160, 376512, 752864, 1129216, 1505568, 1881920

rescue-bsd# mount /dev/ad0s1b /mnt
rescue-bsd# cd /mnt
rescue-bsd# dump -0af - /dev/ad0s1a | restore -rf -
  DUMP: Date of this level 0 dump: Sat Oct  8 10:07:58 2011
  DUMP: Date of last level 0 dump: the epoch
  DUMP: Dumping /dev/ad0s1a to standard output
  DUMP: mapping (Pass I) [regular files]
  DUMP: mapping (Pass II) [directories]
  DUMP: estimated 499089 tape blocks.
  DUMP: dumping (Pass III) [directories]
  DUMP: dumping (Pass IV) [regular files]
[...]
  DUMP: finished in 131 seconds, throughput 3816 KBytes/sec
  DUMP: DUMP IS DONE
rescue-bsd# cd
rescue-bsd# umount /mnt

Now we can remove the first partition and create a big ZFS partition spanning from the beginning of the disk to the beginning of the second partition we have just created.


rescue-bsd# gpart show ad0s1
=>        0  488397105  ad0s1  BSD  (233G)
          0   20971520      1  freebsd-ufs  (10G)
   20971520  465326080         - free -  (222G)
  486297600    2099505      2  freebsd-ufs  (1.0G)

rescue-bsd# gpart delete -i 1 ad0s1
ad0s1a deleted
rescue-bsd# gpart show ad0s1
=>        0  488397105  ad0s1  BSD  (233G)
          0  486297600         - free -  (232G)
  486297600    2099505      2  freebsd-ufs  (1.0G)

rescue-bsd# gpart add -s 486297600 -t freebsd-zfs ad0s1
ad0s1a added
rescue-bsd# gpart show ad0s1
=>        0  488397105  ad0s1  BSD  (233G)
          0  486297600      1  freebsd-zfs  (232G)
  486297600    2099505      2  freebsd-ufs  (1.0G)

Now let's create the ZFS pool. But the OVH netboot only provides a read-only root filesystem, so we have to tell zpool(8) to put the cache file into /tmp (this file will be needed to import the pool at boot time). We must also to tell the ZFS layer to temporary mount the pool into /mnt, so it won't try to mount the root of the pool as /.


rescue-bsd# kldload opensolaris
rescue-bsd# kldload zfs
rescue-bsd# zpool create -o cachefile=/tmp/zpool.cache -o altroot=/mnt zroot /dev/ad0s1a
rescue-bsd# zpool export zroot

Install the various bootcodes. The MBR bootcode should already be there anyway. The ZFS bootcode is somewhat strange because it consists actually of two parts that must be written at different places (note that the first dd(1) uses /dev/ad0s1 while the second one uses /dev/ad0s1a):


rescue-bsd# gpart bootcode -b /boot/boot0 ad0
bootcode written to ad0
rescue-bsd# dd if=/boot/zfsboot of=/dev/ad0s1 count=1 bs=512
rescue-bsd# dd if=/boot/zfsboot of=/dev/ad0s1b skip=1 seek=1024 bs=512

Then re-import the pool with the same options used during its creation and create the datasets for the base filesystem (I'm using the same layout as described on the FreeBSD wiki):


rescue-bsd# zpool import -o cachefile=/tmp/zpool.cache -o altroot=/mnt zroot 
rescue-bsd# zfs set checksum=fletcher4 zroot
rescue-bsd# zfs set mountpoint=none zroot
rescue-bsd# zfs create -o mountpoint=/ zroot/rootfs
rescue-bsd# zpool set bootfs=zroot/rootfs zroot
rescue-bsd# zfs create -o compression=on -o exec=on -o setuid=off zroot/rootfs/tmp
rescue-bsd# chmod 1777 /mnt/tmp/
rescue-bsd# zfs create zroot/rootfs/usr
rescue-bsd# zfs create zroot/rootfs/usr/home
rescue-bsd# ln -s /usr/home /mnt/home
rescue-bsd# zfs create -o compression=lzjb -o setuid=off zroot/rootfs/usr/ports
rescue-bsd# zfs create -o compression=off -o exec=off -o setuid=off zroot/rootfs/usr/ports/distfiles
rescue-bsd# zfs create -o compression=off -o exec=off -o setuid=off zroot/rootfs/usr/ports/packages
rescue-bsd# zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/rootfs/usr/src
rescue-bsd# zfs create zroot/rootfs/var
rescue-bsd# zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/rootfs/var/crash
rescue-bsd# zfs create -o exec=off -o setuid=off zroot/rootfs/var/db
rescue-bsd# zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/rootfs/var/db/pkg
rescue-bsd# zfs create -o exec=off -o setuid=off zroot/rootfs/var/empty
rescue-bsd# zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/rootfs/var/log
rescue-bsd# zfs create -o compression=gzip -o exec=off -o setuid=off zroot/rootfs/var/mail
rescue-bsd# zfs create -o exec=off -o setuid=off zroot/rootfs/var/run
rescue-bsd# zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/rootfs/var/tmp
rescue-bsd# chmod 1777 /mnt/var/tmp

Note that I've activated compression on some datasets as in the FreeBSD wiki, but on a low-end box with little CPU power, I advise to turn it off.

Now let's copy our data to the ZFS partition.


rescue-bsd# mount /dev/ad0s1b /media
rescue-bsd# cd /media
rescue-bsd# find . | cpio -dump /mnt/
rescue-bsd# cd
rescue-bsd# umount /media
rescue-bsd# zfs set readonly=on zroot/rootfs/var/empty

Note that cpio(1) does not handle file flags set by chflags(8). Your system will be able to boot, but some security seatbelt won't be here until you perform an installworld.

Let's create the swap partition instead of the transient UFS filesystem:


rescue-bsd# gpart show ad0s1
=>        0  488397105  ad0s1  BSD  (233G)
          0  486297600      1  freebsd-zfs  (232G)
  486297600    2099505      2  freebsd-ufs  (1.0G)

rescue-bsd# gpart delete -i 2 ad0s1
ad0s1b deleted
rescue-bsd# gpart add -t freebsd-swap ad0s1
ad0s1b added
rescue-bsd# gpart show ad0s1
=>        0  488397105  ad0s1  BSD  (233G)
          0  486297600      1  freebsd-zfs  (232G)
  486297600    2099505      2  freebsd-swap  (1.0G)

Now we need to configure the system to be able to boot from ZFS:


rescue-bsd# echo 'zfs_load="YES"' > /mnt/boot/loader.conf
rescue-bsd# echo 'vfs.root.mountfrom="zfs:zroot/rootfs"' >> /mnt/boot/loader.conf
rescue-bsd# cp /tmp/zpool.cache /mnt/boot/zfs/
rescue-bsd# vi /mnt/etc/fstab
[...]
rescue-bsd# cat /mnt/etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ad0s1b             none            swap            sw      0       0
proc                    /proc           procfs  rw              0       0

Et voilà! You can reboot your server (don't forget to deactivate netbooting from the OVH web interface).

Configuring FreeBSD with dual console

2011-06-30T12:18:00.012+02:00

This post is short as I intend to use it more as a reminder than a full-fledged article.

As an introduction for the un-educated reader, here is a simple paste of the boot(8) manpage:


   By default, a three-stage bootstrap is employed, and control is automati-
   cally passed from the boot blocks (bootstrap stages one and two) to a
   separate third-stage bootstrap program, loader(8).  This third stage pro-
   vides more sophisticated control over the booting process than it is pos-
   sible to achieve in the boot blocks, which are constrained by occupying
   limited fixed space on a given disk or slice.

In summary: boot0 -> boot2 -> loader -> kernel

The first stage (boot0) cannot be configured, as the code as to fit in 512 bytes. It will simply use the default system console (the screen).

However the following things can be configured more or less independently:
- boot2 (stage 2);
- loader(8) (stage 3);
- the kernel;
- login(8).

Configuring boot2

boot2 is configured through /boot.config. This file contains the flags documented in boot(8), as though they were given on the boot2 prompt. Therefore if you want to see boot2 output on both your screen and the serial console, you have to put "-D" in it.


shell# cat /boot.config
-D

Configuring `loader(8)`

loader(8) is configured through /boot/loader.conf. The console variable can be set either "vidconsole", "comconsole" or "vidconsole,comconsole" to have both "comconsole,vidconsole" works too, we will see the difference later)


shell# grep ^console /boot/loader.conf
console="vidconsole,comconsole"

Configuring the kernel

/boot/loader.conf also contains variables that will set kenv variables, which will define the kernel behaviour. See this comment in /boot/defaults/loader.conf:


##############################################################
###  Kernel settings  ########################################
##############################################################

# The following boot_ variables are enabled by setting them to any value.
# Their presence in the kernel environment (see kenv(1)) has the same
# effect as setting the given boot flag (see boot(8)).

#boot_askname=""        # -a: Prompt the user for the name of the root device
#boot_cdrom=""          # -C: Attempt to mount root file system from CD-ROM
#boot_ddb=""            # -d: Instructs the kernel to start in the DDB debugger
#boot_dfltroot=""       # -r: Use the statically configured root file system
#boot_gdb=""            # -g: Selects gdb-remote mode for the kernel debugger
#boot_multicons=""      # -D: Use multiple consoles
#boot_mute=""           # -m: Mute the console
#boot_pause=""          # -p: Pause after each line during device probing
#boot_serial=""         # -h: Use serial console
#boot_single=""         # -s: Start system in single-user mode
#boot_verbose=""        # -v: Causes extra debugging information to be printed
#init_path="/sbin/init:/sbin/oinit:/sbin/init.bak:/rescue/init:/stand/sysinstall"
                      # Sets the list of init candidates
#init_shell="/bin/sh"   # The shell binary used by init(8).
#init_script=""         # Initial script to run by init(8) before chrooting.
#init_chroot=""         # Directory for init(8) to chroot into.

So basically, the kernel defaults to use the screen only, but you can override this by setting the boot_multicons variable:


shell# grep ^boot_multicons /boot/loader.conf
boot_multicons="YES"

How the whole stuff works

Actually when you configure one stage, subsequent stages will use the same settings unless configured to do differently. So in the end you just have to configure boot2.

Userland output

Contrary to the other parts, userland boot output can only be sent to one device at time. Even when configured with the above settings, the userland boot output will only appear on screen.

Actually, the kernel will pick the first entry from the console kenv variable to sent userland output to. So if you are not often behind the screen and you prefer to see the userland boot output on the serial console:


shell# grep ^console /boot/loader.conf
console="comconsole,vidconsole"

Configuring `login(8)`

Not seeing the userland boot output on one console or the other doesn't mean it is unusable. FreeBSD is configured by default to spawn a login: prompt on the screen. You can easily configure it to spawn another one one the serial console, as explained in this chapter on the handbook:


shell# grep ttyu0 /etc/ttys
ttyu0   "/usr/libexec/getty std.9600"   dialup  on secure

That's all.

Debian KVM console on a headless server

2010-06-02T18:03:00.019+02:00

At work I use a Debian KVM with an encrypted root filesystem as a workstation (our physical workstations run Windows) running on a headless server. This means that I have to use the QEMU' VNC console to enter the password for the root filesystem very early in the boot process.

Unfortunately VNC is unsecure and anyway QEMU only binds VNC on 127.0.0.1. It would be easy to create an SSH tunnel, but this is administratively prohibited here and it is cumbersome to temporarily modify sshd_config(5) each time. So I tried a Netfilter DNAT rule as a workaround but Linux' network stack contains a very annoying line of code which checks that packets destined 127.0.0.1 comes from 127.0.0.1 as well. If you see some logs like this, you have probably been biten by it too:

Jun  2 18:14:20 srv kernel: martian destination 127.0.0.1 from 10.1.2.2, dev br0

So I gave up VNC and configured the KVM domain to use the serial port like any other headless server.

Supposedly your VM is already running so we will make the changes here first. There are three things to be told to use the serial console, which are in time-order:

the bootloader (GRUB here);

the kernel;

init(8) for the login prompt.

On Debian, the first two things can be done easily through /etc/default/grub.

# Bootloader part.
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

# Kernel command-line ("quiet" has no matter in our business):
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,9600n8 quiet"

Then regen the grub.cfg:

# upgrade-grub

If you do not use Debian, here is the relevant part of the generated /boot/grub/grub.cfg:

serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1
if terminal_input serial ; then true ; else
  # For backward compatibility with versions of terminal.mod that don't
  # understand terminal_input
  terminal serial
fi
if terminal_output serial ; then true ; else
  # For backward compatibility with versions of terminal.mod that don't
  # understand terminal_output
  terminal serial
fi

menuentry "Linux 2.6.32-trunk-amd64" {
        insmod ext2
        set root='(hd0,1)'
        search --no-floppy --fs-uuid --set 9245a9e3-8ea5-4170-a19b-17d10051c107
        echo    Loading Linux 2.6.32-trunk-amd64 ...
        linux   /vmlinuz-2.6.32-trunk-amd64 root=/dev/mapper/vg0-root ro  console=tty0 console=ttyS0,9600n8 quiet
        echo    Loading initial ramdisk ...
        initrd  /initrd.img-2.6.32-trunk-amd64
}

Regarding the login prompt on serial console, edit /etc/inittab:

T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100

Now your VM is configured, let's configure your KVM domain. Dump the configuration of your vm, and change the <serial> and <console> part to use a PTY (you can choose an arbitrary PTY, /dev/pts/24 here, as it seems to be redefined each time the VM is started). Other interfaces are possible, like TCP, pipe, stdio... (see the libvirt domain XML format) but I chose PTY because it can be easily attached using screen(1) and cannot be easily snooped:

# virsh dumpxml mykvm > mykvm.xml
# vi mykvm.xml
   <serial type='pty'>
     <source path='/dev/pts/24'/>
     <target port='0'/>
   </serial>
   <console type='pty' tty='/dev/pts/24'>
     <source path='/dev/pts/24'/>
     <target port='0'/>
   </console>

Then stop your VM, redefine your KVM domain and restart it:

# virsh shutdown mykvm      # or run shutdown(8) inside the VM
# virsh undefine mykvm
# virsh define mykvm.xml
# virsh start mykvm

You can attach the console using:

# virsh console mykvm

To detach, use Ctrl + $

If you attach quickly enough after starting it, you will even see the Grub menu!

Quick n' Dirty Linux WPA-PSK Wireless AP

2010-05-24T23:19:00.007+02:00

On saturday evening, there was a party at home. One of the guests poured her glass of champagne on the ADSL modem lended by my ISP. Undoubtly it wasn't champagne-proof. I have about a week to wait before getting a new one. Fortunately I have my 3G connection but only one person can use it at a given time... and we are two at home. So I have created a very quick and dirty access-point to share my 3G connection. This post has two purpose: record how I did it and show how it eventually turned out to be really easy. Ironically, it was more difficult to configure a new wireless connection on Windows XP than creating the AP.

I am assuming you are running mac80211 wireless stack, which is standard from recent kernels 2.6.30+). You will need hostapd and ISC's DHCPd.

First set up your wireless interface as you would with any other wired interface:


# ifconfig wlan0 inet 192.168.10.1 netmask 0xffffff00 up

Next, configure /etc/hostapd/hostapd.conf:


driver=nl80211
interface=wlan0
channel=13
ssid=3g2wifi
auth_algs=1
wpa=1
wpa_passphrase=XXXXXXXX

And run it:


# hostapd /etc/hostapd/hostapd.conf

From now on, you can configure a smartphone or another computer with this wireless network and see DHCP traffic when using tcpdump -ni wlan0. You may enable debugging with hostapd's -d flag if it doesn't work.

Next step is to provide connectivity to the Internet through 3G (interface ppp0). We have to masquerade the computers behind the access-point (I assume there are no filtering rules):


# iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o ppp0 -j MASQUERADE
# sysctl net.ipv4.conf.all.forwarding=1

Here you can manually configure the IPv4 layer on another computer, setting the DNS servers to the ones provided by you 3G provider, and it should work.

But the sugar on the cake would be to have a DHCP server, to minimize manual configuration. This is straightforward. Here is my /etc/dhcp3/dhcpd.conf (note that I used Google's open DNS resolvers for example purpose):


subnet 192.168.10.0 netmask 255.255.255.0 {
  range 192.168.10.20 192.168.10.30;
  option routers 192.168.10.1;
  option domain-name-servers 4.4.4.4, 4.4.8.8;
}

Start the DHCP server:


# dhcpd3 -cf /etc/dhcp3/dhcpd.conf wlan0

And voila! Now good luck if you have to configure a Windows 7 computer to use this connection :).

column.sh - columizator

2010-05-20T18:45:00.011+02:00

It has been a long time since I've last written on this blog. I am indeed very busy by my work so I don't have much time to write. Nonetheless a positive aspect of this is that I happen to write some tools for me to alleviate some tasks.

I am pretty sure that many if not all of you have already been annoyed by the output format such as vmstat(8), iostat(8), ... They are great commands because they produce very valuable information but they are often very difficult to read, especially on busy servers when you need them most, because of the misalignment. We cannot blame them because it is the Unix way of doing thing: do one thing and do it well. It's not their job to pretty print the output. The "column" script exposed here will realign the output for you.

This command acts like a standard Unix command: it may be used alone or as a filter. It takes one mandatory argument, namely a keyword that will be used to recognize the «caption» line.


jlehen@warg:~$ ./column -h
Pretty-print columns, for iostat or vmstat.
OS: All.

Usage: column [-g] 

Keyword is used to identify the caption line.  This may be an
awk/nawk regex (thus don't forget to escape "/").
Options:
 -g     Change the output to what I called "giant mode".
        This is useful for iostat with an awful lot of disks.

Let's take an example. Here is a classical vmstat(8) output on a busy Solaris server:


05:26:32   kthr      memory            page            disk          faults      cpu
05:26:32   r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 m1   in   sy   cs us sy id
05:26:32   0 29 0 139019936 26258128 4776 17739 38816 8 0 0 0 4 4 4 0 29969 129465 33184 21 25 54
05:26:33   0 40 0 138993232 26235120 5240 19492 47250 39 31 0 0 7 7 7 0 30665 133160 34489 23 25 51
05:26:34   0 28 0 139002648 26230848 4878 18717 36828 16 16 0 0 3 3 3 0 31368 144148 35850 22 24 54
05:26:35   1 26 0 138978136 26211456 4046 14760 30778 8 8 0 0 4 4 4 0 26295 158433 27845 20 20 61
05:26:36   0 29 0 138980472 26212968 4469 15525 33361 23 16 0 0 4 4 4 0 25826 118810 27653 21 19 60
05:26:37   0 22 0 139004424 26227824 4208 16734 29367 8 8 0 0 4 4 4 0 24879 122903 28154 23 19 58
05:26:38   0 18 0 139010608 26232864 3958 15324 27984 0 0 0 0 2 2 2 0 22422 105347 23563 17 17 66
05:26:39   0 25 0 139022192 26237584 4076 15124 27770 31 31 0 0 2 2 2 0 25046 115101 26467 17 17 65
05:26:40   0 21 0 139035416 26248720 5205 15215 38873 0 0 0 0 1 1 1 0 26961 131668 28818 22 18 60
05:26:41   0 17 0 139014056 26239552 5132 17257 31959 0 0 0 0 0 0 0 0 23780 116586 25043 18 19 63
05:26:42   0 13 0 139028280 26250352 3661 17215 17719 23 16 0 0 3 3 3 0 22048 110885 23717 15 17 67
05:26:43   0 11 0 138993472 26220968 3796 19241 16468 0 0 0 0 0 0 0 0 21640 115744 23173 18 15 66
05:26:44   0 12 0 138972904 26199192 2607 14008 15327 8 8 0 0 2 2 2 0 20428 120489 21064 18 15 67
05:26:45   0 11 0 138973008 26194376 2122 9228 21017 23 23 0 0 5 5 5 0 24543 123981 25423 20 16 64
05:26:46   2 12 0 138977112 26194960 1929 12010 23333 0 0 0 0 1 1 1 0 32145 144927 34596 26 16 59
05:26:47   1 21 0 139032632 26240256 2361 11192 32759 16 16 0 0 2 2 2 0 36018 204602 39325 27 21 51
05:26:49   1 25 0 139043840 26245952 2628 9895 35478 24 24 0 0 3 3 3 0 32645 136756 34903 26 20 54
05:26:50   0 23 0 139043552 26246304 1942 9952 21395 0 0 0 0 0 0 0 0 24948 121252 25918 21 14 66
05:26:51   0 19 0 139053752 26249176 1949 7929 26141 0 0 0 0 6 7 7 0 24490 113321 26220 19 14 67

This is difficult to read because of the large amount of memory and swap. Let's look at the same output now filtered through column (keyword used to match caption line is "swap"):


05:26:32   kthr      memory            page            disk          faults      cpu
05:26:32 r b  w swap      free     re   mf    pi    po  fr  de sr m0   m1  m2  m1 in    sy     cs    us sy id
05:26:32 0 29 0 139019936 26258128 4776 17739 38816 8   0   0  0  4    4   4   0  29969 129465 33184 21 25 54
05:26:33 0 40 0 138993232 26235120 5240 19492 47250 39  31  0  0  7    7   7   0  30665 133160 34489 23 25 51
05:26:34 0 28 0 139002648 26230848 4878 18717 36828 16  16  0  0  3    3   3   0  31368 144148 35850 22 24 54
05:26:35 1 26 0 138978136 26211456 4046 14760 30778 8   8   0  0  4    4   4   0  26295 158433 27845 20 20 61
05:26:36 0 29 0 138980472 26212968 4469 15525 33361 23  16  0  0  4    4   4   0  25826 118810 27653 21 19 60
05:26:37 0 22 0 139004424 26227824 4208 16734 29367 8   8   0  0  4    4   4   0  24879 122903 28154 23 19 58
05:26:38 0 18 0 139010608 26232864 3958 15324 27984 0   0   0  0  2    2   2   0  22422 105347 23563 17 17 66
05:26:39 0 25 0 139022192 26237584 4076 15124 27770 31  31  0  0  2    2   2   0  25046 115101 26467 17 17 65
05:26:40 0 21 0 139035416 26248720 5205 15215 38873 0   0   0  0  1    1   1   0  26961 131668 28818 22 18 60
05:26:41 0 17 0 139014056 26239552 5132 17257 31959 0   0   0  0  0    0   0   0  23780 116586 25043 18 19 63
05:26:42 0 13 0 139028280 26250352 3661 17215 17719 23  16  0  0  3    3   3   0  22048 110885 23717 15 17 67
05:26:43 0 11 0 138993472 26220968 3796 19241 16468 0   0   0  0  0    0   0   0  21640 115744 23173 18 15 66
05:26:44 0 12 0 138972904 26199192 2607 14008 15327 8   8   0  0  2    2   2   0  20428 120489 21064 18 15 67
05:26:45 0 11 0 138973008 26194376 2122 9228  21017 23  23  0  0  5    5   5   0  24543 123981 25423 20 16 64
05:26:46 2 12 0 138977112 26194960 1929 12010 23333 0   0   0  0  1    1   1   0  32145 144927 34596 26 16 59
05:26:47 1 21 0 139032632 26240256 2361 11192 32759 16  16  0  0  2    2   2   0  36018 204602 39325 27 21 51
05:26:49 1 25 0 139043840 26245952 2628 9895  35478 24  24  0  0  3    3   3   0  32645 136756 34903 26 20 54
05:26:50 0 23 0 139043552 26246304 1942 9952  21395 0   0   0  0  0    0   0   0  24948 121252 25918 21 14 66
05:26:51 0 19 0 139053752 26249176 1949 7929  26141 0   0   0  0  6    7   7   0  24490 113321 26220 19 14 67

The -g option is meant to be used when the output is giant. For instance, when a system has a lot of disks, the iostat(8) command prints so much lines that the caption line is swept out of your terminal immediately..

Let's take a sample output of iostat(8). I won't show a lot of disks because it is worthless, but keep in mind than when there are tenths or hundreds of disks, the caption line no longer appear on the screen:


                  extended device statistics
device      r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b
md0         5.0    8.1   36.3  155.5  0.0  0.1    6.7   0   6
md1         3.0    8.1   20.2  155.5  0.0  0.1    7.3   0   6
md2         2.0    8.1   16.2  155.5  0.0  0.1    5.1   0   4
md10        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
md11        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
md12        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
md50        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
md51        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
md52        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
md60        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
md61        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
md62        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
ramdisk1    0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
sd0         0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
[...]

Filtering with column -g actv (using "device" would be useless as the very first line also contains it):


                  extended device statistics
device:md0      r/s:5.0  w/s:8.1 kr/s:36.3  kw/s:155.5 wait:0.0 actv:0.1 svc_t:6.7   %w:0 %b:6
device:md1      r/s:3.0  w/s:8.1 kr/s:20.2  kw/s:155.5 wait:0.0 actv:0.1 svc_t:7.3   %w:0 %b:6
device:md2      r/s:2.0  w/s:8.1 kr/s:16.2  kw/s:155.5 wait:0.0 actv:0.1 svc_t:5.1   %w:0 %b:4
device:md10     r/s:0.0  w/s:0.0 kr/s:0.0   kw/s:0.0   wait:0.0 actv:0.0 svc_t:0.0   %w:0 %b:0
device:md11     r/s:0.0  w/s:0.0 kr/s:0.0   kw/s:0.0   wait:0.0 actv:0.0 svc_t:0.0   %w:0 %b:0
device:md12     r/s:0.0  w/s:0.0 kr/s:0.0   kw/s:0.0   wait:0.0 actv:0.0 svc_t:0.0   %w:0 %b:0
device:md50     r/s:0.0  w/s:0.0 kr/s:0.0   kw/s:0.0   wait:0.0 actv:0.0 svc_t:0.0   %w:0 %b:0
device:md51     r/s:0.0  w/s:0.0 kr/s:0.0   kw/s:0.0   wait:0.0 actv:0.0 svc_t:0.0   %w:0 %b:0
device:md52     r/s:0.0  w/s:0.0 kr/s:0.0   kw/s:0.0   wait:0.0 actv:0.0 svc_t:0.0   %w:0 %b:0
device:md60     r/s:0.0  w/s:0.0 kr/s:0.0   kw/s:0.0   wait:0.0 actv:0.0 svc_t:0.0   %w:0 %b:0
device:md61     r/s:0.0  w/s:0.0 kr/s:0.0   kw/s:0.0   wait:0.0 actv:0.0 svc_t:0.0   %w:0 %b:0
device:md62     r/s:0.0  w/s:0.0 kr/s:0.0   kw/s:0.0   wait:0.0 actv:0.0 svc_t:0.0   %w:0 %b:0
device:ramdisk1 r/s:0.0  w/s:0.0 kr/s:0.0   kw/s:0.0   wait:0.0 actv:0.0 svc_t:0.0   %w:0 %b:0
device:sd0      r/s:0.0  w/s:0.0 kr/s:0.0   kw/s:0.0   wait:0.0 actv:0.0 svc_t:0.0   %w:0 %b:0
[...]

This nice script is available here.
Beware that is used /bin/sh. If you want to use it on Solaris, use /bin/ksh instead. I have yet to find a shebang line that will work on *BSD, Linux and Solaris. If you know one, contact me on < jeremie le-hen org > (up to you to but the "@" and "." where you think it best fits...).

Bash prompt trick: cheap emulation of tcsh's pwd trailing component

2009-02-18T12:40:00.016+01:00

I started Unix with Linux where Bash has always been the default shell, at least in my own reckoning. Hence I've always been using Bash as I never found the necessity nor the motivation to really switch to another shell.

When FreeBSD turned to be my favorite operating system, I had a chance to fiddle with tcsh. One thing I really liked was the "%c" prompt sequence, a.k.a. "trailing component of the current working directory". Basically, "%c02" in "/home/jlh/a/b" expands to something like "/<2>/a/b". Since then, I've always missed this sequence in Bash, as "\w" often expands to something far too long and "\W" is often not enough.

Therefore I implemented a function to mimic this behaviour in tcsh. I devised it to only use builtins so as to be cheap (read inexpensive, not lousy). Indeed, I think it would be really stupid to spawn many processes each time Enter is hit in my shell. Here it is:


traildir() {
    local n=$1 dir=$2
    local sl tildedir homelen shifted  traildir

    sl=/
    tildedir=${dir#$HOME}
    if ! [[ "$tildedir" == "$dir" ]]; then
            tildedir="~$tildedir"
            sl=
    fi

    set -- ${HOME//\// }
    homelen=$#

    shifted=0
    set -- ${tildedir//\// }
    if [[ $# -gt $n ]]; then
            shifted=$(($# - $n))
            shift $shifted
            [[ -z "$sl" ]] && shifted=$(($shifted + $homelen - 1))
            traildir="<$shifted>/$@"
    else
            traildir="$sl$@"
    fi

    echo ${traildir// /\/}
}

Then, you just have to set, for example:
TRAILPWD=2 export PS1='\u@\h:$(traildir $TRAILPWD $PWD) \#\$'

And that's it! Ok it still forks a process each time you hit enter, but it's not possible to achieve it less expensively with Bash anyway. Note that the example above uses a neat trick: if you need more trailing components in your prompt for your current task, just set TRAILPWD to the desired value.

Addendum on 2009/06/22

A smaller but less resilient version of this prompt is:
export PS1='\u@\h:$(set -- ${PWD//\// }; n=$(($# - $TRAILPWD)); s=; [[ $n -le 0 ]] && s=/ || shift $n; d="$@"; echo $s${d// /\/}) \#\$'

One advantage of this one is that is only relies on an environment variable, thus is inherited across forks. This is useful for instance if you call bash through sudo(8) and you wish to use the same prompt. This is impossible with the function-based prompt.

Get rid of Vodafone Mobile Connect Card driver for Linux

2008-12-10T14:52:00.004+01:00

Back in August, I explained why I switched from shipped Xandros to the Debian Eee blend. I bought my EeePC with a 3G USB modem from Huawei:
Bus 001 Device 003: ID 12d1:1003 Huawei Technologies Co., Ltd. E220 HSDPA Modem / E270 HSDPA/HSUPA Modem
Thanks to Vodafone Mobile Connect Card driver for Linux, I could use it very easily. But compared to the Asus dialer, it is way slow! Between the time I click on the icon and the time I am connected, there is nearly two minutes because EeePC is not powerful enough to process quickly this bloated Python software.

While inspecting this program, you will notice that it uses wvdial behind the scene:
jlh 11626 11579 0 15:22 pts/5 00:00:00 /opt/vmc/bin/wvdial --config /tmp/VMC_uJJ0r/VMCYy5LXzwvdial.conf connect
Just copy the configuration file to wvdial.conf:
[Dialer Defaults] Phone = *99***1# Username = slsfr Password = slsfr Stupid Mode = 1 Dial Command = ATDT Check Def Route = on Dial Attempts = 3 [Dialer connect] Modem = /dev/ttyUSB0 Baud = 460800 Init2 = ATZ Init3 = ATQ0 V1 E0 S0=0 &C1 &D2 +FCLASS=0 Init4 = AT+CGDCONT=1,"IP","slsfr" ISDN = 0 Modem Type = Analog Modem

But running wvdial with this configuration will only work if you have already entered the PIN code with Vodafone Mobile Connect Card driver for Linux. I needed to find a way to automate this. I quickly googled for a solution without luck, so I devised a way to do it myself.

What I basically did was to use strace(1) on Vodafone Mobile Connect Card driver for Linux just when I entered my PIN code. Then I looked for the PIN code itself in the output file and watched about so I could figure out the dialog with the USB modem to unlock it.

Let's say my PIN code is 5678. First check where my PIN code is used:
jlh@r2d2:~$ grep -n 5678 strace.vodafone 68172:12333 write(29, "AT+CPIN=5678\r\n"..., 14) = 14

Line 68172. The line is sent on file descriptor 29. Let's find where this file descriptor is opened before line 68172. It may be recycled during the execution of the program, so only take the last one:
jlh@r2d2:~$ cat -n strace.vodafone | head -n 68172 | grep 'open.*= 29' | tail -n 1 66643 12333 open("/dev/ttyUSB1", O_RDWR|O_NOCTTY|O_NONBLOCK|O_LARGEFILE) = 29

And where it's closed:
jlh@r2d2:~$ cat -n strace.vodafone | tail -n +66643 | grep 'close(29' 74856 12333 close(29) = 0

We just have to get read(2) and write(2) calls on file descriptor 29 between line 66643 and line 74856:
jlh@r2d2:~$ sed -n '66643,74856{ /$read\|write$(29/p }' strace.vodafone 12333 write(29, "AT+CSQ\r\n"..., 8) = 8 12333 read(29, "AT+CSQ\r\r\n+CME ERROR: SIM PIN requ"..., 8192) = 39 12333 write(29, "ATZ\r\n"..., 5) = 5 12333 read(29, "ATZ\r"..., 8192) = 4 12333 read(29, "\r\nOK\r\n"..., 8192) = 6 12333 write(29, "ATE0\r\n"..., 6) = 6 12333 read(29, "ATE0\r\r\nOK\r\n"..., 8192) = 11 12333 write(29, "AT+CPIN?\r\n"..., 10) = 10 12333 read(29, "\r\n+CPIN: SIM PIN\r\n\r\nOK\r\n"..., 8192) = 24 12333 read(29, "\r\n^BOOT:12633134,0,0,0,64\r\n"..., 8192) = 27 12333 write(29, "AT+CPIN=5678\r\n"..., 14) = 14 12333 read(29, "\r\nOK\r\n"..., 8192) = 6 12333 read(29, "\r\n^SIMST:1\r\n"..., 8192) = 12 12333 read(29, "\r\n^SRVST:2\r\n"..., 8192) = 12 12333 read(29, "\r\n^RSSI:13\r\n"..., 8192) = 12 12333 read(29, "\r\n^BOOT:12633134,0,0,0,64\r\n"..., 8192) = 27 ...

The important thing is to send the PIN, so we will just stop after sending the PIN (if it doesn't work, you can still try to add a few lines). Let's translate this to a chat(8) script:
ECHO ON TIMEOUT 1 '' AT+CSQ 'SIM PIN required' ATZ OK ATE0 OK AT+CPIN? OK AT+CGSN OK AT+CPIN=5678

Then, just before running wvdial with the stolen configuration file, just use:
jlh@r2d2:~$ /usr/sbin/chat -f pin.chat < /dev/ttyUSB0 > /dev/ttyUSB0 [...]

Note that if you try to rerun this chat(8) script, it won't work because the modem won't return what is expected. Given there is no way to implement branches in chat(8), I used a very small timeout so it will exit very quickly if the modem is already unlocked.

Global Offset Table, why is it needed?

2008-09-26T12:33:00.005+02:00

In ELF land, GOT stands for Global Offset Table. It is a very important component of Position Independant Code (PIC), used in shared libraries. In this article, we try to understand why it is needed.

Preface

Understanding linking and loading is difficult. At least it was for me because I mainly read documentation, without writing source which usually helps to stick things in mind. If you are really interested in linking and loading, I can only advice you to read the following documentation:

John R Levine's "Linkers and Loaders" book (ISBN 1-55860-496-0). This is the best documentation you can find on this topic. You can find the first chapter here which is a very good reading to understand the history of linkers.

Ian Lance Taylor's "Linkers" blog entries contains a overall explanation of the linking process.

Also, have a look in the bookmarks on my website there are a couple of interesting pointers too in "Code"/"Linkers/Loaders".

So, why GOT?

Quote from Wikipedia in the Linker article:

In computer science, a linker or link editor is a program that takes one or more objects generated by a compiler and assembles them into a single executable program.

As you may already know when linking binary objects together into a program, the linker has to handle relocations. Relocations are basically hints left by the assembler so the linker knows where there are addresses to fix once everything has been put together. Let's take a simple example to illustrate the need of relocations. Imagine that i is a global variable defined in a source file and you assign it a value in another source file: i = 1. On i386, the compiler translates this in movl $1, i. When the assembler processes this, it doesn't know the actual address of i since it can see only one file at time, so it just set the address to zero; on the other hand the linker sees all files at once and can figure out the final value of a symbol (or, address of a variable). So in this case, the assembler generates a relocation informing the linker to put the value of symbol "i" at such or such offset in the code section, once every binary objects have been combined and the final address of all symbols is known. I won't go futher into the details about relocations, this deserves another blog article I think. Just keep in mind that most types of relocation are used to "fix" the code of the library. And this is not a problem when linking multiple objects files (including static libraries) together into a program.

Contrariwise to static libraries, shared libraries are dynamically linked, which means that the linker's job is defered to runtime and is handled by the so-called dynamic linker. By definition, shared libraries code is shared among multiple processes: thanks to virtual memory, the same physical memory pages holding the library code are mapped into the address space of every processes that need it. The benefit is savings of physical memory. Thus the shared code must obviously stay the same and should not be fixed directly to suit one particular process memory layout or the other.

The difficulty comes from the fact we don't know where the dynamic linker will decide to load the library in the process memory as it depends on a number of factors such as executable size, number and size of libraries loaded previously, not to mention that actual load address might randomized for security reasons on recent systems. Still when the library code refers to a global variable, its address has to be stashed somewhere so the code can access it.

As we saw in the relocation example, this is normally done directly by fixing the code. In order to avoid modifying shared libraries code at runtime and losing memory sharing, we have to make the runtime relocations point to the process private data. This is done by asking the compiler to generate Position-Independant Code (PIC). The idea is simple: instead of having the final absolute address of a symbol in the code, the code refers to a particular entry of a private array containing this address. At runtime, the dynamic linker will stuff this array with the actual symbols value so the same code will access addresses proper to each process. The sharp minded reader may have noticed that this only moves the problem on step further because the address of the private array itself is still unknown. This is resolved by a small calculation. Given that the library is loaded as a single unit in the memory, the distance between the code and the private array is known by the compiler. The code just have to get the address of the current instruction and the address of the private array can be figured out.

Ladies and gentlemen, this private array of addresses is what we call the Global Offset Table, or GOT.

It is actually perfectly valid to create a non-PIC shared library. It will work. But under the hood, the dynamic linker will probably patch most of the code pages of the library which will be duplicated by the copy-on-write mechanism of the virtual memory. Besides wasting memory, it may take more time to start because if a symbol is used thousands time along the code, the dynamic linker will have thousands relocations pointing into the code section to process at startup. On the other hand, with PIC there is one GOT entry per symbol so there is only one relocation to perform. But to be honest this is more a tradeoff than a pure win because using GOT implies using a more expensive function prologue to compute the GOT address and an additional indirection for access through the GOT.

Shared libraries all have their own GOT, and the program would have his if it was compiled with PIC as well (this is called Position Independant Executable by the way). Actually in any shared library, the GOT is used for every data accessed with an absolute address in the code, which includes static variables and global variables defined in the library itself.

For static variables there is no need for a GOT entry: getting the address is straightforward because the distance between the GOT and the variable itself is known, being in the same loadable unit, so the code can directly access it.

One might think this is the same for global variables defined within the library itself. But this is without counting that ELF allows the executable or a previously shared library to redefine a global variable of a shared library. For example, you can perfectly define a stdout variable in your program despite it is already defined in libc, but keep in mind that your variable will also be used by the libc functions referring to stdout such as printf(3) so be careful that it points to a valid FILE object. In short, all global variables used by the shared library unconditionally have a GOT entry.

I hope you better understand what is GOT now :).

Debian on Asus EeePC 701 with Huawei USB modem from SFR or Vodafone

2008-08-19T18:46:00.014+02:00

A few month ago, I bought a neat bundle from SFR, a french mobile operator, containing Asus EeePC 701 and a subscription to Internet through 2G/3G/3G+. The connection is achieved thanks to an "E220 HSDPA Modem" USB key from Huawei Technologies. All this stuff has been working very well out of the box with the pre-installed Xandros-based distribution (based on Debian).

All I wanted from this netbook was to be able to surf the web and open terminals without wasting time administer the beast. And well, standard EeePC distribution achieves this very well, and use OpenOffice as a bonus. Of course, I was lacking some stuff like gcc, mplayer, screen... That's why I harvested a few unofficial package repositories to cram my sources.list(5). There was obviously some conflicts between the repositories, but I really didn't care (although usually I'm very keen to make my package manager happy): I had a handful of configuration files backup'd on a USB key and in case of unrecoverable failure within the package system, I just had to restore the original state by pressing F9 at startup (EeePC is shipped with a cunning disk setup: two partitions, the first one containing the original system and the second empty one being mounted on top of the former using unionfs, so restoring the system basically means blanking the second partition).

But as time went on, one thing was more and more upsetting me: no package updates from Asus. As you may already know, this EeePC is also shipped out-of-the-box with remote root exploit (through Samba)... This was very annoying for me because I sometimes connect to other boxes using SSH, so one could hack my EeePC to steal my passwords or perform even more subtle things. So I turned off everything I could because the kernel provided by Xandros doesn't contain IPTables. But honestly, I was still worry about security.

I finally decided to spare some time to install an other, more up to date, distribution when I noticed that I couldn't use Firefox 3 because most of the required libraries were not available. A friend of mine had tried Ubuntu EEE or EEE Ubuntu, whatever. At first, I thought it was a good choice because it could fit both the low administration and up-to-date-ness requirements. But he quickly told me that Ubuntu was far more too memory hungry. Moreover, I don't like these kind of bloated distributions; they somewhat remind me Windows where everything is done behind the scene without giving me any choice unless I really dig deep to understand how things work together. So I forgot Ubuntu and kept on with Xandros

Then I read that Asus was working with Debian in order to maybe replace Xandros on EeePC some day. This caught all my attention as this implied that Debian EeePC support should be very good. What finally decided me to give Debian Eee a shot, despite my disappointing experience with Debian on my girlfriend's laptop last year, was this post from the Debian Eee PC Team, which looked quite encouraging. Additionally, Debian is one of the cleanest distribution; or rather I should say this is one of the less messy ;-) (hey, it's Linux!).

I fetched the Debian Eee's WPA Installer and spread it on a USB key. I could install Debian flawlessly through my WPA/TKIP router. I went for a 256 MB swap partition and all the remaining space as one big partition, and asked for a desktop installation, hoping to find again the regretted Xandros' usability.

And I am very happy. Given this is based on Debian testing, all packages are fairly up to date: Firefox 3 is here! This is a great improvement because it can achieve true full screen like Opera; it's damn important because EeePC 701's screen is really small for web surfing. All devices seems to be supported, although I haven't tested all of them; some shortcut keys are working (contrast keys for example), but volume keys don't. But honestly it's not a big deal compared to what I won and hopefully it will be resolved soon.

The "hardest" task was to make the Huawey modem USB key work. The current kernel (2.6.25...something) is supposed to support it, but anyway you have to somehow manage to enter the PIN code because there is a SIM card in it. Fortunately, the Vodafone Mobile Connect Card driver for Linux (wow, what a name!) handle it perfectly: just beware to download the i386 installer, which is not available as a package at time of writing. You just have run it and tell that your user must belong to the "vmc" group. Then don't forget to logout so as to be in the "vmc" group effectively and run the "vodaphone-mobile-connect-card-driver-for-linux" (I'm not kidding). Edit the profile, and change username, password and APN host to "slsfr" and the DNS servers to "172.20.2.10" and "172.20.2.39" as noted in this french forum post. And voila! You can connect to Internet over 3G+ and even read the SMS and directory stored on the SIM card!

Ok, it's not a package and it spreads some files but... it works! And if you ever want to remove it, you could still follow the installation script to know what as been copied (/etc/udev/rules/ and /usr/bin mainly if I recall correctly). The graphical interface is heavy and I would have preferred a neat command-line tool, but I won't complain more. In open-source, if you want it, just code it! :-)

In summary, if you're fed up with your Xandros, go for Debian/Eee! (It's hard to say when you are a BSD guy ;p.)

Chicken and egg problem with Propolice in runtime linker/loader

2008-06-23T10:48:00.016+02:00

Some background first: Back in 2006, I was frustrated because FreeBSD was somewhat lagging behind other open-source operating systems in term of integrated security features. One of them is a GCC extension originally named Propolice or SSP for Stack Smashing Protection. As its name lets sound, it protects (very efficiently) against stack based buffer overflows. Historically Propolice has been developed by Hiroaki Etoh at IBM for gcc-2.95.3 and then gcc-3.4.4 as an external patch, but it has now been included in the mainstream, starting at gcc-4.1. The patch to integrate Propolice in FreeBSD has been existing for more than two years on my website, but then FreeBSD only provided gcc-3.4.4 and heavily patching a contributed software is ruled out by policy, so it couldn't be committed in FreeBSD-6. I missed the FreeBSD-7 window for various reasons, and now I'm working to get it committed to FreeBSD-8 (aka CURRENT).

How does Propolice work? The compiler identifies functions that might be vulnerable (containing a stack based buffer) and during their prologue, pushes a one-word canary between the return address stored in the stack and the local variables. In the function's epilogue, the canary is checked against its original value and if it has changed then a buffer overflow occurred and the program is aborted. The canary is initially in the BSS segment but is initialized to a random value by a function called during the program startup (namely, a constructor). Both the canary and the initializer function are provided in FreeBSD's libc.

When I sent the patch for review back in april, Antoine Brodin noticed that when build world is performed with -fstack-protector-all (which makes GCC to protect all functions instead of only those containing a local buffer), it breaks the whole system. There were actually various problems, such as the
initializer function being protected itself: during its prologue the canary was equal to zero but during the epilogue its value had been set to a random value meanwhile so obviously the saved value did't match... This problem has been resolved quickly. The nasty problem lay in the runtime loader (aka rtld-elf): once it was installed, all programs would fail with SIGSEGV.

When a dynamically-linked program is run, the kernel always transfers control to rtld behing the scene, instead of the actual program. The purpose is to do runtime linking of libraries needed by the program, which includes resolving symbols and performing relocations, before actually transfering control to it. So I've recompiled rtld without SSP, but it was still crashing. I've narrowed down the segfault to a call mmap(2) which turned out to be the first call into libc, against which rtld was statically linked. One of the very first thing rtld has to do is to relocate itself, mainly to be able to access global data which are addressed through GOT (Global Offset Table). This was the very problem. Given that all libc functions were protected with Propolice, mmap(2)'s prologue tried to push the canary, which is accessed through the «__stack_chk_guard» global symbol. This means it used a pointer from the GOT, which had not been initialized at this point.

As an additional note (and a reminder for me ;p), I came to thinking that the problem could also arise in the canary initializer which stands in rtld's .init section. After some thinking, I realized that usual .init and .fini sections were handled by rtld itself, so rtld's ones are actually never run I think.

Obviously rtld must been compiled without SSP. As a temporary solution, libc is not allowed to be compiled with -fstack-protector-all. I think a better solution would be to create a librtld containing symbols required by rtld and compiled without SSP.

Sharp minds have certainly understood that if the original patch worked without -fstack-protector-all it was just a matter of chance because no functions during relocation of rtld's GOT entries had been elected by GCC to be protected.

Sketching Subversion

2008-03-12T16:15:00.007+01:00

Subversion is easy to grasp for long-time CVS users, since it is conceptually very similar. The intent when creating Subversion was simply to create "a better CVS" as they state on their frontpage. I think they succeeded with regard to this.

Revisions
First, each commit is atomic and revision numbers are incremented for each commit repository-wise, whereas in CVS revision numbers are incremented for each commit file-wise. This means each revision number represent a state of the filesystem tree. With CVS, if you want to get the diff of a commit, you could only approximate it with dates. In SVN, you simply ask for the diff between two revisions.

The puzzling side of this is that two successive revisions of a file may not have successive revision number. For example, I've recently worked on a script stored in a repository shared by multiple developpers, and here are the successive revision number:

Externals
Another feature worth noting is called "externals". Each repository entry, file or directory, may carry metadatas as name/value pairs, so-called "properties". Their name may be whatever you want, such as a copyright or even the full license text, but a few have special meaning. I've only used the one named "svn:externals", which it is extremely useful. I've once work on a CVS repository containing multiple projects and a couple home-grown libs shared among them. In order to build the project, we had to write some shell script glue that checked out the required libs automatically into the project directory before building. As far as I know, there is no other way to circumvent this problem (let me know if there is one). This is the exact problem that externals can resolve. By adding a couple of svn:externals properties to your project directory, you compel Subversion to pull down the required libraries along your project. Supposing your repository is laid out like this:


    myproject/
    lib/mylib/

You can add a "svn:externals" property to the "myproject" directory containing:


    mylib /path/to/your/repository/lib/mylib

And the next time you'll check out or update your project, the mylib/ subdirectory will automatically created. You could even use it as a working copy of lib/mylib/ and perform commits in it!

Branches and all
Subversion only knows about "cheap copies" of a directory or file somewhere else in the repository, no tag, no branch. Let me explain. Newly copied entries share their history with their ancestor, therefrom the cheap copy. From here onward, all commits on one or the other copy won't be shared... I'm sure the concept of branch is already looming in your mind :-). You may also have already figured that branches are addressed through the repository namespace: you don't need an extra information as in CVS ("-r BRANCH"). This is why Subversion repositories are commonly laid out like this (this is the advised way in SVN documentation to design your repository, though this is really a matter of policy):


    myproject/
        trunk/
        branches/
            1.0/
            1.1/
            2.0/

You may now wonder how to create a simple tag (i.e. not a branch tag, to use CVS terminology), unless you are especially keen and you have already grasped the whole thing. Actually, there is no difference between a branch and a mere tag, except you haven't performed any commits in the latter. A good practice when using CVS when you want to create a new branch for a given release is to first tag and then branch, so you can address the exact branching point using the tag. In Subversion, a cheap copy creates a new revision number, so you only have to dig the log up to for it and ask for the diff.

This also mean you can easily move a file in the repository without losing its history, by copying and then deleting the ancestor. With CVS, this required a so-called "repo-copy", i.e. duplicating the RCS file on the repository side, which somehow is a waste of space.

To sum up the whole thing, you have a namespace in which you can do cheap copies which will share their history up to the copy revision. Everything else is just a matter of policy.

Conclusion
I really like SVN, for all those aforementioned things above.
But I won't ever blame CVS for its weaknesses. It has been designed more than twenty years ago and relies on the RCS format which was designed in the early eighties. It works so well that numerous projects are still using it, notably the FreeBSD project which is known to have the biggest open-source repository ever.

FreeBSD textdump(4) is awesome

2008-02-21T20:11:00.009+01:00

FreeBSD has had the reputation of being rock solid for a long time. One of the reason for this is that FreeBSD provides a great number powerful debugging tools.

Especially, when your kernel panics, you have three options:

Live debug with ddb(4), but this is not always possible if the box has to be up back quickly.

Dump the memory to perform post-mortem analysis.

Do nothing and pray that the panic won't happen again too soon.

Memory dumps use the swap device. This is perfectly legal because once your OS has crashed, you won't do anything with the data in the swap anyway. On the next reboot, savecore(8) checks if the swap partition contains a memory dump and copies it into a file in /var/crash.

In the beginning, only full memory dumps were possible. In this case if you have 1 GB of RAM, you need a swap partition of at least 1 GB too. So is the file in /var/crash. This worked well but given that most of users are not kernel developpers, kernel dumps are usually useless unless they are transmitted to the right folk. But a 1-GB file is cumbersome.

In April 2006, Peter Wemm introduced minidumps. They are very similar to full dumps except that, from what I've understood, only the kernel memory is dumped. Typically, on my laptop with 1 GB of RAM, minidumps took about 150 MB. The problem, while lessened, was still there though.

A couple of weeks ago, Robert Watson commited a new feature called textdump(4) in FreeBSD 8.0-CURRENT. Basically, this is possible because of two new features of ddb(4):

It is possible to define "scripts" (no loop or condition, only a sequence of commands), certain special names corresponding to events.

ddb(4) output can be captured in an internal buffer and dumped in place of the memory.

In this post, Robert Watson gives numerous informations about textdumps. I strongly advice you to read this. The very important thing is that most of panics reported by users can be solved by a backtrace and a couple of DDB commands. This is precisely what this feature achieves. Moreover, textdumps rarely exceed one megabyte, which is far more convenient than dumps or minidumps and can be easily sent by e-mail.

Moreover, users using FreeBSD as desktop obviously run X.org. When a panic arise, it is not possible to go back to console mode, so ddb(4) is not accessible. If you've asked your kernel to drop to ddb(4) on panic as I did, the kernel dump is not performed automatically and you're screwed. Textdumps removes this needle from your foot.

Now let's see how to use them. FreeBSD will automatically configures (mini)dumps for you. This is possible to do in a single command:

root# ddb script kdb.enter.panic="textdump set; capture on; show pcpu;trace;show locks;ps;alltrace;show alllocks;show lockedvnods; call doadump"

"kdb.enter.panic" is a script name with a special meaning: as its name lets sound, it will be automatically executed on panic. The first command "textdump set", forces the next dump to be the captured ddb(4) output instead of the traditional memory dump. The second one "capture on"... enables the capture of commands output. Next comes a bunch of ddb(4) commands commonly. The final command "call doadump" performs the actual dump. If you want to reboot automatically, you can add the "reset" command afterward.

As far as I know, there is no configuration sugar to enable this automatically at boot time, so for now I stuck it in /etc/rc.local.

Quick HOWTO for building Xen 3.2 on Debian/Ubuntu

2008-01-27T23:14:00.001+01:00

I finally carried out "make world" with Xen 3.2 on Debian/sid after much struggle.

First, contrary to xen-3.1.0-src.tgz, xen-3.2.0.tar.gz doesn't come along with the linux-2.6-xen-sparse/ and patches/ directories which allow to build a «xen-infied» kernel from a vanilla kernel source. Thus it is impossible to use make world XEN_LINUX_SOURCE=tarball.

By default, make world will use Mercurial to pull down (or «clone» in Mercurial vocabulary) the xenified kernel from Xensource's Mercurial repository. Unfortunately, it seems that the current Mercurial version shipped with Debian/Ubuntu is outdated and cannot be used out-of-the-box. Nonetheless, it is possible to fetch the xenified kernel manually.

From my understanding, it is necessary to "make prep-kernels" in order to create the kernel build directory. Indeed if you put your .config file directory into the kernel tree itself, the kernel's build system will complain about the lack of cleanliness and will ask you to run "make mrproper". This is baffling but it appears that whenever the kernel is asked to store the object files in a separate directory (namely build-linux-2.6.18-xen_x86_32/), it makes sure you didn't create your .config file in the wrong directory. I suppose this is a safeguard.

So I devised with the following process to build Xen 3.2.


root# mkdir build
root# wget http://bits.xensource.com/oss-xen/release/3.2.0/xen-3.2.0.tar.gz
root# tar xzf xen-3.2.0.tar.gz
# Download the xenifid kernel tree manually, but NOT in xen-3.2.0/
# because the buildconfig/select-repository script would skip it.
# ! xen-3.2.0/ and linux-2.6.18-xen.hg/ must be at the same level !
root# hg clone http://xenbits.xensource.com/linux-2.6.18-xen.hg
root# cd xen-3.2.0
root# make prep-kernels
root# cp /boot/config-2.6.18-my build-linux-2.6.18-xen_x86_32/.config
# Using the world target will clean everything first.  Don't use it here.
root# make dist

Quick HOWTO for building Qumranet's KVM

2008-01-22T11:07:00.000+01:00

This post really deserves the name scrawl, but I thought sharing my experience in building a Qumranet's KVM snapshot on Linux could helpful for others. Indeed it was not as straightforward as it seemed at first glance and it required some grope and wandering on Google.

You will need KVM (surprising, isn't it?) and Linux kernel sources corresponding to your kernel.

FWIW, KVM snapshots can be downloaded here.

First, let's prepare the kernel tree. This step is only important for people who use packaged kernel (or IOW who don't build their kernel themselves and have extracted the kernel source tree for the sole purpose of building KVM). For others, this step could be avoided because the targets we will use are performed implicitely when building the kernel.


root# tar xzf linux-2.6.23.tar.gz
root# cd linux-2.6.23.tar.gz
root# cp /boot/config-2.6.23 .config
root# make oldconfig prepare scripts
root# cd -

Next we will build KVM. Beware your GCC version! The current major branch is GCC 4 and it is shipped with almost if not all recent Linux distributions. Unfortunately, QEMU (on which KVM's userland is heavily based, not to say they've reused QEMU with little modifications) doesn't build with GCC 4. It requires GCC 3.2 or GCC 3.4 (a thorough explanation is provided in Fabrice Bellard's paper, "QEMU, a fast and portable dynamic translator", USENIX 2005). So you need to install this version of GCC as well. If you are running Debian for instance, this is pretty straightforward :


root# aptitude install gcc-3.4

And you will have a new compiler named gcc-3.4. On other distros, you will have to either find a package of GCC 3 or build it manually as described on Gentoo Wiki.

Finally you will simply have to build KVM with a few special arguments to the configure script:


root# tar xzf kvm-snapshot-20080117.tar.gz
root# cd kvm-snapshot-20080117
root# ./configure --qemu-cc=gcc-3.4 --kerneldir=$OLDPWD/linux-2.6.23 --prefix=/opt/kvm-snapshot-20080117
root# make all install

And this should work. :-)

As you may have noticed, I installed KVM in /opt in order to avoid messing with your tidy package management system.

x86 assembly generated by various GCC releases for a classic function

2008-01-05T00:05:00.000+01:00

In this new article, I will have a look at the same C source as in my previous article, except that it will be in its own function instead of being inlined in main(). We will see that some oddities spotted out previously only occured because we were in the main() function.

All tests are compiled with the following GCC command-line:

gcc -S -O test.c

The C source file is:


int     
function(int ac, char *av[])
{
        char buf[16];
        
        if (ac < 2)
                return 0;
        strcpy(buf, av[1]);
        return 1; 
}

int     
main(int ac, char *av[])
{

        function(ac, av);
        return 0;
}

Expectation, GCC 2.8.1 and GCC 2.95.3

The expectation, GCC 2.8.1 and GCC 2.95.3 assembly versions are the same as in the previous article. The stack frames are therefore identical too.

GCC 3.4.6

Fortunately, the assembly code generated by GCC 3.4.6 is far less puzzling when the C source code stands in a mere function instead of main(). Actually, the code is nearly identical to the one generated by GCC 2.95.3. The stack pointer has been aligned in the main() function on 16 bytes boundary.


function:
        pushl   %ebp
        movl    %esp, %ebp
        subl    $24, %esp        /* Alloc a 24 bytes buffer */
        movl    $0, %eax
        cmpl    $1, 8(%ebp)
        jle     .L1
        subl    $8, %esp         /* Alloc an unused 8 bytes */
                                 /* buffer */
        movl    12(%ebp), %eax
        pushl   4(%eax)
        leal    -24(%ebp), %eax  /* The 24 bytes buffer is */
                                 /* used for strcpy() */
        pushl   %eax
        call    strcpy
        movl    $1, %eax
.L1:    
        leave
        ret

The corresponding stack frame, similar to the GCC 2.95.3 one but the entire 24 bytes buffer is provided to strcpy().


       |   av   |
       |   ac   |
       |   ret  |
 ebp-> |  sebp  |
       |/ / / / | ^
       | / / / /| |
       |/ / / / | |
       | / / / /| | buf, 24 bytes wide
       |/ / / / | |
       | / / / /| v
       |\\\\\\\\| ^
       |\\\\\\\\| v 8 unused bytes
       |  av[1] |
 esp-> |  &buf  |

GCC 4.2.1

Astonishingly, GCC 4.2.1 does not keep with 16 bytes alignment, although it seemed to do so in the main() function.


function:
        pushl   %ebp
        movl    %esp, %ebp
        subl    $24, %esp        /* Alloc a 24 bytes buffer */
        movl    $0, %eax
        cmpl    $1, 8(%ebp)
        jle     .L4
        movl    12(%ebp), %edx
        movl    4(%edx), %eax
        movl    %eax, 4(%esp)    /* Fake push */
        leal    -16(%ebp), %eax  /* A 16 bytes buffer is */
                                 /* used for strcpy() */
        movl    %eax, (%esp)     /* Fake push */
        call    strcpy
        movl    $1, %eax
.L4:
        leave
        ret

And now the stack frame:


       |   av   |
       |   ac   |
       |   ret  |
 ebp-> |  sebp  |
       |/ / / / | ^ ^
       | / / / /| | |
       |/ / / / | | | buf, 16 bytes wide
       | / / / /| | v
       |  av[1] | |
 esp-> |  &buf  | v

The stack frame looks exactly like the expectation. One thing worth noting however is that GCC 4.2.1 always uses peculiar code for arguments storage. Instead of using the push instruction, it reserves space for arguments of further function calls in the same time as local variables are allocated. Arguments are the stored relative to %esp.

x86 assembly generated by various GCC releases for the main() function

2007-12-23T23:53:00.000+01:00

In this article, I will look at the assembly code generated from it by various releases of GCC on FreeBSD and I will try to explain what it does and why (though a couple of things are still mysterious). Moreover I will illustrate each assembly with a snapshot of the call stack right before calling strcpy. This shouldn't change much on other operating systems (such as Linux).

All tests are compiled with the following GCC command-line:

gcc -S -O test.c

Here is the very simple (vulnerable) C program I will use:


int
main(int ac, char *av[])
{
        char buf[16];

        if (ac < 2)
                return 0;
        strcpy(buf, av[1]);
        return 1;
}

Expectation

First let's see what I would expect. This first assembly code comes along with verbose comments. Later ones won't be


main:
  enter                 /* Create a new stack frame */
  sub $16, %esp         /* Allocate buf */
  cmp $1, 8(%ebp)       /* Compare ac to 1... */
  jle .byebye0          /* And set the return value to */
                        /* 0 if lower or equal */
  mov 12(%ebp), %eax    /* Load av */
  push 4(%eax)          /* Push av[1] */
  push -16(%ebp)        /* Push &buf */
  call strcpy
  mov $1, %eax          /* Set the return value to 1 */
  jmp .byebye
byebye0:
  mov $0, %eax
byebye:
  leave                 /* Restore the old stack frame */
  ret                   /* Return to the caller */

Now let's look at the stack frame:


       |   av   |
       |   ac   |
       |   ret  |
 ebp-> |  sebp  | saved ebp from the previous stack frame
       |/ / / / | ^
       | / / / /| |
       |/ / / / | | buf, 16 bytes wide
       | / / / /| v
       |  av[1] |
 esp-> |  &buf  |

GCC 2.8.1

With GCC 2.8.1, the assembly is pretty close to what I expected:


main:
  pushl %ebp            /* Don't know why enter isn't */
  movl %esp,%ebp        /* but this is semantically */
                        /* equivalent */
  subl $16,%esp
  cmpl $1,8(%ebp)
  jle .L2
  movl 12(%ebp),%eax
  pushl 4(%eax)
  leal -16(%ebp),%eax
  pushl %eax
  call strcpy
  movl $1,%eax
  jmp .L3
  .align 4
.L2:
  xorl %eax,%eax
.L3:
  leave
  ret

The stack right before calling strcpy() is identical to the expectation.

GCC 2.95.3

Here, the generated assembly is very close to above, but the size allocated on the stack is a little larger (8 bytes). Moreover, 8 additional bytes are allocated on the stack before pushing the arguments.


main:
   pushl %ebp
   movl %esp,%ebp
   subl $24,%esp        /* 24 bytes are allocated */
                        /* instead of 16 */
   cmpl $1,8(%ebp)
   jle .L3
   addl $-8,%esp
   movl 12(%ebp),%eax
   pushl 4(%eax)
   leal -16(%ebp),%eax  /* surprisingly, a 16 bytes */
                        /* buffer is still provided */
                        /* to strcpy() */
   pushl %eax
   call strcpy
   movl $1,%eax
   jmp .L4
   .p2align 4,,7
.L3:
   xorl %eax,%eax
.L4:
   leave
   ret

And the corresponding stack frame:


       |   av   |
       |   ac   |
       |   ret  |
 ebp-> |  sebp  |
       |/ / / / | ^ ^
       | / / / /| | |
       |/ / / / | | | buf, 16 bytes wide
       | / / / /| | v
       |////////| | ^
       |////////| v v 8 unused bytes
       |\\\\\\\\| ^
       |\\\\\\\\| v 8 unused bytes
       |  av[1] |
 esp-> |  &buf  |

In order to understand what kind of optimization GCC tried to perform, I compile the test program with multiple buffer sizes:


sizeof(buf)       stack alloc
0                 8              (= 16 * 1 - 8)
1-16              24             (= 16 * 2 - 8)
17-32             40             (= 16 * 3 - 8)
33-48             56             (= 16 * 4 - 8)
49-64             72             (= 16 * 5 - 8)

Given that GCC 2.95.3 allocates an additional 8 bytes buffer afterward, it is undoubtly doing 16-bytes alignment here. The intent is good but in my opinion it lacks an initial alignment of the stack frame pointer that later GCC releases do, as we are going to see.

GCC 3.4.6

This release is the greediest among the ones we are looking at with regard to the stack management.


main:
   pushl %ebp
   movl  %esp, %ebp
   subl  $24, %esp      /* Alloc a 24 bytes buffer */
   andl  $-16, %esp     /* Stack alignment on 16 */
                        /* bytes boundary */
   subl  $16, %esp      /* Allocates another 16 */
                        /* bytes buffer */
   movl  $0, %eax
   cmpl  $1, 8(%ebp)
   jle   .L1
   subl  $8, %esp       /* And yet another 8 bytes one */
   movl  12(%ebp), %eax
   pushl 4(%eax)
   leal  -24(%ebp), %eax /* The 24 bytes buffer is */
                         /* used for strcpy() */
   pushl %eax
   call  strcpy
   movl  $1, %eax
.L1
   leave
   ret

Looking at the stack, we can see it is greedily stuffed:


       |   av   |
       |   ac   |
       |   ret  |
 ebp-> |  sebp  |
       |/ / / / | ^
       | / / / /| |
       |/ / / / | |
       | / / / /| | buf, 24 bytes wide
       |/ / / / | |
       | / / / /| v
       |\\\\\\\\| ^
       |\\\\\\\\| | stack alignment on 16 bytes boundary
       |\\\\\\\\| v (runtime-dependent size)
       |/ / / / | ^
       | / / / /| |
       |/ / / / | | 16 unused bytes
       | / / / /| v
       |\ \ \ \ | ^
       | \ \ \ \| v 8 unused bytes
       |  av[1] |
 esp-> |  &buf  |

I performed the same test as above by varying buf size and I get the very same result as GCC 2.95.3. It is however difficult to argue this is to maintain alignment since strcpy() arguments are pushed far below and the stack is aligned explicitely right after anyway. This question stays opened.

Interestingly, while GCC 2.95.3 allocates 24 bytes but only provides an address with 16 available bytes, GCC 3.4.6 allocates 24 bytes too but provides the whole buffer to strcpy(). My feeling is that instead of wasting 8 bytes, GCC developpers prefered to include them in the buffer in order to mitigate off-by-one/two exploitations.

The 8 bytes buffer is certainly here to create a 16 byte block with the two arguments of strcpy(). Nonetheless, I don't understand the purpose of the 16 bytes buffer at all.

GCC 4.2.1

This release generates assembly that seems quite odd at first glance but after thinking a bit about it, everything is (quite) understandable, contrary to GCC 3.4.6 :-).


main:
  leal 4(%esp), %ecx    /* Load &ac in %ecx */
  andl $-16, %esp       /* Stack alignment on 16 */
                        /* bytes boundary */
  pushl -4(%ecx)        /* Push the ret again */
  pushl %ebp            /* Creates a new stack frame */
  movl %esp, %ebp
  pushl %ecx            /* Push &ac */
  subl $36, %esp        /* Create a 36 bytes buffer */
  movl 4(%ecx), %edx    /* Load av in %edx */
  movl $0, %eax         /* Return value will be 0 if */
                        /* the comparison fails */
  cmpl $1, (%ecx)
  jle .L4
  movl 4(%edx), %eax    /* Load av[1] in %eax */
  movl %eax, 4(%esp)    /* Fake push */
  leal -20(%ebp), %eax  /* Load the 16 bytes buffer */
                        /* (jump over &ac, thus 20) */
  movl %eax, (%esp)     /* Fake push */
  call strcpy
  movl $1, %eax
.L4:
  addl $36, %esp        /* Cleanup the stack  */
  popl %ecx             /* Pop &ac in %ecx */
  popl %ebp             /* Restore %ebp */
  leal -4(%ecx), %esp   /* Restore %esp thanks to &ac */
  ret

Evil, isn't it? :-) Let's look at the stack snapshot right before strcpy is called(). You will see that actually the stack is handled very neatly.


      |   av   |
      |   ac   |
      |   ret  |
      |\\\\\\\\| ^
      |\\\\\\\\| | stack alignment on 16 bytes boundary
      |\\\\\\\\| v (runtime-dependent size)
      |   ret  |
ebp-> |  sebp  |
      |   &ac  |
      |/ / / / | ^ ^
      | / / / /| | |
      |/ / / / | | | buf, 16 bytes wide
      | / / / /| | v
      |////////| | ^
      |////////| | | 12 unused bytes
      |////////| | v
      |  av[1] | |
esp-> |  &buf  | v

As you may have already notice, stack alignment is performed before creating a new stack frame. Therefore, contrary to the previous assembly we have seen, it is not possible to get ac or av with simple indexing from %ebp since the gap created by the alignment cannot be known at compile time. So the very first instruction stores the address of ac in %ecx; it is then possible to access main()'s arguments: (%ecx) for ac and 4(%ecx) for av. Note that -4(%ecx) is the address where ret is stored.

Still before creating the new stack frame, ret is pushed once again. I'm not sure about the purpose of this, but my feeling is that it is meant to create an authentic-looking stack frame, in case the program performs nasty things with it. Only then the new stack frame is created. &ac is pushed afterward so we won't lose its track if %ecx is used for something else.

A 36 bytes buffer is allocated on the stack which will hold buf in its end (top) and strcpy() arguments at its beginning (bottom). Indeed you can see that before calling strcpy(), instead of using "push" to set arguments on the stack, it simply stores values using indexation from %esp. Ok that's fine but we only need 16 + 8 = 24 bytes to hold all this, not 36, so there are 12 superfluous bytes. Yes, but GCC wants to keep its 16 bytes alignment! Look: After the explicit stack alignment, 12 bytes have been pushed (3 words); 16 bytes are needed for buf and 8 bytes (2 words) are needed for the arguments of strcpy(). 12 + 16 + 8 = 36; the next multiple of 16 is 48, so it allocates 48 - 36 = 12 "superfluous" bytes. Neat, eh?

Upon return, &ac is popped up in %ecx, %ebp is popped as well and %esp is restored to its initial value.

One thing I'm not sure to understand is why GCC stores &ac in %ecx. Why not simply perform "mov %ebp, %ecx" at the beginning. One could argue this is because ac is used later in a comparison but I tried to remove the test from the C source and this does change anything. So I think this is merely a convention.

Without wanting to be too picky, I also wonder why av is loaded into %edx before the test, while it is only useful after it. I don't have any idea about this :-).

Technicals writings from JLH

Apache mod_rewrite evilness (with dynamic vhosts and .htaccess rewrite rules)

Installing full-ZFS server at OVH

Configuring FreeBSD with dual console

Configuring boot2

Configuring loader(8)

Configuring the kernel

How the whole stuff works

Userland output

Configuring login(8)

Debian KVM console on a headless server

Quick n' Dirty Linux WPA-PSK Wireless AP

column.sh - columizator

Bash prompt trick: cheap emulation of tcsh's pwd trailing component

Addendum on 2009/06/22

Get rid of Vodafone Mobile Connect Card driver for Linux

Global Offset Table, why is it needed?

Preface

So, why GOT?

Debian on Asus EeePC 701 with Huawei USB modem from SFR or Vodafone

Chicken and egg problem with Propolice in runtime linker/loader

Sketching Subversion

FreeBSD textdump(4) is awesome

Quick HOWTO for building Xen 3.2 on Debian/Ubuntu

Quick HOWTO for building Qumranet's KVM

x86 assembly generated by various GCC releases for a classic function

Expectation, GCC 2.8.1 and GCC 2.95.3

GCC 3.4.6

GCC 4.2.1

x86 assembly generated by various GCC releases for the main() function

Expectation

GCC 2.8.1

GCC 2.95.3

GCC 3.4.6

GCC 4.2.1

Configuring `loader(8)`

Configuring `login(8)`