Quick HOWTO for building Xen 3.2 on Debian/Ubuntu

I finally carried out "make world" with Xen 3.2 on Debian/sid after much struggle.

First, contrary to xen-3.1.0-src.tgz, xen-3.2.0.tar.gz doesn't come along with the linux-2.6-xen-sparse/ and patches/ directories which allow to build a «xen-infied» kernel from a vanilla kernel source. Thus it is impossible to use make world XEN_LINUX_SOURCE=tarball.

By default, make world will use Mercurial to pull down (or «clone» in Mercurial vocabulary) the xenified kernel from Xensource's Mercurial repository. Unfortunately, it seems that the current Mercurial version shipped with Debian/Ubuntu is outdated and cannot be used out-of-the-box. Nonetheless, it is possible to fetch the xenified kernel manually.

From my understanding, it is necessary to "make prep-kernels" in order to create the kernel build directory. Indeed if you put your .config file directory into the kernel tree itself, the kernel's build system will complain about the lack of cleanliness and will ask you to run "make mrproper". This is baffling but it appears that whenever the kernel is asked to store the object files in a separate directory (namely build-linux-2.6.18-xen_x86_32/), it makes sure you didn't create your .config file in the wrong directory. I suppose this is a safeguard.

So I devised with the following process to build Xen 3.2.

root# mkdir build
root# wget http://bits.xensource.com/oss-xen/release/3.2.0/xen-3.2.0.tar.gz
root# tar xzf xen-3.2.0.tar.gz
# Download the xenifid kernel tree manually, but NOT in xen-3.2.0/
# because the buildconfig/select-repository script would skip it.
# ! xen-3.2.0/ and linux-2.6.18-xen.hg/ must be at the same level !
root# hg clone http://xenbits.xensource.com/linux-2.6.18-xen.hg
root# cd xen-3.2.0
root# make prep-kernels
root# cp /boot/config-2.6.18-my build-linux-2.6.18-xen_x86_32/.config
# Using the world target will clean everything first.  Don't use it here.
root# make dist

Quick HOWTO for building Qumranet's KVM

This post really deserves the name scrawl, but I thought sharing my experience in building a Qumranet's KVM snapshot on Linux could helpful for others. Indeed it was not as straightforward as it seemed at first glance and it required some grope and wandering on Google.

You will need KVM (surprising, isn't it?) and Linux kernel sources corresponding to your kernel.

FWIW, KVM snapshots can be downloaded here.

First, let's prepare the kernel tree. This step is only important for people who use packaged kernel (or IOW who don't build their kernel themselves and have extracted the kernel source tree for the sole purpose of building KVM). For others, this step could be avoided because the targets we will use are performed implicitely when building the kernel.

root# tar xzf linux-2.6.23.tar.gz
root# cd linux-2.6.23.tar.gz
root# cp /boot/config-2.6.23 .config
root# make oldconfig prepare scripts
root# cd -

Next we will build KVM. Beware your GCC version! The current major branch is GCC 4 and it is shipped with almost if not all recent Linux distributions. Unfortunately, QEMU (on which KVM's userland is heavily based, not to say they've reused QEMU with little modifications) doesn't build with GCC 4. It requires GCC 3.2 or GCC 3.4 (a thorough explanation is provided in Fabrice Bellard's paper, "QEMU, a fast and portable dynamic translator", USENIX 2005). So you need to install this version of GCC as well. If you are running Debian for instance, this is pretty straightforward :

root# aptitude install gcc-3.4

And you will have a new compiler named gcc-3.4. On other distros, you will have to either find a package of GCC 3 or build it manually as described on Gentoo Wiki.

Finally you will simply have to build KVM with a few special arguments to the configure script:

root# tar xzf kvm-snapshot-20080117.tar.gz
root# cd kvm-snapshot-20080117
root# ./configure --qemu-cc=gcc-3.4 --kerneldir=$OLDPWD/linux-2.6.23 --prefix=/opt/kvm-snapshot-20080117
root# make all install

And this should work. :-)

As you may have noticed, I installed KVM in /opt in order to avoid messing with your tidy package management system.

x86 assembly generated by various GCC releases for a classic function

In this new article, I will have a look at the same C source as in my previous article, except that it will be in its own function instead of being inlined in main(). We will see that some oddities spotted out previously only occured because we were in the main() function.

All tests are compiled with the following GCC command-line:

gcc -S -O test.c

The C source file is:

int     
function(int ac, char *av[])
{
        char buf[16];
        
        if (ac < 2)
                return 0;
        strcpy(buf, av[1]);
        return 1; 
}

int     
main(int ac, char *av[])
{

        function(ac, av);
        return 0;
}

Expectation, GCC 2.8.1 and GCC 2.95.3

The expectation, GCC 2.8.1 and GCC 2.95.3 assembly versions are the same as in the previous article. The stack frames are therefore identical too.

GCC 3.4.6

Fortunately, the assembly code generated by GCC 3.4.6 is far less puzzling when the C source code stands in a mere function instead of main(). Actually, the code is nearly identical to the one generated by GCC 2.95.3. The stack pointer has been aligned in the main() function on 16 bytes boundary.

function:
        pushl   %ebp
        movl    %esp, %ebp
        subl    $24, %esp        /* Alloc a 24 bytes buffer */
        movl    $0, %eax
        cmpl    $1, 8(%ebp)
        jle     .L1
        subl    $8, %esp         /* Alloc an unused 8 bytes */
                                 /* buffer */
        movl    12(%ebp), %eax
        pushl   4(%eax)
        leal    -24(%ebp), %eax  /* The 24 bytes buffer is */
                                 /* used for strcpy() */
        pushl   %eax
        call    strcpy
        movl    $1, %eax
.L1:    
        leave
        ret

The corresponding stack frame, similar to the GCC 2.95.3 one but the entire 24 bytes buffer is provided to strcpy().

       |   av   |
       |   ac   |
       |   ret  |
 ebp-> |  sebp  |
       |/ / / / | ^
       | / / / /| |
       |/ / / / | |
       | / / / /| | buf, 24 bytes wide
       |/ / / / | |
       | / / / /| v
       |\\\\\\\\| ^
       |\\\\\\\\| v 8 unused bytes
       |  av[1] |
 esp-> |  &buf  |

GCC 4.2.1

Astonishingly, GCC 4.2.1 does not keep with 16 bytes alignment, although it seemed to do so in the main() function.

function:
        pushl   %ebp
        movl    %esp, %ebp
        subl    $24, %esp        /* Alloc a 24 bytes buffer */
        movl    $0, %eax
        cmpl    $1, 8(%ebp)
        jle     .L4
        movl    12(%ebp), %edx
        movl    4(%edx), %eax
        movl    %eax, 4(%esp)    /* Fake push */
        leal    -16(%ebp), %eax  /* A 16 bytes buffer is */
                                 /* used for strcpy() */
        movl    %eax, (%esp)     /* Fake push */
        call    strcpy
        movl    $1, %eax
.L4:
        leave
        ret

And now the stack frame:

       |   av   |
       |   ac   |
       |   ret  |
 ebp-> |  sebp  |
       |/ / / / | ^ ^
       | / / / /| | |
       |/ / / / | | | buf, 16 bytes wide
       | / / / /| | v
       |  av[1] | |
 esp-> |  &buf  | v

The stack frame looks exactly like the expectation. One thing worth noting however is that GCC 4.2.1 always uses peculiar code for arguments storage. Instead of using the push instruction, it reserves space for arguments of further function calls in the same time as local variables are allocated. Arguments are the stored relative to %esp.