Installing FreeBSD on a Dedibox (online.net)

Created: 2014/07/27
Updated: 2014/08/12 - higher-end Dedibox
Updated: 2014/08/14 - software RAID-1

My first server was a Dedibox. I then switched to OVH's Kimsufi (for the anecdote, "Kimsufi" sounds like "enough for me" in French) which at the time was more attractive (15 EUR/month instead of 20).  My setup then evolved to make use of failover IPs (an extra IP address which you can switch from one OVH server to another).

But now my Kimsufi servers are getting old and slow or expensive, depending for which side you're looking at it and I'd like to upgrade them. OVH has very attractive prices like 5 EUR/month for the cheapest one (at least theoritically given I've never seen them available despite them constantly announcing the last server was shipped a few hours ago).  For the price I pay today overall, I could get an i5 CPU on the main server and a cheap and low-grade server for the failover...

... if only OVH was still proposing failover IPs on Kimsufi grade servers. The Kimsufi offer has spun off to really focus on home-servers where -- I guess -- people don't need such fancy services. If you want to have a failover IP at OVH now, you need to get an enterprise grade server which costs at least 80 EUR/month.  I guess this move has been motivated by the IPv4 addresses crunch.

Anyway.  I'm now going back to Dedibox which still offers failover IPs. Also they offer access to the server's console through a Java applet, which can be super useful. Something OVH did back in the time but then removed. However Dedibox don't offer FreeBSD by default (which Kumsufi did at the time, and maybe still do) so you have to install it yourself. Here is how:

• Boot an Ubuntu rescue system, login and run sudo to be root.
• Download a FreeBSD release or snapshot ISO; 10-STABLE can be found here: http://ftp2.fr.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/10.0/
• Install QEMU:

  apt-get update && apt-get install qemu-kvm
  

• Start QEMU with a VNS server, attached to the raw disk, booting on the ISO:

  qemu-system-x86_64 -no-kvm -hda /dev/sda -cdrom FreeBSD-*.iso -net nic,model=e1000 -vnc :1 -boot d

  or, if you have two disks:

  qemu-system-x86_64 -no-kvm -hda /dev/sda -hdb /dev/sdb -cdrom FreeBSD-*.iso -net nic,model=e1000 -vnc :1 -boot d

• Connect in using VNC:

  xvncviewer ${server_ip}:1

Now you can install FreeBSD. The only thing is that the bootloader won't be installed (correctly?) for some unknown reason. So you need to do it yourself.

I couldn't come up with the a way to partition the server the way I want with the bsdinstaller, so I typically switch to the console (Alt-F4) right after choosing the keymap to make it using gpart. I'll describe it here for the record, so it will save me from research the next time I'll do it.

Here is the partitioning scheme:

• We're in 2014, so we're using GPT boot, which requires a small partition for the bootloader.
• I want the base system in UFS, mainly because it's less brittle to deal with remotely and you can use nextboot(8) fully (the new kernel will be booted only once and they the previous one will be reinstated, so you can try kernels); although all this is less mandatory if you have a working console as with Dedibox.
• A small swap partition.
• The remaining in ZFS.

You have a single disk or a hardware RAID controller

Here is how to do it from scratch:

###
### Setup partitions
###
# dd if=/dev/zero of=/dev/ada0 bs=64k count=128
128+0 records in
128+0 records out
8388608 bytes transferred in 0.042453 secs (197599244 bytes/sec)
# gpart show
# gpart create -s GPT ada0
ada0 created
# gpart add -t freebsd-boot -s 64k -i 1 ada0
ada0p1 added
# gpart add -t freebsd-ufs -s 20G -i 2 ada0
ada0p2 added
# gpart add -t freebsd-swap -s 2G -i 3 ada0
ada0p3 added
# gpart add -t freebsd-zfs -i 4 ada0
ada0p4 added
###
### Create the filesystems
###
# newfs -j /dev/ada0p2
[...]
# zpool create tank /dev/ada0p4
cannot mount '/tank': failed to create mountpoint (this is expected and harmless)
###
### Install the bootcode manually as it will fail for some reason
###
# gpart bootcode -b /boot/pbmr -p /boot/gptboot -i 1 ada0
bootcode written to ada0

Now go ahead and install FreeBSD.

You want to software RAID-1

Note that we use geom_mirror for the first three partition, and ZFS mirroring for the pool as I think it has better performances.

Also, note that the current bsdinstaller does not seem to understand what is a geom_mirror device and wants us to create a partitioning scheme on it.  We will therefore mount the partition to /mnt manually.

###
### Setup partitions on the first disk, ada0
###
# dd if=/dev/zero of=/dev/ada0 bs=64k count=128
128+0 records in
128+0 records out
8388608 bytes transferred in 0.042453 secs (197599244 bytes/sec)
# gpart show
# gpart create -s GPT ada0
ada0 created
# gpart add -t freebsd-boot -s 64k -i 1 ada0
ada0p1 added
# gpart add -t freebsd-ufs -s 20G -i 2 ada0
ada0p2 added
# gpart add -t freebsd-swap -s 2G -i 3 ada0
ada0p3 added
# gpart add -t freebsd-zfs -i 4 ada0
ada0p4 added
###
### Now duplicate those steps for ada1.
###
[...]
###
### Now create the mirror
###
# kldload geom_mirror
# gmirror label gm-root ada0p2 ada1p2
# gmirror label gm-swap ada0p3 ada1p3
###
### Create the filesystems
###
# newfs -j /dev/ada0p2
[...]
# zpool create tank mirror /dev/ada0p4 /dev/ada1p4
cannot mount '/tank': failed to create mountpoint (this is expected and harmless)
###
### Install the bootcode manually as it will fail for some reason
###
# gpart bootcode -b /boot/pbmr -p /boot/gptboot -i 1 ada0
bootcode written to ada0
# gpart bootcode -b /boot/pbmr -p /boot/gptboot -i 1 ada1
bootcode written to ada1
###
### Mount the partition to install FreeBSD
###
# mount /dev/mirror/gm-root /mnt

Now when the installer asks you about the partitioning, select "Shell" and FreeBSD will be installed to /mnt once you exit this shell.

End of installation

A few notes (some for myself):

• The default router in online.net's network is .1.
• Add the following lines to /etc/rc.conf

sendmail_submit_enable=NO
sendmail_outbound_enable=NO
sendmail_msp_queue_enable=NO

• YMMV but on QEMU I have an em0 interface, but on the real server it can be igb0 or bce0; so I need to change my /etc/rc.conf accordingly. What I typically do if I'm not sure, is duplicate the ifconfig_em0 line to ifconfig_igb0 and ifconfig_bce0.
• Add a user or enable root login on sshd.
• If you are using software RAID, add geom_mirror_load="YES" to /boot/loader.conf.
• If your server has a PERC h200 controller, the disk won't be /dev/ada0 but /dev/da0, even on FreeBSD10, so change /etc/fstab accordingly.
• Add the swap to /etc/fstab.

Small example of what /etc/fstab look like (with a software mirror in that case, but you cannot boot on a mirror, the subsystem is not there yet at boot, so you need to pick one of the disk):

/dev/mirror/gm-root     /               ufs     rw      1       1
/dev/mirror/gm-swap     none            swap    sw      0       0

Now before trying to boot FreeBSD on the real server, just try it on QEMU, this will save you some time if you missed something. First umount cleanly your disks and then kill QEMU from the host. Now re-run it using:

qemu-system-x86_64 -no-kvm -hda /dev/sda -net nic,model=e1000 -vnc :1 -boot c

This should go to the boot pompt. Shut down cleanly. Then you can try to boot it on the real server. If it does not come only, you can still use the Java console to debug it (for example, the interface name may not be correct).