Boot All The Things!

Frequent readers of this blog probably know I’m a big fan of free and open source software and also that I have a warm spot in my heart for the BSDs – especially OpenBSD.  Unfortunately, some of the tools I need to do my job just simply aren’t available on the BSDs and rather than sully myself with some nasty proprietary operating system from some company in Washington state, I will typically use Arch Linux to get work done.  On those rare occasions where I absolutely have to run the NSA endpoint known as Windows, I will run it in a VM and quickly shower afterwards.  🙂

Sometimes getting dual booting working can be a bit of a pain.  But, as a guy who likes to turn his guitar amplifier to 11, I started wondering how much self-inflicted pain I could take if I wanted to set up the most massively over-engineered laptop that would use rEFInd to partake of ALL of my favorite operating systems just by simply rebooting.  Well, not one to back down from an entirely self-imposed challenge, I decided to give it a go with my trusty old test laptop, a Lenovo Thinkpad x230 with a 1 TB SSD drive and 16 GB of RAM.

I’ll note that along the way I managed to trip over my own shoelaces several times and wiped out up to 4 working operating systems with one poor choice so this path isn’t for the faint of heart!

NOTE:  Pro tip here – I spent a fair amount of time chasing my tail with UEFI problems until I pulled my head out of… well, let’s just say a dark and smelly place <grin>, and checked the age of my Thinkpad x230’s BIOS.  Turns out it was created somewhere along the same time that the Battle of Hastings.  I found this great resource that pointed out how I could update to the latest BIOS without having to install Windows and it solved SOOOOO many problems for me.  Save yourself some pain and make sure your BIOS is updated, regardless of what hardware you are using.

The Goal

Be able to multi-boot the following FOSS operating systems using rEFInd to select between them:

  1. Arch Linux
  2. DragonFlyBSD
  3. FreeBSD
  4. OpenBSD
  5. NetBSD

So without further ado, lets download the install images, burn them to thumb drives and get started feeling the pain!

Arch Linux

I decided to start by installing Arch with EFI support and the bootctl bootloader using an encrypted partition.  I downloaded the latest media and put it on a thumb drive:

# dd if=archlinux-2017.06.01-x86_64.iso of=/dev/sda bs=1M
# sync

It kills me to have to use that uppercase M (in BSD-land it is lowercase!) but I then had a thumbdrive I could boot from on the x230 and start the Arch install.  It’s always a good idea to start with the lsblk command to ensure you know what disk names were picked at boot time as they can vary.

# timedatectl set-ntp true
# gdisk /dev/sda

The commands I issued in gdisk were first ‘o’ to create a new GPT partition table for the drive, then ‘n’ to create a 200M /boot partition of type ef00 (EFI), and finally ‘n’ again to create a 180G data partition with the standard Linux partition type. Then I create the encrypted volume and slice it up into /root, /home and swap:

# cryptsetup -c aes-xts-plain64 -s 512 -h sha512 -i 5000 -y luksFormat /dev/sda2
# cryptsetup luksOpen /dev/sda2 crypt
# lvm pvcreate /dev/mapper/crypt
# lvm vgcreate lvmpool /dev/mapper/crypt
# lvcreate -L 100GB -n root lvmpool
# lvcreate -L 32GB -n swap lvmpool
# lvcreate -l 100%FREE -n home lvmpool
# mkfs.fat -F32 /dev/sda1
# mkfs.ext4 /dev/mapper/lvmpool-root
# mkfs.ext4 /dev/mapper/lvmpool-home
# mkswap /dev/mapper/lvmpool-swap
# swapon /dev/mapper/lvmpool-swap

Now it’s time to mount all of that and install the base system:

# mount /dev/mapper/lvmpool-root /mnt
# mkdir -p /mnt/boot
# mount /dev/sda1 /mnt/boot
# mkdir -p /mnt/home
# mount /dev/mapper/lvmpool-home /mnt/home
# pacstrap -i /mnt base base-devel

Create the /etc/fstab for the new system and chroot into it:

# genfstab -U /mnt >/mnt/etc/fstab
# arch-chroot /mnt /bin/bash

Go ahead and install Arch as you normally would.  Uncomment en_US.UTF-8 in /etc/locale.gen, run locale-gen, add LANG=en_US.UTF-8 in /etc/locale.conf and:

# tzselect
# ln -sf /usr/share/zoneinfo/Zone/SubZone /etc/localtime
# hwclock –systohc

Edit /etc/mkinitcpio.conf and insert encrypt and lvm2 between block and filesystems hooks:

# mkinitcpio -p linux

Install the bootloader:

# bootctl install

Edit the /boot/loader/entries/arch.conf file and put in the following:

title   Arch Linux

linux  /vmlinuz-linux
initrd  /initramfs-linux.img
options cryptdevice=/dev/sda2:crypt ro root=/dev/mapper/lvmpool-root rw

Change the hostname of your system by editing /etc/hostname then install the necessary wifi software:

# pacman -S iw wpa_supplicant dialog

Use the passwd command to set the root password and then finally enable the dhcpcd service:

# systemctl enable dhcpcd.service

Now, we need to install rEFInd:

# pacman -S refind-efi
# refind-install
# efibootmgr –create –disk /dev/sda –part 1 –loader /EFI/refind/refind_x64.efi –label “rEFInd Boot Manager”

Hit CTRL-D to exit the chroot environment, unmount the /mnt partitions using umount -R and reboot the new system.  NOTE:  Don’t forget to set your BIOS settings to boot UEFI first or you won’t be able to pull anything up.

If you did everything right, you should be presented with a password prompt to decrypt your volume.  If you did what I always do and fat finger the /boot/loader/entries/arch.conf entry for initrd as “initrc” then just boot off of your thumb drive, decrypt the volume, mount things up, chroot into it and fix your typo.

When you boot up, start the dhcpcd service so you have access to the Internet:

# systemctl start dhcpcd.service

Reboot to verify that everything is working and then install the rest of your packages and configure everything the way you like it.

OpenBSD

Now I’m back in my element!  So, I pulled down the install61.fs image from my favorite mirror site in Toronto, dd’ed it to a thumb drive and booted.  On the x230, you have to modify the firmware to boot in legacy mode first or it won’t properly boot.  To save me the trouble of doing math to calculate what sector I wanted my OpenBSD partition to start in, I just used gparted and created a 180G unformatted partition that I’ll repurpose on the OpenBSD side of the world.

After booting the installer thumb drive I immediately dropped to the shell to modify the partition to type A6 using fdisk:

# fdisk -e sd0
fdisk: 1> edit 2
Partition id:  A6
Partition offset: [enter]
Partition size: [enter]
Partition name: OpenBSD
fdisk:*1> write
fdisk:  1> quit

Now that we have an OpenBSD partition on the drive, we need to use disklabel to create an encrypted slice:

# disklabel -E sd0
> a
partition: a
offset: [enter]
size: [enter]
FS type: RAID
> w
> q
# bioctl -c C -l /dev/sd0a softraid0

After entering and confirming my passphrase, I exited back to the installer, making note that my encrypted volume was mounted as sd3.  I told the installer to install to sd3 using the whole disk and modified the auto layout to be the way I like it.

Now I needed to do some voodoo in order to get rEFInd to work properly.  Since I didn’t let OpenBSD have its way with the disk, it has no knowledge of my EFI partition and thus couldn’t put its EFI loader into it.  To work around this, I booted back into Arch, downloaded the BOOTX64.EFI file from the mirror and plopped it down into the EFI directory:

# cp /tmp/BOOTX64.EFI /boot/EFI/BOOT/bootx64_openbsd.efi

Let’s take a second to clean up rEFInd so that we get an Arch icon and an OpenBSD icon. To do that, either boot Arch (which will mount the EFI partition under /boot) or just mount the darned thing in OpenBSD (doas mount /dev/sd0i /mnt) and navigate to the EFI/refind folder.  In there, edit the refind.conf file and add the following:

hideui hints
scanfor manual
menuentry “Arch” {
icon \EFI\refind\icons\os_arch.png
loader \EFI\systemd\systemd-bootx64.efi
}
menuentry “OpenBSD”
{
icon \EFI\refind\icons\os_openbsd.png
loader \EFI\boot\bootx64_openbsd.efi
}

Stick this icon in the EFI/refind/icons directory as os_openbsd.png and you’ll be good to go.

Testing it worked just fine.  I rebooted and was able to see the entry on the screen in rEFInd and selecting it presented me with the drive password prompt which, when entered allowed me to boot to a login prompt.  I logged in as root and configured /etc/doas.conf to allow anyone in the wheel group to escalate their privileges and then added my local user to the wheel group:

# echo “permit keepenv persist :wheel as root” > /etc/doas.conf
# usermod -G wheel <my_user>

I added /etc/installurl to point to my favorite mirror (https://openbsd.cs.toronto.edu/pub/OpenBSD) and ran syspatch to patch me to the latest level for my release.  At this point, I logged out and back in as my local user and installed the gnome desktop environment:

# pkg_add gnome
# rcctl enable avahi_daemon
# rcctl enable gdm
# rcctl enable messagebus
# rcctl enable apmd
# rcctl set apmd flags -A
# echo ‘multicast_host=YES’ >> /etc/rc.conf.local

Wow.  That was hard, wasn’t it?  Reboot and you should be staring at the GDM login screen.  You can now install the rest of the package you want for OpenBSD.  My favorites are libreoffice, firefox, chromium, evolution and evolution-ews.

DragonFlyBSD

Interestingly enough, the first dozen times I tried this (lol – I made a lot of mistakes and wiped out previous work) I discovered that DragonFlyBSD will try to boot my FreeBSD partition if it has a lower partition number on the disk.  I’m still looking for a workaround for that but in the meanwhile, I solved the problem by just installing DragonFlyBSD first and then FreeBSD next.

Since DragonFlyBSD 4.8 supports EFI, I downloaded the installer, dd’ed it to a thumb drive and booted it up.  The installer for DragonFlyBSD is interesting in that you are actually booting a live image.  Depending on who you log in as you are either in a shell (user=root) or the installer (user=installer).

I logged in as root and once I was at the shell prompt, I started poking around in man pages and discovered to my delight that it feels very much like OpenBSD so I was much more at home than I expected to be.  After some research, I discovered that there are two disk partition management tools: fdisk for MBR partition schemes and gpt for GPT schemes.  Since my disk is a GPT scheme, that’s the one I use.  By issuing the following command, I was able to show my GPT partitions on the disk:

# gpt show da0

Sure enough, there were my six partitions numbered 0 through 5:

  • 0 – EFI partition
  • 1 – Linux encrypted container
  • 2 – OpenBSD encrypted slice

I will be adding a fourth partition for DragonFlyBSD and will then be using disklabel to create the necessary filesystems to install things.  To do that, use the following gpt command:

# gpt add -s 377075713 da0

The command returns the name of the partition created.  In my case that was da0s3.  Now I use disklabel to create the filesystems:

# disklabel -B -r -w da0s3 auto
# disklabel -e da0s3

The -e switch pulls up a file in vi that you need to add the following lines to:

a:    1g    0    4.2BSD
b:    32g    *    swap
d:    *    *    HAMMER

Save the file and you should be good to move on to the next step.  I tried multiple times to get an encrypted filesystem working by following these instructions; however, I kept running into panics on boot and my search of the mailing list archives was fruitless so I ended up going with an unencrypted setup.

First, set up the filesystems for your DragonFlyBSD slice, mount them and copy the contents of the install thumb drive’s filesystems over to the new filesystems:

# newfs /dev/da0s3a
# newfs_hammer -L ROOT /dev/da0s3d
# mount /dev/da0s3d /mnt
# mkdir /mnt/boot
# mount /dev/da0s3a /mnt/boot
# cpdup /boot /mnt/boot
# cpdup / /mnt

Update your /mnt/boot/loader.conf to include a reference to your disk:

vfs.root.mountfrom=”hammer:/dev/da0s3d”

Update your /mnt/etc/fstab to include:

/dev/da0s3a /boot ufs rw 1 1
/dev/da0s3b none swap sw 0 0
/dev/da0s3d / hammer rw 1 1

Now we need to set up the EFI bootloader for rEFInd.  To do that mount the EFI partition and do the following:

# mkdir /tmp/boot
# mount_msdos /dev/da0s0 /tmp/boot
# cd /tmp/boot/EFI/BOOT
# cp /boot/boot1.efi bootx64_dragonflybsd.efi

I tried editing the refind.conf file in DragonFlyBSD but I got a bunch of unprintable characters in it so I rebooted to Arch and finished.  Add the following to /boot/EFI/refind/refind.conf:

menuentry “DragonFlyBSD” {
icon \EFI\refind\icons\os_dragonflybsd.png
loader \EFI\boot\bootx64_dragonflybsd.efi
}

Download the png file from here and put it in /boot/EFI/refind/icons.  Finally, unmount your partitions and reboot:

# cd /
# umount /tmp/boot
# umount /mnt/boot
# umount /mnt
# reboot

FreeBSD

If you intend to run DragonFlyBSD as well, STOP!!!  Please go back and read my note at the top of the DragonFlyBSD section.  You have to install it first and THEN install FreeBSD if you want things to work ok.  Back to your regularly scheduled programming…

OK.  Time for some uncharted territory for me.  I’ve installed one FreeBSD box that I use as a webserver and a mailing list server but I did it up in DigitalOcean so now I get to actually go through the nitty gritty of creating a similar setup that I have in Arch and OpenBSD.  I want to have encrypted swap and an encrypted filesystem and run Gnome 3 as my desktop environment.  But hey, its a BSD so how hard can this be?

First off, I downloaded the latest production memstick image file from the FreeBSD master FTP site.  I dd’ed it onto a thumb drive and booted from it.  (What a fancy installer you guys have BTW!)  I started the install process and when I got to Partitioning, I dropped out to a shell.  This being a BSD, I expected (and received) very detailed man pages on the commands I was about to run.

My boot drive was ada0 so a simple gpart list ada0 showed me the four partitions I already expected to be there (my EFI partition, my Linux encrypted container, my OpenBSD encrypted slice and my DragonFlyBSD encrypted slice).  Leveraging a HOWTO on the FreeBSD forums, I did the following to set up a swap partition, a boot partition (unencrypted and UFS) and my main encrypted ZFS filesystem partition:

# gpart add -t freebsd-swap -s 32G -a 1M -l FreeBSD-swap /dev/ada0        #(creates ada0p5)
# gpart add -t freebsd-ufs -s 10G -a 1M -l FreeBSD-ufsboot /dev/ada0        #(creates ada0p6)
# gpart add -t freebsd-zfs -s 170G -a 1M -l FreeBSD-enczroot /dev/ada0    #(creates ada0p7)

The next step was to set up the ufs boot partition and mount it so I can access it:

# newfs -L ufsboot -S 4096 /dev/ada0p6
# mkdir /tmp/ufsboot
# mount /dev/ada0p6 /tmp/ufsboot

Now I need to create and attach to my encrypted GELI container:

# mkdir -p /tmp/ufsboot/boot/geli
# dd if=/dev/random of=/tmp/ufsboot/boot/geli/ada0p7.key bs=64 count=1
# geli init -e AES-XTS -l 128 -s 4096 -b -K /tmp/ufsboot/boot/geli/ada0p7.key /dev/ada0p7
# cp /var/backups/ada0p7.eli /tmp/ufsboot/boot/geli/
# geli attach -k /tmp/ufsboot/boot/geli/ada0p7.key /dev/ada0p7

Next, it’s time to set up and configure the ZFS pool:

# zpool create -R /mnt -O canmount=off -O mountpoint=none -O atime=off -O compression=on zroot /dev/ada0p7.eli
# zfs create -o canmount=off -o mountpoint=none zroot/ROOT
# zfs create -o mountpoint=/ zroot/ROOT/master
# zfs create -o mountpoint=/usr/jails zroot/ROOT/master/jails
# zfs create -o mountpoint=/usr/local zroot/ROOT/master/local
# zfs create -o mountpoint=/usr/ports zroot/ROOT/master/ports
# zfs create -o mountpoint=/var zroot/ROOT/master/var
# zfs create -o mountpoint=/var/log zroot/ROOT/master/log
# zfs create -o mountpoint=/usr/home zroot/home
# zfs create -o mountpoint=/usr/obj zroot/obj
# zfs create -o mountpoint=/usr/ports/distfiles zroot/distfiles
# zfs create -o mountpoint=/usr/src zroot/src101
# zfs create -o mountpoint=/tmp zroot/tmp
# zfs create -o mountpoint=/var/tmp zroot/vartmp

The next step is to mount our filesystem where the installer can find it.  To do that do the following:

# mkdir /mnt/ufsboot
# umount /dev/ada0p6
# mount /dev/ada0p6 /mnt/ufsboot
# ( cd /mnt && ln -s ufsboot/boot boot )

Now, create the /tmp/bsdinstall_etc/fstab file for the installer with the following content:

# Device    Mountpoint    FStype    Options    Dump    Pass#
/dev/ada0p5.eli    none    swap    sw,ealgo=AES-XTS,keylen=128,sectorsize=4096    0    0
/dev/ada0p6    /ufsboot    ufs    rw    1    1

At this point, type exit to get back to that fancy installer so we can finish the OS installation.  During the install, I chose to add all of the security hardening options and created a non-root user who I added to the wheel group.

When given the ability to drop to a shell in the new system and make manual changes, choose to do that.  First, edit /boot/loader.conf and add the following:

aesni_load=”YES”
geom_eli_load=”YES”
geli_ada0p7_keyfile0_load=”YES”
geli_ada0p7_keyfile0_type=”ada0p7:geli_keyfile0″
geli_ada0p7_keyfile0_name=”/boot/geli/ada0p7.key”
zfs_load=”YES”
vfs.root.mountfrom=”zfs:zroot/ROOT/master”

Next, edit /etc/rc.conf and add this line to mount all ZFS filesystems at boot:

zfs_enable=”YES”

Finally, you need to add the first stage FreeBSD bootloader to your EFI partition:

# mkdir /tmp/efi
# mount -t msdosfs /dev/ada0p1 /tmp/efi
# cd /tmp/efi/EFI/BOOT
# cp /boot/boot1.efi bootx64_freebsd.efi

Edit the refind.conf file on your EFI partition to include the following:

menuentry “FreeBSD” {
icon \EFI\refind\icons\os_freebsd.png
loader \EFI\boot\bootx64_freebsd.efi
}

At this point, exit back to the installer and reboot.  If you did things correctly, you should see a third icon in rEFInd that allows you to boot to your new encrypted FreeBSD install!

NOTE:  When I was figuring all of this out, I ran into a really weird EFI Framebuffer problem using my ancient BIOS (this went away when I updated it) where it was trying to squash everything into one line of text at the top of the screen that I fixed by adding the following to /boot/loader.conf:

kern.vty=vt
i915kms_load=”YES”

The first thing to do after booting the new system is to update it:

# freebsd-update fetch
# freebsd-update install

After that you should probably reboot and log back in.  Next, I need to give my local user the ability to escalate privileges using sudo so I need to first install that:

# pkg install sudo
# visudo

To set up Gnome as my desktop environment, I first have to install xorg:

# pkg install xorg

This installs approximately 200 packages that make up the dependencies.  Next, install gnome itself:

# pkg install gnome3 gnome-shell-extra-extensions

The gnome package has almost 500 dependencies so it will take a while.  After that is complete, it’s time to do the necessary system configuration.  Gnome apparently requires that the /proc filesystem be mounted at boot time so add the following to /etc/fstab:

proc    /proc    procfs    rw    0    0

Now you need to enable dbus and HAL (required by Gnome).  To do this, add the following lines to /etc/rc.conf:

hald_enable=”YES”
dbus_enable=”YES”
gdm_enable=”YES”
gnome_enable=”YES”

A dependency of the gnome-tweak-tool is python and it isn’t installed by default.  You should add that now as well:

# pkg install python

Reboot at this point to let all of these changes take effect.  You should be at the gdm login prompt if all went well!  Install any interesting packages you’d like to use on the system (I’d recommend firefox, chromium, libreoffice, evolution and evolution-ews as a start.)

NetBSD

OK.  If FreeBSD was uncharted territory for me because I had only set up one server, NetBSD is going to be another planet as I’ve never used the operating system.  But hey, its a BSD so how hard can it be?

Guess what?  NetBSD as of 7.1 doesn’t support EFI.  However, 8.0 is right around the corner and it does.  Given that, I downloaded the latest nightly build (pro tip – look in the images directory for the install images not the amd64 one) and dd’ed it to a thumb drive. I spent a ton of time (and many wrecked partition tables) trying to get this onto my existing drive and wasn’t successful.  Given that, I’ll wait for 8.0 to be released and save space at the end of the drive for this operating system when it is released.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s