Where the rubber meets the road (part two)

So in my last post, I detailed what it took to get Arch Linux running on my brand spanking new Thinkpad X1 Carbon (4th Generation) and was getting ready to install OpenBSD on the same disk.  In the intervening time, I banged my head against the wall over and over again trying to figure out what my problem was on OpenBSD with not seeing the partition structure correctly on the disk and have finally given up.  Time to NUKE AND PAVE baby!

I backed up my data from Arch to an external drive, downloaded the latest snapshot installer for OpenBSD, burned it to a thumb drive and booted the laptop from it.  My goal is to reverse the approach I took last time and install OpenBSD first.  I’m going to GPT partition the disk and then, once I have it working correctly, reboot into the Arch installer and install Arch second.

After booting the OpenBSD installer, I bailed out to the (s)hell at the prompt and rewrote the partition on my 1TB NVMe drive with:

# fdisk -gi -b 4096000 sd0

This repartitions the drive with GPT and creates a boot EFI volume that is 2G in size.  I then used:

# fdisk -e sd0

to edit the drive and resized the OpenBSD partition to 490 GB in size.  Next, I used disk label to create my encrypted partitions for swap and data:

# disklabel -E sd0

I created an “a” partition (don’t forget to make it RAID type)  hat took the rest of the OpenBSD area for my encrypted data drive.  I then used bioctl to create the encrypted volume:

# bioctl -c C -l /dev/sd0a softraid0

The crypto volume was now available as /dev/sd2 so I used “exit” to get back to the installer and installed OpenBSD on that volume as I normally do.  To be safe, I mounted the EFI partition (/dev/sd0i in my case) and copied the /efi/boot files into /efi/openbsd.  I vaguely recall a Linux installer overwriting the OpenBSD files at some point in my checkered past so it never hurts to be safe.

I then reformatted my thumb drive as FAT32 and grabbed the missing firmware tarballs I needed to, rebooted the new OpenBSD install, and untarred them into /etc/firmware.  After rebooting and verifying that everything was working fine, I downloaded the latest Arch installer, burned it to the thumb drive and booted from that.

I installed Arch as I detailed in my last post; however, when I fired up gdisk I got a weird error message:

“Warning! Disk size is smaller than the main header indicates! Loading secondary header from the last sector of the disk! You should use ‘v’ to verify disk integrity, and perhaps options on the expert’s menu to repair the disk.”

Immediately after this, I saw a second warning:

“Caution: Invalid backup GPT header, but valid main header; regenerating backup header from main header.”

And, not to be outdone, there was a third:

“Warning! Main and backup partition tables differ! Use the ‘c’ and ‘e’ options on the recovery & transformation menu to examine the two tables.”

Finally (not kidding), there was a fourth:

“Warning! One or more CRCs don’t match. You should repair the disk!”

Given all of that, I thought to myself, “This is probably why I couldn’t see the disk properly when I partitioned it under Linux on the OpenBSD side.  I’ll let it repair things and I should be good to go.”  I then followed the recommendation and repaired things, using the primary GPT table to recreate the backup one.  I then installed Arch and figured I was good to go.

Now I was ready to put on rEFInd as my boot manager for switching between the two.  I installed the “refind-efi” package using pacman:

$ sudo pacman -S refind-efi

I then ran refind-install and had what I needed working.  I rebooted to OpenBSD and was greeted with a kernel panic because it couldn’t find my root volume.  Uh oh.

I repeated this process a couple of additional times (starting from scratch, partitioning under OpenBSD, installing Arch, repairing the disk, etc.) and not surprisingly got exactly the same results.

OK.  What to do now.  I decided to rock this old school and dump the whole crazy GPT thing and do this MBR.  After all, I only need three partitions anyhow (the EFI one, the luks encrypted partition for Linux and the softraid encrypted partition for OpenBSD).

I booted up off of the OpenBSD installer thumb drive, jumped out to the (s)hell and did:

# fdisk -i -b 4096000 /dev/sd0

I then resized down the default OpenBSD partition (#3 by the way) to half its size using fdisk, and used disklabel to create a single “a” slice that took up the entire area (make it RAID for the type).  Then, bioctl and I created the encrypted softraid container and I rejoined the installer and installed normally on the new virtual device (sd2 in my case).

Installing Arch was slightly tricker this time because apparently very few people use EFI and MBR together, let alone with luks and lvm.  🙂

The trick was realizing that I couldn’t use systemd-boot as my boot manager because it doesn’t work with MBR nor do they apparently have plans to make it work (I get it, this is the true definition of a corner case).  I dropped back to GRUB for my boot manager with the only trick being that I changed the GRUB_CMDLINE_LINUX variable in /etc/default/grub to be:

GRUB_CMDLINE_LINUX=”cryptdevice=/dev/nvme0n1p2:lvmpool”

I then ran the configuration file generator:

# grub-mkconfig -o /boot/grub/grub.cfg

At this point, I crossed my fingers and booted back and forth (using the F12 boot menu on the Thinkpad and choosing the nvme drive booted OpenBSD and I could choose “grub” from the boot list to boot Linux) between OpenBSD and Arch multiple times to ensure there wasn’t any funny business going on.

Victory!!!!!!!

Now that I have everything working, I’ll restore my config and data to Arch, configure OpenBSD the way I like it and get moving.  I’ll take some time and drop a note on the tech@ mailing list for OpenBSD to see if they can figure out what the GPT problem was I was running into.  Hopefully it will make that part of the code stronger to get an edge-case bug report like this.