Thinkpad T14 (AMD) Gen 2 – A Brave New World!

As long-time readers of this blog are aware, I’m a bit of a Thinkpad fanatic. I fell in love with these durable machines when I was working for IBM back in the late 90’s and accidentally had one fall out of my bag, bounce down the jetway stairs and hit the runway hard – amazingly enough it had a few scuffs but zero damage! After the purchase of the brand by Lenovo, I was a bit worried, but they continue to crank out (at least in the Thinkpad T and X model lines) high-quality, powerful machines.

Thinkpad T480 – RIP

I ran into a nasty problem with my Thinkpad T480 where the software on the machine actually physically damaged the hardware. I know! I thought that was impossible too (other than the 70’s PET machine that had a software-controlled relay on the motherboard that you could trigger continuously until it burned out) but nope – the problem is real.

Essentially, the Thunderbolt I/O port on the machine is driven by firmware running out of an NVRAM chip on the motherboard that can be software-updated as new firmware comes out. As with any NVRAM chip, there are a finite number of write-cycles before the chip dies, but the number of times you will update your firmware is pretty small so it works out well.

Unfortunately, Lenovo pushed out a firmware update that wrote continuously to the NVRAM chip and if you didn’t patch fast enough (they did release an urgent/critical update), then the write-cycles would be exceeded, the chip would fail and the bring-up code would not detect the presence of the bus and thus you had no more Thunderbolt on the laptop. Well, I didn’t update fast enough so “boom” – it is now a Thunderbolt-less laptop.

The New T124 (AMD) Gen 2

Well, enter the need for a new laptop. I decided to jump ship from the Intel train and try life out on the “other side” but ordering a Thinkpad T14 (AMD) Gen 2 machine with 16gb of soldered RAM (there is a slot that I will be populating today that can take it up to 48gb max – I’m going with 32gb total by installing an $80 16gb DIMM) and the Ryzen Pro 5650U that has 6 cores and 12 threads of execution. The screen was a 1920×1080 400 nit panel and looks really nice.

When the laptop showed up, I booted the OpenBSD installer from 6.9-current and grabbed a dmesg and discovered that I lost the Lenovo lottery and had a Realtek WiFi card in the machine. Well, the good news was that I had upgraded the card in my T480 to an Intel AX200 so I swapped it for the one I took out of the T480 and then used it in the T14 to replace the Realtek card. Worked like a charm.

The Ethernet interface on this machine is a bit odd. It’s a Realtek chipset as well, but it shows up as two interfaces (re0 and re1). The deal is that re0 is the interface that is exposed when the machine is plugged into a side-connecting docking station and re1 is the interface that is connected to the built-in Ethernet port. The device driver code that is in 6.9-current as of this writing works just fine with it, however, so I’m happy.

Now for the bad news. Every Thinkpad I have owned for the last decade allows me to plug an m.2 2240 SATA drive into the WWAN slot and it works great. I assumed that would be the case with this machine. While I had the bottom off to replace the WiFi card, I slipped the 1TB drive from the WWAN slot of my T480 into the WWAN slot of the T14 and booted up. I was immediately presented with an error message stating effectively that the WWAN slot was white-listed by Lenovo and would only accept “approved” network cards. I was beyond frustrated by this.

Given that I want to get this machine into my production workflow, I decided that I’d slog along for the time being by putting a larger m.2 2280 NVMe drive in, installing rEFInd to allow me to boot multiple partitions from a single drive and then clone the 512gb drive that is in the machine to the 1GB drive out of the T480. Then, the remaining space on the new drive will contain an encrypted partition for my OpenBSD install.

Installing rEFInd

I followed the instructions from the rEFInd site on how to manually install under Windows 10 and the steps I followed included downloading and unpacking the ZIP file and then running the following commands from an administrative command prompt:

C:\Users\xxxx\Downloads\refind-bin-0.13.2\> mountvol R: /s
C:\Users\xxxx\Downloads\refind-bin-0.13.2\> xcopy /E refind R:\EFI\refind\
C:\Users\xxxx\Downloads\refind-bin-0.13.2\> r:
R:\> cd \EFI\refind
R:\EFI\refind\> del /s drivers_aa64
R:\EFI\refind\> del /s drivers_ia32
R:\EFI\refind\> del /s tools_aa64
R:\EFI\refind\> del /s tools_ia32
R:\EFI\refind\> del refind_aa64.efi
R:\EFI\refind\> del refind_x64.efi
R:\EFI\refind\> rmdir drivers_aa64
R:\EFI\refind\> rmdir drivers_ia32
R:\EFI\refind\> rmdir tools_aa64
R:\EFI\refind\> rmdir tools_ia32R:\EFI\refind\> rename refind.conf-sample refind.conf
R:\EFI\refind\> mkdir images
R:\EFI\refind\> copy C:\Users\xxx\Pictures\mtstmichel.jpg images
R:\EFI\refind\> bcdedit /set "{bootmgr}" path \EFI\refind\refind_x64.efi

That next to the last line is because I wanted to have a picture of my “happy place” (Mount Saint Michel off of the northern coast of France) as the background for rEFInd. I edited the refind.conf file and added the following lines:

banner images\mtstmichel.jpg
banner_scale fillscreen

A quick reboot shows that rEFInd is installed correctly and has my customized background. Don’t be alarmed that the first time you boot up with rEFInd is slow, I think it is doing some scanning and processing and caching because the second and subsequent boots are faster.

Cloning the Drives

The process that I am going to follow, at a high level, is to first clone the contents of my primary 1TB 2280 NVMe drive in my T480 to a spare 256GB drive. I will then erase the 1TB drive and clone the contents of my T14’s drive to it (it’s only 512GB). I will then erase the 512GB drive and clone the 256GB drive back to it. Finally, for good operational security (OpSec) purposes, I’ll use the open source Windows program Eraser erase the 256GB drive. At this point I should have a bootable T480 (with a fried Thunderbolt bus – grr…) on the 512GB drive, and a bootable T14 on the 1TB drive.

I’m using Clonezilla, an open source tool that I burn to a bootable USB drive to do the cloning. For hardware that I am using to accomplish all of this, first I use a Star Tech device that allows me to plug m.2 drives into a little box that then acts as a 2.5 inch SSD drive. I plug that into a Wavlink USB drive docking station that can hold either 3.5″ or 2.5″ drives.

Another piece of software that I use as part of this process is GPartEd Live – an open source tool that allows you to create a USB drive that boots into the GPartEd software (the Gnu Partition Editor). This allows me to view the partition structure of one drive and create an analagous partition structure on another drive. The built-in tools for Windows to do this work (Disk Manager for example) can create hidden partitions under the covers that can cause problems with this process. I prefer to use GPartEd to ensure that I can see and control everything that is going on.

Step One is to take the T480, boot it into Windows and connect the Wavlink device to it with the 256GB NVMe drive plugged into it via the StarTech adapter. While I’m using Eraser to wipe the 256GB drive, I also go into Windows settings and decrypt the Windows disk by turning off BitLocker for it. This may not be necessary but it makes me feel more comfortable to do the cloning with unencrypted Windows drives because the key for the encryption is store in the TPM device on the motherboard and I’m not sure if the fact that the underlying hardware changes would muck that up. After the erase and decrypt is finished, I shrank the partition using “Disk Management” on Windows to be smaller than the new physical disk. If you don’t do this, then Clonezilla won’t allow you to clone from a larger partition to a smaller one.

Next we will need to reboot the machine to GPartEd Live. For the destination drive, you will need to use the “Device” menu and create a new GPT partition table. Take a look at the source drive and make a note of the various partitions, their flags, and their sizes. On the destination drive, recreate that partition structure with the same flags and the same or slightly larger size. I generally bump up the size of the partition by just a bit in order to avoid getting into trouble with rounding the size for display on the screen. If you get it wrong, don’t worry, Clonezilla will yell at you and you’ll have to go back and do this over again. 🙂

When launching Clonezilla, since I have the high resolution display on the T480 (a mistake I’ll never make again, HiDPI is a PITA in everything but Windows) I had to use my cell phone to zoom in on the microscopic text and select the “use 800×600 with large fonts from RAM” option. With readable text, I then make sure that I’m choosing “device-device” from the first menu (not the default). Next, select “Beginner Mode” to reduce the complexity of the choices you’ll have to make. After that, you want to select “part_to_local_part” to clone from one partition on the source drive to the corresponding partition on the destination drive. Finally, select the source partition and the destination partition. I recommend you do the smaller partitions first and then let the main C: partition (the largest one) grind because it can take a long time to clone.

After cloning the T480 drive, I removed it from the machine and was ready to clone the T14’s drive to it. This is where I ran into a “keying” problem with m.2 drives. Some are “B” keyed, and some are “B+M” keyed. This refers to the number of cutouts where they plug into the slot. Well, it looks like the NVMe drives in both the T480 and the T14 don’t fit the StarTech adapter. After some juggling around I found an old 256MB drive that I was able to use to get the swap completed.

Creating the OpenBSD Partition

To do this, I will use “Disk Manager” on Windows and shrink the NTFS partition (if necessary) to make room for OpenBSD and then create a new partition on the drive that takes up the remaining space. If you check the “don’t assign a drive letter” box and the “don’t format the partition” box, you’ll get a raw, unformatted partition that takes up the remaining space on the disk.

That new raw partition will be changed in OpenBSD to be the home of the encrypted slice on which I’ll be installing the operating system. After creating that partition, it’s time to download the 6.9-current .IMG file for the latest snapshot and use Rufus on Windows to create the USB drive and reboot from it.

Once in the OpenBSD installer, drop immediately to the shell and convert that NTFS partition into an OpenBSD partition. That will be where we we put the encrypted slice that we will be installing to. To do this, run the following commands:

# cd /dev
# sh ./MAKEDEV sd0
# fdisk -E sd0

sd0: 1> print
sd0: 1> edit 4
Partition id: A6
Partition offset <ENTER>
Partition size <ENTER>Partition name: OpenBSD
sd0*: 1> write
sd0: 1> exit

The print command above should show you the 4 partitions on your drive (the EFI partition, the Windows partition, the WindowsRecovery partition and your fourth partition that will hold OpenBSD that you created above).

Now that you have a partition for OpenBSD, you’ll want to copy the EFI bootloader over to your EFI drive. You’ll later make a configuration change in rEFInd to not only display it on the screen, but also show a cool OpenBSD “Puffy” logo for it!

# cd /dev
# sh ./MAKEDEV sd1
# mount /dev/sd1i /mnt
# mkdir /mnt2
# mount /dev/sd0i /mnt2
# mkdir /mnt2/EFI/OpenBSD
# cp /mnt/efi/boot/* /mnt2/EFI/OpenBSD
# umount /mnt
# umount /mnt2

Now that you have an OpenBSD EFI bootloader in its own directory on the EFI partition, you’ll want to create the encrypted slice for the operating system install:

# disklabel -E sd0

sd0> a a
sd0> offset: <ENTER>
sd0> size: <ENTER>
sd0> FS type: RAID
sd0*> w
sd0> q

# bioctl -c C -l sd0a softraid0
New passphrase: <your favorite passphrase>
Re-type passphrase: <your favorite passphrase>

Pay attention to the virtual device name that bioctl spits out for your new encrypted “drive”. That’s what you will tell the OpenBSD installer to use. To re-enter the installer, type “exit” at the command prompt. Do your install of the operating system as you normally do. When you reboot, go into Windows.

First, download an icon for OpenBSD from here (or pick your favorite elsewhere). Next, bring up an administrative command prompt and use the following commands to mount the EFI partition and add the icon for OpenBSD:

C:\Windows\system32> mountvol R: /s
C:\Windows\system32> r:
R:> cd \EFI\refind
R:\EFI\refind> copy "C:\Users\<YOUR USER>\Download\495_openbsd_icon.png" icons\os_openbsd.png

Save your changes, exit notepad and then reboot. rEFInd is smart enough to find your OpenBSD partition and use the icon you just added. When you select it from the rEFInd UI, you should be presented with your OpenBSD encrypted disk password and be able to boot for the first time. I ran into a weird thing with my snapshot where it couldn’t download the firmware. I formatted a USB thumb drive as FAT32, downloaded the amdgpu, iwx, uvideo and vmm firmware from the site, mounted the drive in my OpenBSD system and ran fw_update -p /mnt to get the firmware.

At this point, you should be able to reboot and select either Windows or OpenBSD from your rEFInd interface. My hope is that Lenovo will remove this absurd white-listing of the WWAN devices from their UEFI/BIOS code and I’ll be able to plug drives into it again; however, if (and this is more likely) they do not, I’ll at some point buy a 2TB m.2 NVMe drive for this machine, repeat this process and be able to add Linux to it.

I hope folks find this guide helpful.

OpenBSD 6.9 – Help with the “Failed to install bootblocks” issue

Hi everyone!

I purposely chose a non-catchy title so that it would be more easily found by the search engines as this one has been a challenge for me in my last several laptop installs and I always manage to fix it after fiddling around for a while. This time around, I thought I’d actually produce a decent (hopefully!) write-up on just how I go about addressing the problem from scratch. This will provide two benefits: 1) I’ll have a nice step by step the next time I install my machine <grin>; and 2) It might help some other intrepid soul who is running into the same issue!

While the FAQ is always the best place to go for the most up to date steps on formatting and installing a system, I tend to run a “weird” setup that it seems like confounds the installer and most easily-accessible information. What I normally do in my Thinkpad laptops is install a second (or third) SSD or NVMe drive and then dedicate the entire disk to a given operating system. For example, if I’m running Windows 10 and OpenBSD 6.9 on my Thinkpad T480, I install Windows on the first drive (so that if my machine falls into evil hands and they power it on, it will just default boot into Windows and they might not even suspect OpenBSD is on the machine) and then I install OpenBSD onto the second drive. I then use the UEFI or BIOS boot menu to choose the OpenBSD drive to boot from.

Install Windows

I started off by installing Windows from a USB key to the primary drive in the laptop. As is my custom, after install, I put on all of the drivers and used the group policy editor to increase the BitLocker encryption from 128-bit AES to 256-bit AES. I also edited the registry to allow Outlook’s OST file to expand beyond the pitiful limit that it defaults to. After a reboot, I start the BitLocker encryption process and connect my email accounts.

If you are installing OpenBSD on a drive that has previously had something on it, it’s always a good idea to erase that drive. I use an open source tool for Windows called Eraser if I’m on Windows or good old dd if I’m on Linux. Eraser’s UI is a bit weird. It requires that you create a task that you can “run manually”, select the disk to be erased (in my cased “Hard disk 1”) and then select an erasure method (I use Pseudorandom 1-pass), then run the task manually.

I then download the install69.img file from my favorite mirror (https://openbsd.cs.toronto.edu/pub/OpenBSD) and use Rufus to transfer it to a bootable USB drive. I reboot, hit <F12> to get a boot menu from the UEFI, select my USB drive and then boot into the OpenBSD installer.

Install OpenBSD

The first thing I do is look at my dmesg to see what devices my drives have been attached to:

# dmesg | grep -i sd

This shows (in my case) that my Windows drive is connected to sd0, my blank drive that I will put OpenBSD on is connected to sd1 and my USB installer device is connected to sd2. Next, I need to create the necessary /dev devices:

# cd /dev
# sh ./MAKEDEV sd1
# sh ./MAKEDEV sd2

If you do a quick ls, you should see that the MAKEDEV script created the necessary device files and you should be good to proceed to the next step. Next, we want to initialize the sd1 drive to a GPT partitioning scheme and create the initial EFI partition on the disk. Fun fact, the EFI partition (while its own partition type) is formatted using FAT32 so thanks Windows 95! Here’s how you do this:

# fdisk -iy -g -b 960 sd1
# newfs_msdos /dev/rsd1i

Note my use of the /dev/r device (the raw device) and not the /dev/sd1i (normal device) in that second command. I’m not entirely sure if that is necessary, but the nice Reddit post that sparked me to think about how to do this did so why not, eh? If you get a weird error message trying to run newfs_msdos, it is likely that you have some previous partitioning data on that drive and it would be a good idea to completely erase it (see above).

Now, we need to mount the new partition, create the necessary directory structure that UEFI looks for and put the UEFI loader file from our installer USB drive into that directory:

# mount /dev/sd2i /mnt
# mount /dev/swd1i /mnt2
# mkdir -p /mnt2/efi/boot
# cp /mnt/efi/boot/* /mnt2/efi/boot

Now, we need to create the slice in the OpenBSD partition for the encrypted filesystem (you can skip this if you want to not have an encrypted drive):

# disklabel -E sd1

a a [ENTER]
offset: the default given
size: *
type: RAID
w [ENTER]
q [ENTER]

At this point, we have a slice set up as type “RAID” so we need to use the bioctl program to set up the encryption information along with the drive’s encryption password:

# bioctl -c C -l /dev/sd1a softraid0

You should see in the response to the above command the name of the new “virtual” encrypted disk. That is the disk that you will be installing OpenBSD onto. When you reach the question in the installation program about “Which disk is the root disk?”, enter that value (in my case, sd4). When i tasks whether or not you want to “Use (W)hole disk MBR, whole disk (G)PT or (E)dit?”, pick the MBR option (I know, this is counter intuitive but trust me here).

After the installer reboots the system, I press the [F12] key to get the boot menu (your key might be different if you aren’t running a Thinkpad) and select the disk I have installed OpenBSD on. I am immediately presented with the password prompt to decrypt the encrypted slice “virtual” disk and, upon entering it, I get the boot prompt. Everything proceeds as normal from that point forward and I am presented with the login prompt for my new system.

Updated Laptop Setup

If you are still with me and want to see how I set up my OpenBSD desktop (I get criticized slightly for making it “too heavy” with “too many packages” but I have to use Ubuntu as well for what I do and I like to have the UI be as consistent across the two operating systems as I can. Therefore I install Gnome 3 along with some gnome tweaks and plugins that give me the same theme and dock as Ubuntu.

To start out, I log in as root and enable my user account:

# echo "permit persist keepenv [my_non_root_user] as root" > /etc/doas.conf

At this point, I log out and back in as my unprivileged user account and work from there using the doas command to escalate privileges when needed. I start out by updating my system:

$ doas syspatch

Now, set up power management (this is a laptop):

$ doas rcctl enable apmd
$ doas rcctl set apmd flags -A
$ doas rcctl start apmd

I also add the following line to /etc/rc.conf.local (I haven’t cracked the code on how to do this with rcctl yet):

ntpd_flags=""

Now I need to make sure that I have the right level of resources available to my non-privileged user for tools like nextcloudclient (which opens a TON of files during its synchronization process). To do this I typically put myself in the “staff” and “operator” groups:

$ doas usermod -G staff MY_USERNAME
$ doas usermod -G operator MY_USERNAME
$ doas usermod -L staff MY_USERNAME

I then make the following changes to the “staff” section in /etc/login.conf:

...
staff:\
  :datasize-cur=4096M\
  :datasize-max=infinity\
  :maxproc-max=512:\
  :maxproc-cur=256:\
  :openfiles-max=102400:\
  :openfiles-cur=102400

I then have to add a line to /etc/sysctl.conf to take complete the work on allowing more open files on this system:

kern.maxfiles=102400

Now that I have modified all of this stuff and patched the system, it’s a good time to reboot.

Next, I add all of the packages I can’t live without (I know it seems like a small list, but they pull in a lot of others):

$ doas pkg_add gnome gnome-tweaks gnome-extras firefox chromium libreoffice nextcloudclient keepassxc \
   aisleriot evolution evolution-ews tor-browser shotwell gimp vim colorls cups reposync

A few changes to /etc/rc.conf.local are needed to boot into Gnome3:

$ doas rcctl disable xenodm
$ doas rcctl enable multicast messagebus avahi_daemon gdm cupsd

To avoid taking a kernel panic in my use-case (I have multiple monitors connected through a Lenovo Thunderbolt/USB-C dock), I have to manually switch to the Intel DRM driver in my /etc/X11/xorg.conf by adding the following section:

Section "Device"
  Identifier "Intel Graphics"
  Driver "intel"
EndSection

At this point, it’s time to reboot and go into GUI land. If you run into a situation where you have a monitor mirrored and no way to turn that feature off, I have found that turning all of the monitors off and back on generally fixes things. Once I have everything the way I would like it, I then download the yaru-remix-complete theme and install it manually by doing this:

$ cd ~
$ mkdir .themes
$ cd .themes
$ mv ~/Downloads/yaru-remix-complete-20.04.tar.xz .
$ unxz yaru-remix-complete-20.04.tar.xz
$ tar xf yaru-remix-complete-20.04.tar
$ mv themes/* .
$ rmdir themes
% doas mv icons/* /usr/local/share/icons
$ rmdir icons
$ doas mv wallpaper/* /usr/local/share/backgrounds/gnome
$ rmdir wallpaper
$ rm yaru-remix-complete-20.04.tar

Now launch gnome-tweaks and from the “Extensions” tab, turn on “user-themes”. Restart gnome-tweaks, go to the “Appearance” tab and select “Yaru-remixt” for applications, icons, and shell. On the “Top Bar” tab, enable “Battery Percentage” and “Weekday”. In the “Window Titlebars” tab, enable “Maximize” and “Minimize”.

Next, we want to put the wonderful extension Dash-To-Dock into the environment. To download it, go to https://extensions.gnome.org/extension/307/dash-to-dock/ and pick the right sehll version and extension version to match your install of Gnome shell. You will have to manually install it because the Gnome shell extension integration doesn’t appear to be enabled for OpenBSD:

$ cd ~/Downloads
$ unzip dash-to-docmicxgx.gmail.com.v67.shell-extension.zip
$ cat metadata.json

The value for “uuid” in that file is what you want to use in the next step:

$ mkdir -p ~/.local/share/gnome-shell/extensions/dash-to-dock@micxgx.gmail.com
$ cd ~/.local/share/gnome-shell/extensions/dash-to-dock@micxgx.gmail.com
$ unzip ~/Downloads/dash-to-docmicxgx.gmail.com.v67.shell-extension.zip

At this point, reboot to pick up the changes you’ve made, log in and launch gnome-tweaks again. On the “Extensions” tab, enable dash to dock. From the settings gear icon, select “extend to edge” and “show on all monitors” and you should have a very serviceable dock that is quite similar to the one in Ubuntu.

I then switch the terminal to “White on Black” for a better look and a 16-point font, and pin my favorite apps to the dock. Now for some terminal-level tweaks. I typically edit my ~/.profile file and add a couple of things:

export PS1="\[033[01;32m\]\u@\h\[\033[00m\]:\[033[01;34m\]\w[\033[00m\]$ "
export ENV=$HOME/.kshrc
export CVSROOT=/home/cvs

I then edit the ~/.kshrc file to add some aliases:

alias ls="colorls -G"
alias vi="vim"

A couple of other changes I typically make include turning off suspend when I’m plugged in (Settings | Power | Automatic Suspend), setting Firefox as my default browser (Settings | Default Applications), and setting my Time Format to “AM/PM” instead of “24-hour” (Settings | Date & Time).

I also take a moment to switch to “View -> User Interface -> Tabbed” in the Write, Calc, and Present applications in LibreOffice. This gives an interface reminiscent of the one in Microsoft Office – which I find helpful in terms of standardizing my workflow across operating systems.

After installing the appropriate browser security plugins and configuration changes from my favorite https://privacytools.io site, it’s time to set up CVS on my system for development purposes. To do this, I always double-check the AnonCVS link from the OpenBSD website left navigation panel and follow the steps to:

  1. Pre-load the source tree (for src, sys, ports and xenocara)
  2. Follow the instructions to give your non-root user write access to the src, ports and xenocara directories
  3. Mirror the repository with reposync (Note: I have had the best luck using anoncvs.comstyle.com as my mirror)

I then typically will add a crontab entry to keep things in sync:

$ doas crontab -e

...
0    */4    *    *    *    *    -n su -m cvs -c "reposync rsync://anoncvs.comstyle.com/cvs /home/cvs"

After syncing up my NextCloud data and my email data, I now have what I consider to be a secure, fully-functional OpenBSD laptop, configured the way I like it.