OpenBSD at HiDPI!

Some of the recent work done by the core OpenBSD developers is really incredible.  Specifically, the work that has been done for EFI booting of more “modern” hardware has started to open up a whole new vista of machines that we can run our favourite operating system on.

I normally rock the Lenovo Thinkpad x220 and find it to be an outstanding machine for both general purpose computing and software development.  However, I have a Macbook Pro 11,1 (Haswell Retina laptop manufactured in late 2013) that I have tried (and failed) in the past to put OpenBSD on.  The challenge came from the fact that this particular laptop has USB 3 but not USB 2.  The symptom of this is that you are able to use the builtin keyboard when you hit the initial “boot” prompt, but after loading the installer, the keyboard is non-responsive.  Using an external USB keyboard didn’t help because the problem is in how the operating system talks to the USB 3 controller.

I started seeing traffic on the misc@ mailing list about people having success with Macbook Air models and the new 12″ Macbook (all very recent vintage machines), so I thought I’d give it a try again.  In the end I was able to get everything working really well with a Gnome desktop that did the right thing for the HiDPI retina screen and everything (well, almost everything, I did have to use a USB dongle for WiFi because the builtin one wasn’t recognized).

So given that I had a functional, nearly modern laptop what else should I do but reformat the hard drive and start over!  I figured that it would be a nice thing to do for the community to start this bad boy from scratch and document all of the steps along the way.  Therefore I turned it off, held down on Command+R and powered it up.  For the non cultists reading this, that puts the laptop in “Internet Recovery” mode where you can attach to a WiFi network and it will give you all of the tools to repartition the drive and install the operating system over the Internet.  Slooooowwwwwwlllllyyyyyy…..

Once I had the operating system reinstalled and verified that a reboot worked, I figured I was at a clean starting point. Since I want to make my life hard, I thought it would be fun to preserve Mac OS X on this machine and dual boot.  To make that happen, I need to shrink the partition and create a new one in the free space created. Unfortunately I could not get that to work to save my life – it always would try to boot the Boot Camp partition as bios and not as EFI which, on my laptop would hang trying to configure a mystery second processor (cpu1) in the system.

Given that, I decided to make my life simple and go back to a single-boot system (which I really wanted anyhow).  I leveraged the howto was written by Jasper Lievisse Adriaanse to get the EFI bits on the disk.  I’ll detail it step by step here.

First, I created a thumb drive with the latest install58.fs snapshot on it from my favorite Canadian mirror.  I then rebooted the laptop with the “Option” key held down and picked the “EFI Boot” partition off of the thumb drive.  You’ll be astonished how small the text is so break out those reading glasses if you are over 40 like me!

Once I hit the first prompt from the installer, I jumped to the shell and ran this:

# fdisk -i -b 960 sd0
Do you wish to write new MBR and partition table? [n] y

Now I like to run an encrypted boot drive, so from my favorite tutorial from the bsdnow.tv site, I did the following:


# disklabel -E sd0
> a b
offset: [1024]
size: [xxxxx] 32g
FS type: [swap]
> a a
offset: [xxxxx]
size: [xxxxx] *
FS type: [4.2BSD] RAID
> w
> q
# bioctl -c C -l /dev/sd0a softraid0

Make sure you pick a good passphrase that you can remember but would be hard to guess or brute force.  Make a note of what your CRYPTO volume is attached as.  In my case that is sd2 – this is the “drive” I will be installing to.

Now, run the “install” script and install as you normally would.  Make sure you install to the crypto drive that you just created.  I normally put 100g in place for just /usr (I don’t break it down) so I can install source and ports and have room to build and then use the remainder of the drive for /home.

Once the install has finished DO NOT REBOOT YET because there is some additional work to do from the EFI tutorial I referenced above.

# /mnt/sbin/newfs_msdos sd0i
# mount /dev/sd0i /mnt2
# mkdir -p /mnt2/efi/boot
# cp /mnt/usr/mdec/BOOTX64.EFI /mnt2/efi/boot

Now you should be free and clear to reboot the laptop.  If all goes well, you should be presented with the passphrase prompt to decrypt your boot partition. Be patient, it took nearly 30 seconds for it to come up on my system.  Reminds me of when you don’t have a default boot volume selected on OS X.  Oh well, a mystery for another day.

After the reboot and logging in as my root user, I modified /etc/fstab to add “,softdep,noatime” to all of my read/write partitions and also a line for /dev/sd0b as my swap partition.

Now that I have this done, since there isn’t a functional WiFi device (yet) on this platform, I plugged in my handy-dandy little ASUS USB-N10 USB adapter, installed the firmware from http://firmware.openbsd.org (it’s the rsu one) into the /etc/firmware directory using a thu, rebooted and configured my network:

# ifconfig rsu0 nwid MYNETWORK wpakey MYKEY
# ifconfig rsu0 up
# dhclient rsu0

Now I need to add my PKG_PATH to be able to install binary packages.  I added the following to my .profile file:

export PKG_PATH=http://openbsd.cs.toronto.edu/pub/OpenBSD/snapshots/packages/`machine -a`/

After logging out and back in, I then installed my desktop of choice, Gnome as follows based on this undeadly.org post:

# pkg_add gnome
# echo 'multicast_host=YES' >> /etc/rc.conf.local
# echo 'pkg_scripts="messagebus avahi_daemon gdm" >> /etc/rc.conf.local

A quick reboot then shows me I can log into Gnome.  What’s astonishing is that the HiDPI stuff in Gnome seems to be working perfectly out of the box!

Where do binary packages come from?

To follow up on my last post where I talked about the high-level concepts involved with OpenBSD ports & packages and why you would want to work on them (Time to make the donuts), I thought I would share an actual experience I had where there was a Python library I needed in OpenBSD that wasn’t there.

Create Directory

The library is dnslib and it was used by a security tool I was porting to OpenBSD.  I first started out by creating a subdirectory under the /usr/ports/net called “py-dnslib”.  I choce “net” because this is a network library and I prefixed the name of the library with “py-” because that is the convention I observed browsing around in the ports tree.

Create Makefile

The next step in the process is to create a basic Makefile.  Remember that in OpenBSD, for licensing reasons, the ports tree does not contain the actual source code of the ports themselves.  It provides information to the ports build subsystem on where to download the source and process it.  The best way to build a new Makefile is not from scratch, but by looking for something similar that already exists and modifying it.  My initial Makefile looked like this:

# $OpenBSD$

COMMENT=                library to en/decode DNS wire-format packets for Python

MODPY_EGG_VERSION=      0.9.4
DISTNAME=               dnslib-${MODPY_EGG_VERSION}
PKGNAME=                py-${DISTNAME}
CATEGORIES=             net

HOMEPAGE=               https://pypi.python.org/pypi/dnslib

MAINTAINER=             Bryan Everly <bryan@bceassociates.com>

# BSD
PERMIT_PACKAGE_CDROM=   Yes

MASTER_SITES=           https://pypi.python.org/packages/source/d/dnslib/

MODULES=                lang/python

FLAVORS = python3
FLAVOR ?=

do-test:
${SUBST_CMD} ${WRKSRC}/run_tests.sh
cd ${WRKSRC} && ./run_tests.sh

.include <bsd.port.mk>

To walk through this, the first line is a comment that describes (as briefly as possible) what this port does.  The next section contains the author’s version number of this library, the name of the distribution file to download from the source server, the package name that will be used in the ports tree and finally the subdirectory (category) this port lives in.

Next, we see the homepage of this project on the Internet and my information as the maintainer of this port going forward.  Please note that if you sign up to be a maintainer of a port it is a serious commitment.  You will be responsible for making sure that you keep the OpenBSD version of this port current and to work with the upstream authors to ensure that it works cleanly and well on OpenBSD.  It is not a commitment to be taken lightly.

We then see a comment that indicates that this is a BSD license for the library (you can find that out by looking at the source code for the library or its homepage).  Next we have a line that controls whether or not the license for this port would allow it to be included in the release CD-ROM images for OpenBSD.

MasterSites points at where the source code is going to be downloaded from.  In this case it is pypi.python.org that we are fetching the code from.  The lang/python modules line pulls in support in the build infrastructure to simplify and standardize the way Python packages are built.  We also use Python 3 as a “flavor” for the package.  Check the OpenBSD documentation to get a better understanding of flavors.

Finally, the “do-test” target executes the built-in unit tests that are included with the dnslib module from the upstream author.  Now that we have the Makefile skeleton in place, time to actually build and test this port.

Other bits and pieces

Now that we have a Makefile created, we also need to create some additional files:

  • patches – This subdirectory will contain patches we have to make to the downloaded source to get it to build successfully on Open BSD.  See below for more details.
  • pkg/DESCR – This file is a 72 column wide text file that contains a broader description of what the port/package is and how to use it.

What do I do first?

The OpenBSD build subsystem has many options.  It is definitely worth looking at the bsd.port.mk manpage to better understand the complexities that are available to you.  In our case, there are some simple targets of “make” that we will use to get this port built and tested.  You can just be crazy and leap right to “make install” but I like to take it one step at a time and correct any problems as they crop up.

  1. “make fetch” is the first target you want to try.  This will execute the part of the build system that pulls down the source code from the MasterSites location.  The files are placed into the /usr/ports/distfiles subdirectory in a specific directory based on your $DISTNAME from your Makefile.
  2. “make extract” is the next target to try.  This will take the compressed source archive that was downloaded in step 1 and extract it into the /usr/ports/pobj subdirectory in a directory controled by $WRKDIR from your Makefile (derived from your $PKGNAME).
  3. “make makesum” – This target creates a checksum hash of your downloaded source package and stores it in a “distinfo” file.  That way if someone upstream monkeys with the source archive, we won’t just blindly pull it down and build it.
  4. “make update-patches” – If you have to patch a file (and in our case we do have to patch the test script to run the tests with our default version of Python), the way you do it is to first copy the file in the /usr/ports/distfiles/${DISTDIR} to a file with a .orig suffix (i.e. you copy foo.sh to foo.sh.orig).  Then you perform your edits on the non suffixed file and run “make update-patches” from your ports directory.  It will then create the diff you need in the patches subdirectory and it will be preserved across build and clean cycles.
  5. “make build” is the next target to try.  This will actually build your source code into its final form.  Since this is a Python module, not much exciting stuff happens here.  If it were a C program, then you’d get to watch the compile and link steps whiz past.
  6. “make update-plist” – This target creates or updates a file called pkg/PLIST that contains the manifest of files that will be part of the package.  Any time you add or remove a file (or change its name) you need to run this target and (if you are updating) remove the pkg/PLIST.orig file that is created.
  7. “make do-test” runs our unit tests.  More on this later…
  8. “make fake” is a fun one.  Sine the binary packages are built from the ports tree (actually, when you build a port from source it really creates and installs a binary package to be precise) there is a “fake” directory that mirrors the root filesystem of an OpenBSD install that all of your files from your port get packaged up into.  This lives in /usr/ports/pobj/$WRKDIR/fake-${MACHINE_ARCH} and if you browse it you will see the usual subject directories (/etc, /usr, /var).  When this target completes, the fake directory should contain all of the files that will be added by your binary package in the directories in which they should land.
  9. “make package” is next.  It takes the fake directory and creates the binary package from it.  The resulting file is placed in /usr/ports/packages/${MACHINE_ARCH} (mine is amd64).
  10. “make install” is where all the work pays off.  This takes the binary package you created and installs it on your system.

Cleaning up

To clean each step along the way and move back in the process (sometimes a necessary thing to do if upstream updates its version number, etc.) there are a variety of targets that are pretty self-explanatory:

  • “make uninstall” – uninstall the binary package
  • “make clean=package” – removes the binary package from the /usr/ports/packages/${MACHINE_ARCH} directory
  • “make clean=fake” – removes the fake directory from the /usr/ports/pobj/$WRKDIR/fake-${MACHINE_ARCH} directory
  • “make clean=all” – undoes any work done by “make build”
  • “make clean=distfiles” removes the source archive downloaded from the $MASTER_SITES directory

Did it work?

Well, unfortunately there were two unit test failures in the test.  Since the OpenBSD project is not a fan of just hacking stuff through, I started working with the author of this module (known as “upstream” in the vernacular of the people) to see what could be done to fix the problems.  After a couple of back and forth sessions, he kindly found and corrected a pretty subtle bug that hadn’t been seen before.  I asked him to cut a new version and put it up on the repository and updated my Makefile to use the new version.

Now what?

At this point, I created a compressed tarball of the entire port directory (after testing that the new versions worked without error of course) and sent an email to the ports@ mailing list (ports@openbsd.org) with the tarball attached.  The syntax you should use in these emails to simplify things is as follows:

  • The subject line should start out with a tag in square brackets indicating if this is a new port or an update (in our case “[NEW]”
  • The tag should be followed by the name of the port subdirectory with its category (in our case net/py-dnslib)
  • The body of the message should include a description of the port (I just include the text from dist/DESCR) and a request for review and committing if everything looks good (I am not a commiter).

In the process of going back and forth on the mailing list, I got and incorporated more feedback to simplify things in my Makefile to finally generate the following file that was committed on my behalf:

# $OpenBSD: Makefile,v 1.1.1.1 2015/10/30 16:44:54 rpointel Exp $

COMMENT=        library to en/decode DNS wire-format packets for Python

MODPY_EGG_VERSION=    0.9.6
DISTNAME=        dnslib-${MODPY_EGG_VERSION}
PKGNAME=        py-${DISTNAME}
CATEGORIES=        net
MODPY_PI=        Yes

HOMEPAGE=        https://pypi.python.org/pypi/dnslib

MAINTAINER=        Bryan Everly <bryan@bceassociates.com>

# BSD
PERMIT_PACKAGE_CDROM=    Yes

MODULES=        lang/python

FLAVORS = python3
FLAVOR ?=

do-test:
${SUBST_CMD} ${WRKSRC}/run_tests.sh
cd ${WRKSRC} && ./run_tests.sh

.include <bsd.port.mk>

Wrap up

Well, that was fun wasn’t it?  While it seems like a lot of work, it is important to point out that this level of attention to detail and quality is what makes OpenBSD the stellar operating system that it is today.  I hope that you were able to learn a bit more about binary packages and how the ports system creates them.  As always, any questions or feedback would be welcome.

Time to make the donuts… ports…

One of the things I enjoy doing as I learn more about OpenBSD and the community around it is to help with the effort to port software to the platform that I personally find useful and think others will.  The ports system is where all of this third party software lives (/usr/ports) and you can read about it more here:

http://www.openbsd.org/faq/ports/ports.html

Keep in mind that the ports tree isn’t just an archaic way of building software from source, it is also where all of the binary packages come from. This is worth repeating – if you want a binary package, someone first had to create a port of it. 

The capabilities of the ports and packages system are pretty daunting when you first start to look at them.  Since I’ve muddled through a few ports successfully so far (with a lot of help from the community), I thought I would outline the general process I have followed here as a guide to others.

Step 1 – Figure out what you want to port

This may sound obvious, but you probably want to give some thought to what software that you find useful with an eye towards software that other people will find useful.  Sometimes, searching around on Google for folks complaining that X or Y isn’t available on OpenBSD can be helpful if you don’t have any solid ideas here.  The community wants the ports and packages for OpenBSD to be useful and will frequently prune things that don’t make sense or are hopelessly out of date.

Step 2 – Be ready to make a commitment

If you list yourself as the port maintainer, then you are taking on a serious responsibility to keep up with the upstream work on this piece of software and keep it current in the OpenBSD ports tree.  This can be time-consuming and will require you to work some to a lot with the “upstream” folks who actually author the software in the first place.

Step 3 – Be friendly to upstream

This is critically important.  A really poor OpenBSD port is one that has a bunch of patches in the ports subdirectory and more or less hacks the software together into some semblance of usefulness.  A truly good port is one where you have worked with the upstream authors and have contributed patches back to them that allow the OpenBSD version to build cleanly with no extra patching necessary.  By investing the time up-front, you not only help the upstream project but you also help yourself in the future as you maintain the port going forward.

Step 4 – Be polite to the kind folks at ports@

Since you are not likely an OpenBSD committer, you will need to get help from the people on the ports@openbsd.org mailing list to get your changes committed to the source tree.  Sometimes people can be in a hurry or focused on other things and maybe you won’t get a quick response to your port.  Be patient and polite and you will eventually get where you are.  If you don’t, it might be an indication that this particular piece of software isn’t terribly interesting to the community.

Step 5 – Read the documentation

There is a tremendous amount of very good documentation available online as part of the OpenBSD website.  Read it.  Then read it a second time.  Then when you hit problems, read it a third time.  If you are still stuck, politely request some help on either IRC or the ports@ mailing list and tell them what you have tried so far.  If you genuinely show an effort was made and you are just stuck, you are far more likely to find someone who will help you out.

How does a port work?

This was interesting for me to learn.  The /usr/ports directory tree doesn’t actually contain any source code (outside of patch files and some scripts).  The ports system (when building from source) pulls the source code from the upstream repository, deposits it into a /usr/ports/distfiles/PORT-v.v.v subdirectory and works on it in a subdirectory called /usr/ports/pobj/PORT-v.v.v.  This allows OpenBSD to have a light footprint and not pull down tons of source code that may or may not be of interest to the person building the machine.

The ports directory is subdivided based on the type of software.  For example, developer tools go in “devel”, database tools go in “database”, etc.  Not too tricky is it?  Each port then lives in a subdirectory under its category name.

The directory for a given port contains several standard files.  There is a “Makefile” which contains a very abbreviated set of build instructions that heavily leverages the ports infrastructure kindly provided by the OpenBSD developers.  There is a “distinfo” file that contains a manifest of data that validates the source file downloaded from upstream is the one the port maintainer was expecting.  Finally there is a pkg subdirectory with two files – “DESCR” and “PLIST”.  The “DESCR” file contains a brief description of what the port is and the “PLIST” file contains a package manifest of where the various components of the port are installed in a working system.

Most ports include a file at the bottom of their Makefile called “bsd.port.mk”.  Read the manpage for this file as it really does an excellent job of describing how the ports infrastructure works.  Given everything that you can build on OpenBSD from the ports tree, it is astonishing how well thought-out the infrastructure is.

That’s all for this posting.  Next time, we’ll get into the meat of how you actually port something with a real-world example of something I just finished porting.