To follow up on my last post where I talked about the high-level concepts involved with OpenBSD ports & packages and why you would want to work on them (Time to make the donuts), I thought I would share an actual experience I had where there was a Python library I needed in OpenBSD that wasn’t there.
Create Directory
The library is dnslib and it was used by a security tool I was porting to OpenBSD. I first started out by creating a subdirectory under the /usr/ports/net called “py-dnslib”. I choce “net” because this is a network library and I prefixed the name of the library with “py-” because that is the convention I observed browsing around in the ports tree.
Create Makefile
The next step in the process is to create a basic Makefile. Remember that in OpenBSD, for licensing reasons, the ports tree does not contain the actual source code of the ports themselves. It provides information to the ports build subsystem on where to download the source and process it. The best way to build a new Makefile is not from scratch, but by looking for something similar that already exists and modifying it. My initial Makefile looked like this:
# $OpenBSD$
COMMENT= library to en/decode DNS wire-format packets for Python
MODPY_EGG_VERSION= 0.9.4
DISTNAME= dnslib-${MODPY_EGG_VERSION}
PKGNAME= py-${DISTNAME}
CATEGORIES= net
HOMEPAGE= https://pypi.python.org/pypi/dnslib
MAINTAINER= Bryan Everly <bryan@bceassociates.com>
# BSD
PERMIT_PACKAGE_CDROM= Yes
MASTER_SITES= https://pypi.python.org/packages/source/d/dnslib/
MODULES= lang/python
FLAVORS = python3
FLAVOR ?=
do-test:
${SUBST_CMD} ${WRKSRC}/run_tests.sh
cd ${WRKSRC} && ./run_tests.sh
.include <bsd.port.mk>
To walk through this, the first line is a comment that describes (as briefly as possible) what this port does. The next section contains the author’s version number of this library, the name of the distribution file to download from the source server, the package name that will be used in the ports tree and finally the subdirectory (category) this port lives in.
Next, we see the homepage of this project on the Internet and my information as the maintainer of this port going forward. Please note that if you sign up to be a maintainer of a port it is a serious commitment. You will be responsible for making sure that you keep the OpenBSD version of this port current and to work with the upstream authors to ensure that it works cleanly and well on OpenBSD. It is not a commitment to be taken lightly.
We then see a comment that indicates that this is a BSD license for the library (you can find that out by looking at the source code for the library or its homepage). Next we have a line that controls whether or not the license for this port would allow it to be included in the release CD-ROM images for OpenBSD.
MasterSites points at where the source code is going to be downloaded from. In this case it is pypi.python.org that we are fetching the code from. The lang/python modules line pulls in support in the build infrastructure to simplify and standardize the way Python packages are built. We also use Python 3 as a “flavor” for the package. Check the OpenBSD documentation to get a better understanding of flavors.
Finally, the “do-test” target executes the built-in unit tests that are included with the dnslib module from the upstream author. Now that we have the Makefile skeleton in place, time to actually build and test this port.
Other bits and pieces
Now that we have a Makefile created, we also need to create some additional files:
- patches – This subdirectory will contain patches we have to make to the downloaded source to get it to build successfully on Open BSD. See below for more details.
- pkg/DESCR – This file is a 72 column wide text file that contains a broader description of what the port/package is and how to use it.
What do I do first?
The OpenBSD build subsystem has many options. It is definitely worth looking at the bsd.port.mk manpage to better understand the complexities that are available to you. In our case, there are some simple targets of “make” that we will use to get this port built and tested. You can just be crazy and leap right to “make install” but I like to take it one step at a time and correct any problems as they crop up.
- “make fetch” is the first target you want to try. This will execute the part of the build system that pulls down the source code from the MasterSites location. The files are placed into the /usr/ports/distfiles subdirectory in a specific directory based on your $DISTNAME from your Makefile.
- “make extract” is the next target to try. This will take the compressed source archive that was downloaded in step 1 and extract it into the /usr/ports/pobj subdirectory in a directory controled by $WRKDIR from your Makefile (derived from your $PKGNAME).
- “make makesum” – This target creates a checksum hash of your downloaded source package and stores it in a “distinfo” file. That way if someone upstream monkeys with the source archive, we won’t just blindly pull it down and build it.
- “make update-patches” – If you have to patch a file (and in our case we do have to patch the test script to run the tests with our default version of Python), the way you do it is to first copy the file in the /usr/ports/distfiles/${DISTDIR} to a file with a .orig suffix (i.e. you copy foo.sh to foo.sh.orig). Then you perform your edits on the non suffixed file and run “make update-patches” from your ports directory. It will then create the diff you need in the patches subdirectory and it will be preserved across build and clean cycles.
- “make build” is the next target to try. This will actually build your source code into its final form. Since this is a Python module, not much exciting stuff happens here. If it were a C program, then you’d get to watch the compile and link steps whiz past.
- “make update-plist” – This target creates or updates a file called pkg/PLIST that contains the manifest of files that will be part of the package. Any time you add or remove a file (or change its name) you need to run this target and (if you are updating) remove the pkg/PLIST.orig file that is created.
- “make do-test” runs our unit tests. More on this later…
- “make fake” is a fun one. Sine the binary packages are built from the ports tree (actually, when you build a port from source it really creates and installs a binary package to be precise) there is a “fake” directory that mirrors the root filesystem of an OpenBSD install that all of your files from your port get packaged up into. This lives in /usr/ports/pobj/$WRKDIR/fake-${MACHINE_ARCH} and if you browse it you will see the usual subject directories (/etc, /usr, /var). When this target completes, the fake directory should contain all of the files that will be added by your binary package in the directories in which they should land.
- “make package” is next. It takes the fake directory and creates the binary package from it. The resulting file is placed in /usr/ports/packages/${MACHINE_ARCH} (mine is amd64).
- “make install” is where all the work pays off. This takes the binary package you created and installs it on your system.
Cleaning up
To clean each step along the way and move back in the process (sometimes a necessary thing to do if upstream updates its version number, etc.) there are a variety of targets that are pretty self-explanatory:
- “make uninstall” – uninstall the binary package
- “make clean=package” – removes the binary package from the /usr/ports/packages/${MACHINE_ARCH} directory
- “make clean=fake” – removes the fake directory from the /usr/ports/pobj/$WRKDIR/fake-${MACHINE_ARCH} directory
- “make clean=all” – undoes any work done by “make build”
- “make clean=distfiles” removes the source archive downloaded from the $MASTER_SITES directory
Did it work?
Well, unfortunately there were two unit test failures in the test. Since the OpenBSD project is not a fan of just hacking stuff through, I started working with the author of this module (known as “upstream” in the vernacular of the people) to see what could be done to fix the problems. After a couple of back and forth sessions, he kindly found and corrected a pretty subtle bug that hadn’t been seen before. I asked him to cut a new version and put it up on the repository and updated my Makefile to use the new version.
Now what?
At this point, I created a compressed tarball of the entire port directory (after testing that the new versions worked without error of course) and sent an email to the ports@ mailing list (ports@openbsd.org) with the tarball attached. The syntax you should use in these emails to simplify things is as follows:
- The subject line should start out with a tag in square brackets indicating if this is a new port or an update (in our case “[NEW]”
- The tag should be followed by the name of the port subdirectory with its category (in our case net/py-dnslib)
- The body of the message should include a description of the port (I just include the text from dist/DESCR) and a request for review and committing if everything looks good (I am not a commiter).
In the process of going back and forth on the mailing list, I got and incorporated more feedback to simplify things in my Makefile to finally generate the following file that was committed on my behalf:
# $OpenBSD: Makefile,v 1.1.1.1 2015/10/30 16:44:54 rpointel Exp $
COMMENT= library to en/decode DNS wire-format packets for Python
MODPY_EGG_VERSION= 0.9.6
DISTNAME= dnslib-${MODPY_EGG_VERSION}
PKGNAME= py-${DISTNAME}
CATEGORIES= net
MODPY_PI= Yes
HOMEPAGE= https://pypi.python.org/pypi/dnslib
MAINTAINER= Bryan Everly <bryan@bceassociates.com>
# BSD
PERMIT_PACKAGE_CDROM= Yes
MODULES= lang/python
FLAVORS = python3
FLAVOR ?=
do-test:
${SUBST_CMD} ${WRKSRC}/run_tests.sh
cd ${WRKSRC} && ./run_tests.sh
.include <bsd.port.mk>
Wrap up
Well, that was fun wasn’t it? While it seems like a lot of work, it is important to point out that this level of attention to detail and quality is what makes OpenBSD the stellar operating system that it is today. I hope that you were able to learn a bit more about binary packages and how the ports system creates them. As always, any questions or feedback would be welcome.
Pingback: 2 – Where do binary packages come from? - Exploding Ads