Wait… usenet is still… alive?!?!

In my previous post, I described getting re-acquainted with Fortran after 38 years and discovering that the language had really added some nice features over that time.  I also mentioned how my “curly brace bigotry” was starting to thaw out.  Well, now let’s talk about what I’ve learned over the past week.

First things first, like a good OpenBSD user, I found the mailing list for the GNU Fortran compiler and sent a question to it.  Wrong!  That list is for people working on the compiler itself, not for dumb n00b questions like mine.  Oh whelp.  They were pretty friendly about pointing me to the last place on earth I’d ever expect to be going – usenet.

For those of you who aren’t greybeards like me and have no idea what usenet is, I’ll enlighten you.  Back in the dark ages of the internet, when we used dial-up access, bandwidth was very dear to everyone – even the servers on the Internet itself.  Usenet was essentially a de-centralized message board that you could download information from in phases.  First, you pulled down the list of boards themselves, then when you selected your board, it pulled down the most recent topics, then when you clicked on a topic, you would get the actual text of the topic and any replies to it.

The topics were arranged hierarchically like comp.os.bsd and such.  There was a creepy part under the “alt” heading that had such jewels as “alt.wil.wheaton.die.die.die” for people who didn’t like the character he played on Star Trek TNG.  At one point, I could swear remembering that usenet went away and was really nothing more than an archive that some sites saved.  Apparently I was wrong because my friends on the compiler mailing list directed me to comp.lang.fortran.  On a side note, one of the folks on that list was very kind and helped kick start some of my early learning.

After a brief internet search, I found a link and clicked on it.  I don’t know if I should be seeking professional help or if others have dreams like this, but occasionally I have a “neglect dream” where (in the dream) I suddenly remember something I was supposed to have been doing but wasn’t – like feeding my non-existent giraffe.  Well, the feeling I have in the dream when I head out to where the giraffe lives (expecting at the least a thin and angry horned long-neck horse) was the feeling I had clicking on that link.  However, instead of a dead or murderous giraffe, I found a healthy message board that used Google Groups.

In addition to posting questions when I got stuck and reading code on the Internet, I also downloaded and read (yes – cover to cover) a Fortran programming book called “Introduction to Programming with Fortran:  Third Edition” on my Kindle web app so I could read it in my browser on OpenBSD.  After perusing this information, I was even more impressed with what Fortran has grown up into.  One of the folks on the comp.lang.fortran group pointed out that the oldest date he was aware of was October 15, 1956 (an old IBM 704 manual apparently) so we are essentially looking at Fortran’s 60th birthday in a few months.  That’s a long time for something to stay this relevant.

OK.  Time to dive in.  My use case was to create a string class and then add a subclass to it that uppercased all strings before storing them.  Pretty simple, eh?  You’d think until you learn that, while Fortran has great numerical and scientific intrinsic functions and features, it really lacks character handling capabilities.  No problem, the Internet to the rescue.  I found a string upper-casing and lower-casing implementation that I baked into what Fortran calls a “module”.  Essentially a module is a way to encapsulate types, code (functions and subroutines) and data.  Modules can be in stand-alone files and can be included into another piece of code using the “use” statement.

module foo_m
end module foo_m

program bar
use foo_m
end program bar

As you can see from the above example, this is quite similar to the “require” functionality that you see in Javascript programs of the Node.js variety, the “#include” feature of C/C++ and the “using” feature in C#.

After some pointers from the kind folks on the comp.lang.fortran mailing list, I cobbled together a bit of a style guide that I tried to follow as I built my example.  By the way, if you would like to see the finished product, take a look on github and you should have a fully buildable version on OpenBSD.  If you want to build this on another platform, you’ll probably have to change from “egfortran” to whatever the gfortran compiler is called on your platform (most likely gfortran).

My string_utility.f90 file ended up with the following structure:

module string_utility
implicit none
public :: ucase
public :: lcase
pure function ucase(in_string) result(out_string)
character (len=*), intent(in) :: in_string
character(len(in_string) :: out_string
end function ucase

pure function lcase(in_string) result(out_string)
end function lcase

Some things to point out here.  The use of “implicit none” brought back some fond memories.  FORTRAN used to automatically type declared variables in a really cute way.  If the variable name started with a letter between ‘I’ and ‘N’, it assumed it was an INteger.  Ha!  Since we want to declaratively type things these days and not have “odd” side-effects from features like that, implicit none turns this feature off for the module.

Fans of other OO languages like C++, Java, etc. will recognize the use of “public” and “private”.  Fortran is public by default, so putting the private declaration at the top of the module flips this to the way I feel most comfortable.

The “pure” keyword isn’t really needed perhaps, but I threw it in to illustrate that feature of the language.  Fortran is very focused on parallel programming and this feature is essentially a compiler hint that the code in that procedure is deterministic and can be ran in parallel if necessary.  For a better description than a n00b like I can produce, take a look at this stack overflow post.

The last interesting bit in this code snippet is how I declared the return type from my function.  You can do it two ways, the traditional supply-the-type-before-the-name way (in which case you specify the return value by assigning your result to the name of the function in its body) or the way I did it here where I have a specific variable I declare to hold the result.  No preference, this just looks tidy to me.

By the way, it is probably worth pointing out that, unlike the C languages and their offspring, Fortran has two types of procedures – functions and subroutines.  A function returns a value and requires no special calling syntax and a subroutine returns no value but requires the use of the call subroutine_name() syntax to invoke.

The next thing we’ll talk about is how you build the equivalent of a class from the curly-brace languages.  That is done in Fortran by creating a type in a module.  Take a look at the following code skeleton:

module string_m
implicit none
type, public :: string_t
character(:), allocatable :: str_m
procedure, public, pass(this) :: get_value => get_value_string_t
end type string_t
end module string_m

Ok.  There are many interesting things going on in this code.  In the type declaration, you see me creating a type called “string_t”.  The “_t” bit is just convention, it isn’t syntactically necessary.  The private variable “str_m” is an “allocatable” character array, or what I would call a “string”.  You also see a public procedure called “get_value” that is aliased to a local function I called “get_value_string_t”.

Finally the “pass(this)” part is some cool magic.  This is where you declare the equivalent of a “this” pointer in C++ that contains a reference to the object in who’s context your procedure is being invoked.  The slick thing is you can call it whatever you want so you aren’t tied to a particular name.  Plus (to me at least) the fact that you have to explicitly declare it takes away some of the “magic stuff” in C++.

To subclass this little gem, it is pretty straightforward.  Take a look at the following code skeleton to see how that happens in another module:

module ustring_m
use string_m, only : string_t
use string_utility, only : ucase
implicit none
type, public, extends(string_t) :: ustring_t
procedure, public, pass(this) :: set_value => set_uvalue
end type ustring_t
end module ustring_m

OK.  Lots of stuff going on here.  For one, did you notice the new “use” feature I pulled out of the air?  If you say “, only :: ” and list a set of components, only those components are used from the included module.  This helps you avoid unintended name space collisions from crud that happens to live in a module.

You can see from the “extends” piece of syntax, that this is how you subclass your base class.  Finally, if you look at the procedure, you see that I am overriding the “set_value” procedure from the base class and mapping it to the local function that I named “set_uvalue”.

At this point, I knew enough to be dangerous and, as is my speciality, decided to soar over the tips of my skis and crash into the mountain – I had read about operator overloading and decided to create an overload of the equals operator!


Well, that was painful.  Back to the comp.lang.fortran group for some healing and education.  I learned that, while you can overload operators in Fortran, what I wanted to do was overload “assignment”.  After a lot of back and forth with some incredibly patient people out there, I discovered the solution that makes all of this make sense and be pretty simple to do.

You declare the assignment operator in your base class and use the ability to alias the names of procedures in your subclass to overload the actual methods performed.  Here is how it looks in the base class:

type, public :: string_t
character(:), allocatable :: str_m
generic, public :: assignment (=) => string_t_assign_string_t, &
! Procedure declaration for internal methods
procedure, private, pass(lhs) :: string_t_assign_string_t, &
end type string_t

Believe me, this looks way cleaner than the messes I created along the way to learning how to do it.  Oh, and did I mention that Fortran uses the exclamation mark (bang) for comments?  It feels slightly ironic to me like, “Holy crap!  I’m actually putting a comment in my code!  Can you believe it?”  I used to work with a guy who said that anyone not smart enough to understand his code by reading it shouldn’t be in there in the first place – that’s why he never commented it.  😉

So the interesting bits here are the use of the “generic” keyword.  This allows us to genericise what can be on the right-hand side of that assignment operator.  Very similar to generics in other languages.  Finally, notice that we have to declare the actual procedures that do the assignment (one from a string_t type and a second one from a character or character array) twice.  Once in the mapping and a second time for the actual declaration.

The implementation of the procedures is pretty straightforward:

elemental subroutine string_t_assign_character(lhs, rhs)
class (string_t), intent (inout) :: lhs
character(len=*), intent (in) :: rhs
lhs%str_m = rhs
end subroutine string_t_assign_character

The only piece of magic here is the use of the keyword “class” so that you can tell the compiler you might want to allow someone to override this procedure later in a subclass.  If you don’t want that, just use “type”.  Finally, I picked this method to show the implementation of because of the use of “elemental”.  This tells the compiler that you can pass a single character or an array of characters.

One thing that a good object needs is a constructor in C++ to initialize internal data, etc.  Figuring this out in Fortran was a little challenging.  What I ended up with is as follows:

module string_m
implicit none
type, public :: string_t
! Internal private character store
character(:), allocatable :: str_m
end type str_t
! Class constructor
interface string_t
module procedure string_t_constructor
end interface string_t
type (string_t) function string_t_constructor()
string_t_constructor%str_m = ""
end function string_t_constructor

So the trick here is to declare the interface, then actually define the function in the contains section of the module, returning a type of “string_t”.  I’m not certain if the “_constructor” is convention or required.

Now I can build a little test program to exercise my classes like this:

program fortranString
! Pulls in public interface for our String module
use string_m, only : string_t
use ustring_m, only : ustring_t
! Prevent default I-N integer assumption
implicit none
! Declare local variables
type (string_t) :: string1
type (string_t) :: string2
type (ustring_t) :: string3
string1 = "Bonjour"
print *, string1%get_value()
end program fortranString

As you can see, we can call the methods in the classes, use the assignment operator and all sorts of fun things like that!

So, in conclusion, Fortran is a pretty cool language.  The syntax is a little different that a curly-brace guy like me is used to, but once you figure it out, it’s pretty easy to use and has a very nice feature set. Again, if you’d like to look at a functional complete example, check out my source repository on GitHub.

I’m going to do a third post in this series where I actually build a modern web application using Fortran for the middle tier (I’m thinking I need a cool name like LAMP or BCHS so maybe FARM – Fortran, Apache, REST and mySQL?) but that’s for another day.  Hope you enjoyed reading this as much as I enjoyed learning it.

Have you seen FORTRAN lately?

I’m familiar with the FORTRAN programming language.  Heck, the first formal programming class I ever took was in 1978 in FOTRAN-IV on punched cards (don’t laugh kids, I’ll tell you to get off of my lawn if you do).

My next exposure to this venerable language was in college when I took my intro to CS course in the amazing, whizz bang “new” FORTRAN-77.  In that class, I got done with my final so quickly that I spent the rest of the time using VT-100 escape codes on the VAX-11/784 to animate a beer truck across the terminal – again written in FORTRAN-77.

My final trip to FORTRAN-ville was a year later in an algorithm analysis class where we had to solve the “Towers of Hanoi” problem using recursion in both FORTRAN-77 and Pascal.  The only catch – FORTRAN-77 didn’t support recursion so we had to build and manage our own stack.

All of this left me with the feeling that FORTRAN wasn’t a very exciting language and was quite old-fashioned.  Not too long after that, I learned C and my path to the dark side began.  I went from C to C++ and eventually to things like Java, Javascript and C# – quite happy in my little curly-brace world.  And by the way, the only true way to nirvana lies in putting the curly brace on a separate line from the IF statement now doesn’t it my disciples?!?!?!  Bwahahaha…

OK.  Fast forward to a couple of weeks ago.  I had been reading about the BCHS (pronounced “beaches”) web programming stack (see this page for more details) which essentially is using C for your middle-tier code with CGI to write web applications.  Now, regardless of whether or not this was serious or a joke (there seemed to be some debate on this on Reddit at the time), it got me thinking.

I remember CGI programming from the mid 90’s.  Essentially the idea is that you have a chunk of code running on the web server (be it a script or compiled code, it really doesn’t matter) that kicks out HTML for its output.  This allowed you (way, way, way back in the day) to access crazy things like relational databases and such to generate dynamic web pages and perform I/O with users.

I went down the path that said, “Hey.  If you can do this with C, why not other languages?”  I tried thinking what the most absurd, crazy, old-fashioned thing you could use and came with COBOL!  Of course!  That would be the funniest thing imaginable.  Unfortunately, even doing a multi-year stint as a mainframe CICS programmer back in the 80’s and 90’s, I never used COBOL (I used PL/I for those who were curious – a pretty cool language for its time actually).  This left me with FORTRAN – the language I knew from college and my brief punch-carded middle-school experience.

I did some sniffing around in the ports tree and discovered that OpenBSD has the GNU version of FORTRAN and it appears to be fairly recent.  I did some further investigation and hit a wall – we only support FastCGI with our httpd web server and surely that wasn’t the same thing as straight up CGI that I remembered.  After some searching, I confirmed my suspicion – FastCGI counts on a long-running process that the web server communicates with via sockets so that it doesn’t have the overhead of firing up and tearing down a process for each web transaction (boy they sure got smarter after the mid 90’s on this web stuff <grin>).

Not to be deterred, I decided to use the Apache web server from ports.  Installing that was pretty simple via pkg_add and turning on CGI access wasn’t that tough either.  All I had to do was uncomment the LoadModule call to the cgi_module, add a ScriptAlias to a virtual /cgi-bin directory (to contain my scripts outside of the htdocs tree) and add the ExecCGI option to my htdocs directory tree.

From there, I wrote a little shell script to test things and stuck it in my /var/www/cgi-bin directory as testcgi.cgi:

echo "Content-type text/plain"
echo ""
echo "Hello world!"

I then invoked and got the content “Hello world!” in my web browser.  Note the blank line after the content-type header – if you miss that, you will suffer with 500 errors until your web searching fu teaches you the error of your ways.

Now for the fun, let’s use FORTRAN to write our CGI script.  This should be crazy.  I can’t wait for all of the uppercase characters and sequence/line numbers in my xterm under vim.  I’ll fee like Indiana Jones in the Temple of Doom or something – a crazy technology archaeological expedition!  Off to wikipedia and the web to re-learn this language enough to write out strings.

Well, the first thing I learned was we are much more polite in this millennium than we were in the last – we don’t shout the name of the language any more.  FORTRAN (which stood for FORmula TRANslation) was now Fortran.  Also, there were new standards that were established after the 1977 version I was used to.  They had one in 1990, 1995, 2003 and 2008.  Apparently there was even a 2015 version undergoing standards ratification.  Huh.  Looks like people still use this crazy old thing.

Some more research turned up an interesting fact – 15% of the world’s software is written in Fortran.  I started looking at some of the language features that had been added over the years (more out of curiosity than for any other reason) and was shocked, nay horrified to find out that they added recursion.  That would have made my data structures class in college a heck of a lot easier!

I kept looking and discovered other interesting things.  For one, user defined types!  We didn’t have those in FORTRAN-77.  Also fun things like object oriented features, operator overloading, free-form formatting, case-insensitive intrinsic functions, generics…

Holy crap!  I suddenly felt like the prototypical old guy at the high school reunion who discovered that the nerd you remembered had grown up to be attractive and successful.  My world-view (the one that said all curly-brace languages were naturally superior to all others) was in jeopardy.

The only thing that could save me would be if our Fortran compiler on OpenBSD was too archaic for any of this funny-business to work.  Then I would be justified again.  My heart rate started to settle down to a more normal level at this thought.

I started sniffing around in the ports tree and discovered I already had most of what I needed installed.  With the latest (in our tree at least) gcc installed, all I had to do was add the g95 compiler via pkg_add.  I did so, and then tried invoking it with a simple ‘g95 –version’.

No joy.  No binary with that name on my system.  Huh?  I did a ‘find / -name g95’ and still came up dry.  I then did some web searching (notice I don’t say “googling” any more now that I’ve switched to duck duck go for my web searching) and discovered that, silly me, it isn’t called “g95” it is called “gfortran”.  OK, I tried that ‘gfortran –version’.

No joy.  No binary with THAT name on my system.  WTF?  After some more searching, I discovered that we call it “egfortran” on OpenBSD and all was right with the world.

I tried my little test program that I did before with a shell script, but this time did it in Fortran as testcgi.f and came up with:

program testcgi
print *, "Content-Type: text/plain"
print *
print *, "Hello World!"
end program testcgi

When I used the command-line ‘egfortran testcgi.f’ I was expecting to run a.out and move on with my life.  Nope!  All sorts of crazy error messages.

After some more searching, I found the ‘-ffree-form’ compiler flag.  It worked.  (Edit:  I later have learned that if you use a .f90 file suffix, the switch is not necessary.)  I copied the resulting a.out binary into my /var/www/cgi-bin directory and renamed it to testcgi.cgi and went back to my browser.  Drum roll please….

Fail!  I got a 500 error.  This really stymied me for quite some time until I picked up on a subtle thing.  When I ran my binary from the command-line, I suddenly noticed that every line of text was preceded with a blank space in column 0 (or would that be column 1 in a Fortran world?).  Anyhow, after some web searching I discovered that I could use a format specifier and all would be fine.  This left me with the following, which worked:

program testcgi
print '(a)', "Content-Type: text/plain"
print '(a)'
print '(a)', "Hello World!"
end program testcgi

So now that I have scratched that initial itch, I’m curious to see how far Fortran can take me with a more interesting problem to solve.  Look for an upcoming post that will delve into that.

Smaller is better

My home office (where my network and servers live) is a warm, noisy place.  So much so that I really wasn’t enjoying being in the room.  Since I don’t have an air conditioned datacenter with a raised floor in my house, I decided that I wanted to do something about this problem.  Interestingly enough, the biggest source of heat in the room turned out to be my two 24″ Apple Cinema Displays from circa 2006-2008.

One of the others was the little PC I built to be my OpenBSD router / firewall.  I did some research and discovered the APU2 board.  This little beauty is air cooled, about the size of a CD jewel case, and has 4 cores and up to 4GB of RAM with 3 gigabit Intel ethernet ports on it.  Sounds like a winner to me.

I acquired one of these from Mini-Box (http://www.mini-box.com/ALIX-APU-2C4-AMD-G-Series-GX-421TC?sc=8&category=754) along with the case and power supply.  I picked up a cheap mSATA 64GB drive for storage and spent about 3 minutes with a screw driver assembling the thing.  The end result was a small case about the size of three audio CD jewel cases stacked on top of each other.  After that, I popped a USB stick with OpenBSD 5.9-current on it and booted.

The system (which I was connected to with a serial cable to the serial port on the APU2 board) booted up, I saw the boot prompt for OpenBSD and then it booted up, I saw the boot prompt for OpenBSD, and then it booted up…  You see where this was headed.

I reached out for help on the mailing lists and very quickly had two folks clue me in.  When OpenBSD’s kernel starts to load, it looks for a console and if it can’t find one, it exits, creating this boot loop.  At the boot prompt, a few commands had be booting to the installer:

boot:  stty com0 115200
boot: set tty com0
boot: boot /bsd

I decided to do this router “right”.  Since it was just going to be a router and firewall and I wouldn’t run anything else on it, I wanted to go with a small attack surface so I chose to not install any of the X packages.  This turned out to be a “Bad Idea”(TM) since you can’t build any ports on the system if you don’t have X installed.  Since I use tarsnap and it can only be built from source currently due to the licensing model, I went back, installed the X packages from the installer and was good to go.  Make sure you disallow root logins over ssh (the default) as this is going to be internet facing in the end.

Once I logged in as root the first time, I needed to give my unprivelaged user some juice so I created an /etc/doas.conf file, allowing anyone in the “wheel” group to run commands as root (preserving their environment).

# echo "permit keepenv :wheel as root" >/etc/doas.conf

I then added my unprivelaged user to wheel, wsrc and staff and then logged out.  At this point, I shouldn’t need to log in as root at all.

# usermod -G wsrc <your user> (and so on)

Since I have my dotfiles in a github repository, I needed to now log in as my unprivelaged user and generate their SSH keys:

$ ssh-keygen -t rsa -b 4096 "<your email>"

I then copied the ~/.ssh/id_rsa.pub file’s contents and pasted it into a new SSH key in my GitHub account’s settings.  Now I’m good to clone my dotfiles repository and have it set up my environment the way I like it to be set up.  However, first I need to install git.

I temporarily export the PKG_PATH that I like to use:

$ export PKG_PATH=http://openbsd.cs.toronto.edu/pub/OpenBSD/5.9/packages/amd64

Then I install the git package:

$ doas pkg_add git

… and it didn’t work.  Ah!  I forgot to actually connect to the network after rebooting.  A simple:

$ doas ifconfig em0 up
$ doas dhclient em0

and I was good to go.  I added git and then cloned my repository:

$ git clone <your user>@github.com:/<your user>/<your dotfile repo>.git

Then I ran the shell script to set up my dotfiles, logged out and back in to pick up the changes and I was good to go.  Now I needed to actually configure this box as a router and firewall the way I like it.  To do this, I stand on the shoulders of giants and use the awesome OpenBSD FAQ.

The APU2 numbers its ethernet ports from left to right as you look at the back of the case (meaning that em0 is the port closest to the DB9 serial connector).  I’m using port 0 for my WAN interface and port 1 for my internal.  My goal is to use the third port as a private interface to a second setup just like this and use CARP to make it redundant.

So, for my router, I start by adding the following to /etc/sysctl.conf:


Then, we need to enable the dhcp daemon:

$ doas rcctl enable dhcpd
$ doas rcctl set dhcpd flags em0

Now I create this as my /etc/dhcpd.conf file:

option domain-name-servers;
subnet netmask {
option routers;

Following the tutorial, we will be also using unbound for local DNS caching.  Enable it as follows:

$ doas rcctl enable unbound

Create a /var/unbound/etc/unbound.conf file like they did in the tutorial:

do-ip6: no
access-control: allow
do-not-query-localhost: no
hide-identity: yes
hide-version: yes
name: "."

Add dnscrypt-proxy from the packages tree and enable / configure it:

$ doas pkg_add dnscrypt-proxy
$ doas rcctl enable dnscrypt_proxy
$ doas rcctl set dnscrypt_proxy flags "-l /dev/null -R dnscrypt.eu-dk -a"

Finally, prevent your upstream ISP from changing your DNS resolution via DHCP:

# echo 'ignore domain-name-servers;' >> /etc/dhclient.conf

Now we need to set up the firewall.  I used this for my /etc/pf.conf file:

int_if="{ em0 em2 }"
set block-policy drop
set loginterface egress
set skip on lo0
match in all scrub (no-df random-id max-mss 1440)
match out on egress inet from !(egress:network) to any nat-to (egress:0)
antispoof quick for (egress)
block in quick inet6 all
block return out quick inet6 all
block return out quick log on egress proto { tcp udp } from any to any port 53
block return out quick log on egress from any to { no-route $broken }
block in all
pass out quick inet keep state
pass in on $int_if inet
pass in on $int_if inet proto { tcp udp } from any to ! port 53 rdr-
pass in on egress inet proto tcp to (egress) port 222 rdr-to
pass in on egress inet proto tcp from any to (egress) port 2222

Finally, I added “noatime,softdep” to my /etc/fstab for my non-swapfile mount points because I’m running an SSD drive.  I also disabled the sound server to further reduce attack surface on this box:

$ doas rcctl disable dnsiod

Next, I set a static IP address for ethernet port 0 (em0) so that dhcpd could bind to it:

# echo 'inet' > /etc/hostname.em0

Finally, I told ethernet port 1 (em1) to get its IP address via dhcp from my upstream ISP’s cablemodem:

# echo 'dhcp' > /etc/hostname.em1

At this point, I did a reboot, plugged in my upstream router to ethernet port 1, plugged my home network into ethernet port 0 and tested to ensure everything was working the way I expected it to.

The one thing I did have to do was reboot each machine on the home network to pick up the new DHCP stuff.  It all worked like a champ.

Oh my God! I killed Kenny… er the VAX

Have you ever had one of those days where nothing seems to go right?  Like the fictional character from the old “Pebbles & Bam Bam” cartoon “Bad Luck Schleprock” where a cloud follows you around and everything you touch turns to… well crap?

I think I had a day like that yesterday.

It all started when I read a posting on Undeadly.org announcing that OpenBSD was dropping support for the VAX platform:


As I read the posting, especially after seeing the part where it said “after much internal discussion”, I groaned.  Literally groaned out loud.

You see, if we roll the clock back to January, I had picked up a dirt cheap VAXstation 3100 on eBay and was working on getting OpenBSD-current up and running on it.  When I looked at my favorite mirror, however, I couldn’t find packages in the “snapshot” subdirectory for the VAX architecture.

Being the n00b that I am, I posted to the misc@ mailing list and asked the question that I fear started this whole snowball rolling:  “I can’t find -current packages for the VAX, are we dropping support in 5.9 for it?”


Really quickly, someone answered back that we weren’t building packages on that platform because it took so long and that we typically build them near the end of the release cycle.  No big deal, I went on with my day.

Well, if you follow the thread (and I clearly wasn’t doing so), it goes on until we see Theo weigh in sometime later and… I fear I was the guy who accidentally kill the VAX support in OpenBSD.  I’m like the people who keep killing Kenny on South Park.


Sometimes, processors need love too

I have a confession to make.  As a person who has had a passion for computers and programming since the late 70’s, I have spent a lot of time on some really fun, albeit old, platforms.  For example, I started out on a Cyber 171 programming in BASIC and Pascal, then “graduated” to a TRS-80 model 1 doing BASIC as well.

When I was able to buy my first computer, I went with the Apple ][+ (and yes, that’s the right way to spell it!) and eventually upgraded to an Apple //e.  Once I realized the 32-bit revolution was on, I transferred my allegiance to the Commodore Amiga 1000, then the A2000 with its ability to install expansion cards.  Into one of those lovely slots so reminiscent of my old Apple ][+ went a Bridgeboard.  First the 80286 version (on which I ran DOS) and then finally the 80386sx version that I ran Windows 3.x and OS/2 on.

Along the way, I worked on a VAX 11/784 (and was a system administrator) where I learned such useful things as VT100 escape codes that would allow me to write programs that did simple animations on the screen.  I lovingly recall finishing my FORTRAN-77 final early in college and spending the extra time writing an animated beer truck that chugged across the screen and had a little guy get out and dance at the end.  (Of course I turned in both programs to the professor!)

I cut my teeth in the UNIX world working on SunOS 4 on a variety of Sparcstations, as well as logging in remotely to a MicroVAX running Ultrix (as I recall).  That was where I first discovered that these machines were connected over a global network and I could bounce from machine to machine.  Welcome to the Internet!

I finally sold out and went down the dark path of C/C++/VB/C# software development on the Windows platform (starting with Windows 3.x and then moving to Windows NT 3.1 beta).  After many years on that platform, my wife bought me an iPod for Christmas.  As I was thinking to myself “she doesn’t know this won’t work on Windows” she said, “This is the new version that works on Windows.”  (Not the first time she has proven herself to surpass me, nor will it be the last!)

That iPod was the gateway drug that got me to try out a 32-bit PPC Mac Mini.  It was simple to use and pretty useful.  I could even run Windows on it in an emulated (aka slow) mode.  I then discovered that it had UNIX under the hood and my spinal cord started typing things that surprised me at the terminal prompt from my old SunOS 4 days.  When I heard Steve Jobs announce the move to Intel, I told myself that this was the end of the line for the Mac.  Just to be a “kind soul”, I enrolled in the Intel Mac beta program and received a G5 tower that had nothing in it but a tiny little PC motherboard and a running copy of OSX 10.4 on that architecture.

Given that time has proven me wrong there as well (never take a stock tip from me BTW), I was knocking around on various Intel-based Mac desktops and laptops ever since.  Then, I discovered OpenBSD and fell in love.  One of the things I discovered was that it ran on a variety of platforms.  Not like our friends over at NetBSD do (“my toaster runs NetBSD!” <grin>), but they were tier-1 platforms nonetheless and many of them were my old friends (sparc64, ppc32, VAXen, etc.)

Since I had that 32-bit PPC Mac Mini in the closet and an old i386 laptop, I tried installing OpenBSD on these two “exotic” architectures and found it to be drop-dead simple.  I actually had to set the hostnames and the prompt in ksh to remind me I was on a different architecture.

When browsing eBay and finding a SunBlade 100 there for under $30, I couldn’t resist and pulled the trigger.  After a brief amount of fiddling, I had a sparc64 processor in my stable now too!  This was when I noticed something was amiss.  Many of these other platforms had much fewer ports/packages available for them, which makes them marginally less useful.

Given this observation, and an email response I received on the HPPA mailing list from none other than Theo DeRaadt himself where he told me that some of these platforms just need “some love”, I decided this might be a worthy quest for someone who now has a few ports under his belt.  But where to start…

I decided that polishing up my ksh scripting skills might make this more fun so I first set my sites on getting a sqlite3 database with all of the ports tagged for each platform.  I used curl to pull down the list of packages from one of the mirrors, jammed it through a sed script to extract just the package name and used some more sed magic to turn it into an insert statement for sqlite3.  Now I have a database of all of the ports for each platform.  What to do next?

I decided to do some sniffing around to find out how I could determine the dependencies for each port because I discovered that many of these ports are missing on the platforms because their dependencies aren’t there.  Knowing the interrelationships of the ports themselves would make my job easier / more fruitful.  How could I determine this though?

After exercising my Google-fu, I found out that you can use the pkg_info with a ‘-f’ flag to list the dependencies for a given port.  Some chaining of that with grep and cut and voila – I had a little script that, given a port name, could spit out the dependencies for me.

I then wrote a third script to iterate over all of the distinct port names from my sqlite3 database and iteratively call my dependency finding script, storing the results in a second table in my sqlite3 database.  Now I have all of the ports by platform, and a list of the (at least top-level) dependencies for each port.

Given that, it was a simple matter to create an ugly query to give me a text file for each processor of which ports were missing from that architecture, along with the number of dependencies each port had, ordered to make the simple ones (zero dependencies) sort to the top of the list.

Where does that leave me?  Well, my marching orders are pretty clear.  I need to start plowing through some of these ports and see what the collateral impact of this work is in terms of getting more parity in the ports trees between the various architectures.

If you have read this and are interested in the scripts I created, I’ll point you towards my github project for this.  The only caveat is you can’t mock my sketchy SQL statements or even sketchier scripts:


That’s all for now.  Wish me luck as I start giving some love to these old friends.

OpenBSD at HiDPI!

Some of the recent work done by the core OpenBSD developers is really incredible.  Specifically, the work that has been done for EFI booting of more “modern” hardware has started to open up a whole new vista of machines that we can run our favourite operating system on.

I normally rock the Lenovo Thinkpad x220 and find it to be an outstanding machine for both general purpose computing and software development.  However, I have a Macbook Pro 11,1 (Haswell Retina laptop manufactured in late 2013) that I have tried (and failed) in the past to put OpenBSD on.  The challenge came from the fact that this particular laptop has USB 3 but not USB 2.  The symptom of this is that you are able to use the builtin keyboard when you hit the initial “boot” prompt, but after loading the installer, the keyboard is non-responsive.  Using an external USB keyboard didn’t help because the problem is in how the operating system talks to the USB 3 controller.

I started seeing traffic on the misc@ mailing list about people having success with Macbook Air models and the new 12″ Macbook (all very recent vintage machines), so I thought I’d give it a try again.  In the end I was able to get everything working really well with a Gnome desktop that did the right thing for the HiDPI retina screen and everything (well, almost everything, I did have to use a USB dongle for WiFi because the builtin one wasn’t recognized).

So given that I had a functional, nearly modern laptop what else should I do but reformat the hard drive and start over!  I figured that it would be a nice thing to do for the community to start this bad boy from scratch and document all of the steps along the way.  Therefore I turned it off, held down on Command+R and powered it up.  For the non cultists reading this, that puts the laptop in “Internet Recovery” mode where you can attach to a WiFi network and it will give you all of the tools to repartition the drive and install the operating system over the Internet.  Slooooowwwwwwlllllyyyyyy…..

Once I had the operating system reinstalled and verified that a reboot worked, I figured I was at a clean starting point. Since I want to make my life hard, I thought it would be fun to preserve Mac OS X on this machine and dual boot.  To make that happen, I need to shrink the partition and create a new one in the free space created. Unfortunately I could not get that to work to save my life – it always would try to boot the Boot Camp partition as bios and not as EFI which, on my laptop would hang trying to configure a mystery second processor (cpu1) in the system.

Given that, I decided to make my life simple and go back to a single-boot system (which I really wanted anyhow).  I leveraged the howto was written by Jasper Lievisse Adriaanse to get the EFI bits on the disk.  I’ll detail it step by step here.

First, I created a thumb drive with the latest install58.fs snapshot on it from my favorite Canadian mirror.  I then rebooted the laptop with the “Option” key held down and picked the “EFI Boot” partition off of the thumb drive.  You’ll be astonished how small the text is so break out those reading glasses if you are over 40 like me!

Once I hit the first prompt from the installer, I jumped to the shell and ran this:

# fdisk -i -b 960 sd0
Do you wish to write new MBR and partition table? [n] y

Now I like to run an encrypted boot drive, so from my favorite tutorial from the bsdnow.tv site, I did the following:

# disklabel -E sd0
> a b
offset: [1024]
size: [xxxxx] 32g
FS type: [swap]
> a a
offset: [xxxxx]
size: [xxxxx] *
FS type: [4.2BSD] RAID
> w
> q
# bioctl -c C -l /dev/sd0a softraid0

Make sure you pick a good passphrase that you can remember but would be hard to guess or brute force.  Make a note of what your CRYPTO volume is attached as.  In my case that is sd2 – this is the “drive” I will be installing to.

Now, run the “install” script and install as you normally would.  Make sure you install to the crypto drive that you just created.  I normally put 100g in place for just /usr (I don’t break it down) so I can install source and ports and have room to build and then use the remainder of the drive for /home.

Once the install has finished DO NOT REBOOT YET because there is some additional work to do from the EFI tutorial I referenced above.

# /mnt/sbin/newfs_msdos sd0i
# mount /dev/sd0i /mnt2
# mkdir -p /mnt2/efi/boot
# cp /mnt/usr/mdec/BOOTX64.EFI /mnt2/efi/boot

Now you should be free and clear to reboot the laptop.  If all goes well, you should be presented with the passphrase prompt to decrypt your boot partition. Be patient, it took nearly 30 seconds for it to come up on my system.  Reminds me of when you don’t have a default boot volume selected on OS X.  Oh well, a mystery for another day.

After the reboot and logging in as my root user, I modified /etc/fstab to add “,softdep,noatime” to all of my read/write partitions and also a line for /dev/sd0b as my swap partition.

Now that I have this done, since there isn’t a functional WiFi device (yet) on this platform, I plugged in my handy-dandy little ASUS USB-N10 USB adapter, installed the firmware from http://firmware.openbsd.org (it’s the rsu one) into the /etc/firmware directory using a thu, rebooted and configured my network:

# ifconfig rsu0 nwid MYNETWORK wpakey MYKEY
# ifconfig rsu0 up
# dhclient rsu0

Now I need to add my PKG_PATH to be able to install binary packages.  I added the following to my .profile file:

export PKG_PATH=http://openbsd.cs.toronto.edu/pub/OpenBSD/snapshots/packages/`machine -a`/

After logging out and back in, I then installed my desktop of choice, Gnome as follows based on this undeadly.org post:

# pkg_add gnome
# echo 'multicast_host=YES' >> /etc/rc.conf.local
# echo 'pkg_scripts="messagebus avahi_daemon gdm" >> /etc/rc.conf.local

A quick reboot then shows me I can log into Gnome.  What’s astonishing is that the HiDPI stuff in Gnome seems to be working perfectly out of the box!

Where do binary packages come from?

To follow up on my last post where I talked about the high-level concepts involved with OpenBSD ports & packages and why you would want to work on them (Time to make the donuts), I thought I would share an actual experience I had where there was a Python library I needed in OpenBSD that wasn’t there.

Create Directory

The library is dnslib and it was used by a security tool I was porting to OpenBSD.  I first started out by creating a subdirectory under the /usr/ports/net called “py-dnslib”.  I choce “net” because this is a network library and I prefixed the name of the library with “py-” because that is the convention I observed browsing around in the ports tree.

Create Makefile

The next step in the process is to create a basic Makefile.  Remember that in OpenBSD, for licensing reasons, the ports tree does not contain the actual source code of the ports themselves.  It provides information to the ports build subsystem on where to download the source and process it.  The best way to build a new Makefile is not from scratch, but by looking for something similar that already exists and modifying it.  My initial Makefile looked like this:

# $OpenBSD$

COMMENT=                library to en/decode DNS wire-format packets for Python

DISTNAME=               dnslib-${MODPY_EGG_VERSION}
PKGNAME=                py-${DISTNAME}
CATEGORIES=             net

HOMEPAGE=               https://pypi.python.org/pypi/dnslib

MAINTAINER=             Bryan Everly <bryan@bceassociates.com>


MASTER_SITES=           https://pypi.python.org/packages/source/d/dnslib/

MODULES=                lang/python

FLAVORS = python3

${SUBST_CMD} ${WRKSRC}/run_tests.sh
cd ${WRKSRC} && ./run_tests.sh

.include <bsd.port.mk>

To walk through this, the first line is a comment that describes (as briefly as possible) what this port does.  The next section contains the author’s version number of this library, the name of the distribution file to download from the source server, the package name that will be used in the ports tree and finally the subdirectory (category) this port lives in.

Next, we see the homepage of this project on the Internet and my information as the maintainer of this port going forward.  Please note that if you sign up to be a maintainer of a port it is a serious commitment.  You will be responsible for making sure that you keep the OpenBSD version of this port current and to work with the upstream authors to ensure that it works cleanly and well on OpenBSD.  It is not a commitment to be taken lightly.

We then see a comment that indicates that this is a BSD license for the library (you can find that out by looking at the source code for the library or its homepage).  Next we have a line that controls whether or not the license for this port would allow it to be included in the release CD-ROM images for OpenBSD.

MasterSites points at where the source code is going to be downloaded from.  In this case it is pypi.python.org that we are fetching the code from.  The lang/python modules line pulls in support in the build infrastructure to simplify and standardize the way Python packages are built.  We also use Python 3 as a “flavor” for the package.  Check the OpenBSD documentation to get a better understanding of flavors.

Finally, the “do-test” target executes the built-in unit tests that are included with the dnslib module from the upstream author.  Now that we have the Makefile skeleton in place, time to actually build and test this port.

Other bits and pieces

Now that we have a Makefile created, we also need to create some additional files:

  • patches – This subdirectory will contain patches we have to make to the downloaded source to get it to build successfully on Open BSD.  See below for more details.
  • pkg/DESCR – This file is a 72 column wide text file that contains a broader description of what the port/package is and how to use it.

What do I do first?

The OpenBSD build subsystem has many options.  It is definitely worth looking at the bsd.port.mk manpage to better understand the complexities that are available to you.  In our case, there are some simple targets of “make” that we will use to get this port built and tested.  You can just be crazy and leap right to “make install” but I like to take it one step at a time and correct any problems as they crop up.

  1. “make fetch” is the first target you want to try.  This will execute the part of the build system that pulls down the source code from the MasterSites location.  The files are placed into the /usr/ports/distfiles subdirectory in a specific directory based on your $DISTNAME from your Makefile.
  2. “make extract” is the next target to try.  This will take the compressed source archive that was downloaded in step 1 and extract it into the /usr/ports/pobj subdirectory in a directory controled by $WRKDIR from your Makefile (derived from your $PKGNAME).
  3. “make makesum” – This target creates a checksum hash of your downloaded source package and stores it in a “distinfo” file.  That way if someone upstream monkeys with the source archive, we won’t just blindly pull it down and build it.
  4. “make update-patches” – If you have to patch a file (and in our case we do have to patch the test script to run the tests with our default version of Python), the way you do it is to first copy the file in the /usr/ports/distfiles/${DISTDIR} to a file with a .orig suffix (i.e. you copy foo.sh to foo.sh.orig).  Then you perform your edits on the non suffixed file and run “make update-patches” from your ports directory.  It will then create the diff you need in the patches subdirectory and it will be preserved across build and clean cycles.
  5. “make build” is the next target to try.  This will actually build your source code into its final form.  Since this is a Python module, not much exciting stuff happens here.  If it were a C program, then you’d get to watch the compile and link steps whiz past.
  6. “make update-plist” – This target creates or updates a file called pkg/PLIST that contains the manifest of files that will be part of the package.  Any time you add or remove a file (or change its name) you need to run this target and (if you are updating) remove the pkg/PLIST.orig file that is created.
  7. “make do-test” runs our unit tests.  More on this later…
  8. “make fake” is a fun one.  Sine the binary packages are built from the ports tree (actually, when you build a port from source it really creates and installs a binary package to be precise) there is a “fake” directory that mirrors the root filesystem of an OpenBSD install that all of your files from your port get packaged up into.  This lives in /usr/ports/pobj/$WRKDIR/fake-${MACHINE_ARCH} and if you browse it you will see the usual subject directories (/etc, /usr, /var).  When this target completes, the fake directory should contain all of the files that will be added by your binary package in the directories in which they should land.
  9. “make package” is next.  It takes the fake directory and creates the binary package from it.  The resulting file is placed in /usr/ports/packages/${MACHINE_ARCH} (mine is amd64).
  10. “make install” is where all the work pays off.  This takes the binary package you created and installs it on your system.

Cleaning up

To clean each step along the way and move back in the process (sometimes a necessary thing to do if upstream updates its version number, etc.) there are a variety of targets that are pretty self-explanatory:

  • “make uninstall” – uninstall the binary package
  • “make clean=package” – removes the binary package from the /usr/ports/packages/${MACHINE_ARCH} directory
  • “make clean=fake” – removes the fake directory from the /usr/ports/pobj/$WRKDIR/fake-${MACHINE_ARCH} directory
  • “make clean=all” – undoes any work done by “make build”
  • “make clean=distfiles” removes the source archive downloaded from the $MASTER_SITES directory

Did it work?

Well, unfortunately there were two unit test failures in the test.  Since the OpenBSD project is not a fan of just hacking stuff through, I started working with the author of this module (known as “upstream” in the vernacular of the people) to see what could be done to fix the problems.  After a couple of back and forth sessions, he kindly found and corrected a pretty subtle bug that hadn’t been seen before.  I asked him to cut a new version and put it up on the repository and updated my Makefile to use the new version.

Now what?

At this point, I created a compressed tarball of the entire port directory (after testing that the new versions worked without error of course) and sent an email to the ports@ mailing list (ports@openbsd.org) with the tarball attached.  The syntax you should use in these emails to simplify things is as follows:

  • The subject line should start out with a tag in square brackets indicating if this is a new port or an update (in our case “[NEW]”
  • The tag should be followed by the name of the port subdirectory with its category (in our case net/py-dnslib)
  • The body of the message should include a description of the port (I just include the text from dist/DESCR) and a request for review and committing if everything looks good (I am not a commiter).

In the process of going back and forth on the mailing list, I got and incorporated more feedback to simplify things in my Makefile to finally generate the following file that was committed on my behalf:

# $OpenBSD: Makefile,v 2015/10/30 16:44:54 rpointel Exp $

COMMENT=        library to en/decode DNS wire-format packets for Python

PKGNAME=        py-${DISTNAME}
CATEGORIES=        net
MODPY_PI=        Yes

HOMEPAGE=        https://pypi.python.org/pypi/dnslib

MAINTAINER=        Bryan Everly <bryan@bceassociates.com>


MODULES=        lang/python

FLAVORS = python3

${SUBST_CMD} ${WRKSRC}/run_tests.sh
cd ${WRKSRC} && ./run_tests.sh

.include <bsd.port.mk>

Wrap up

Well, that was fun wasn’t it?  While it seems like a lot of work, it is important to point out that this level of attention to detail and quality is what makes OpenBSD the stellar operating system that it is today.  I hope that you were able to learn a bit more about binary packages and how the ports system creates them.  As always, any questions or feedback would be welcome.