Congress just sold your online privacy, now what?

The United States Congress just voted on and passed a bill that allows Internet Service Providers to sell your data.  I thought I’d take a moment and first explain why this is “A Bad Thing(TM)”, then tell you what you can do about it.

Given that I’m going to promote this blog post more widely than I normally do, I think it would be a good idea to get everyone who is reading this level-set on some of the technology here.  Your Internet Service Provider, or ISP, is the company that provides you access to the internet.  For example, on my smartphone, my ISP is AT&T.  In my home, my ISP is Comcast (or Xfinity or whatever they are calling themselves these days).  Any time you fire up a web browser and type in a website’s address, two things happen:

  1. A service on the internet called the “Domain Name Service”, or DNS, turns the human-readable website address into a series of numbers or the “IP address” of that website.
  2. Your web browser then connects to that IP address and sends all of the communication between your computer and that site through your ISP.

Unless you have done something to go out of your way to change things, that DNS lookup (think of it as looking up a phone number in a phone book) is performed on a server owned by, you guessed it, your ISP.  Unless you are using HTTPS for your website address, all of the stuff that the site sends you and that you send the site is completely readable by, you guessed it, your ISP.

So if you are going to http://badthingsidontwantpeopletoknowilookat.com (please tell me that’s not a real website, I’m afraid to type it in) and browsing the content of that site, your ISP knows (and more importantly saves to a database) all of that “traffic”.  Therefore, the fact that your ISP is now soon to be legally allowed to sell that information to whomever they want to sell it to (they couldn’t before because of this silly little thing called “privacy law”) is a very big deal.

Now you may say, I’m not doing bad stuff, I have nothing to hide.  OK, let’s take a little thought experiment here.  Let’s say that you are looking up articles on weight loss and the bad effects of being overweight and reading them.  Nothing nefarious there, right?  Let’s then say that you have private health insurance – many people do.  Well there’s a revenue opportunity for your friendly neighborhood ISP!  Sell your web traffic to your insurance company!  They’d love to know that you might be overweight.  Then, they can charge you more the next time your premiums change.

Far-fetched you say?  Perhaps, but I think you get the idea.  Privacy is important and, while some folks may take it more seriously than others, bad things can happen if you aren’t legally allowed to protect your privacy.  So what can you, as a regular citizen do about this? First, you can contact your congress person and tell them you don’t like what they did.  Here’s the list of who voted in favor of passing this legislation.  Oh and it also shows how much in campaign contributions they got from the ISP lobbyists.  I’ll leave it up to you to connect the dots there.

Now what can you do to make it harder for your ISP to see what you are doing online?  Given that all of your internet traffic flows through them, it’s kind of hard, right?  Sort of…

First off, install a browser plugin on your laptop or desktop like “HTTPS Everywhere”.  This is a little free tool built by the EFF (an organization I really like and support) that forces as many websites as possible to use HTTPS for connectivity.  What this means is that, while your ISP still knows you went to a particular web page, they don’t know what it said or what you said to it because all of the data is encrypted.  This is the first step in undoing some of this damage.  Unfortunately they still know that you went to the site.

Another thing you can do is to install the Tor Browser Bundle.  What the heck is that?  Well, TOR (the acronym stands for “The Onion Router”) is a technology that routes your web traffic through a series of anonymous computers on the internet so that your ISP doesn’t know where you are heading and the site doesn’t know where you came from.  The only downside to TOR is that it is veeeeerrrrrrrryyyyyy sssssslllllloooooowwwww.  Also, there is some controversy as to who runs these servers on the Internet and what nefarious things they could be up to.  However, it will work in this particular use-case because your ISP won’t know where you went.

Another solution to this problem is to use a piece of technology called a Virtual Private Network (or VPN).  This technology opens a connection from your computer to the VPN service provider’s computer and encrypts all of your traffic before it leaves your machine.  The VPN service provider then forwards your traffic to its final destination and sends the results back to you.  If you used a VPN provider in Dallas TX and your lived in Washington DC, it would look like all of your web traffic originated in Dallas – outside of your ISP’s ability to capture and log.

There are some free ones out there such as proXPN but the free ones generally are slow.  You can purchase a monthly VPN service for typically less than $10 per month.  You do want to make sure you are working with a reputable company and one who doesn’t log all of your traffic (the problem we are trying to avoid) so choose carefully.  I’m a fan of Air VPN but there are others out there who are good as well.  A VPN will slow you down somewhat, but it is way faster than trying to use TOR in my experience.

The only problem with everything we have talked about so-far is that it requires that you have flawless operational security (opSec) 100% of the time.  The one time you forget to use TOR or fire up your VPN client and your data is now captured by the ISP and can be sold to whomever they want to sell it to.  What you really need is something that is always on and more foolproof.

Now you are talking about spending some money.  There are privacy protecting routers you can buy on the internet.  Are they any good?  I don’t know.  Are the companies who sell them any good?  Beats me.  This is the point where I put on my tinfoil hat and suggest a “roll your own” strategy.

What we need is a little computer that sits between your home network and your Internet Service Provider that does all of the encryption, routing, etc. so you don’t have to think about it.  I’m a big fan of the PC Engines APU2 single board computer.  For about $150, you can put together a completely silent (it is air cooled) 2 or 3 “NIC” (network interface card) router that even has its own WiFi capability.  I am a big fan of the OpenBSD operating system because its primary mission is to be secure.  The tagline on the website is “only two remote holes in the default install in a heck of a long time”.

This open source project takes its security very seriously.  I have a blog post that I wrote almost a year ago talking about how to build one of these little gems and turn it into a router/firewall for your home network.  If you start with that, you’ll have something that you can plug between your ISP-supplied cable-modem/router and your home network that will give you some additional security.  It is the launching pad for doing ever better stuff as we will see below.

The next step is that pesky DNS lookup.  Some folks might tell you to use Google’s DNS service or something like OpenDNS and now your DNS traffic doesn’t go to your ISP.  However, we know that Google loves to trade in data so that would probably be “out of the frying pan and into the fire” and OpenDNS is actually owned by Cisco, the largest purveyor of networking equipment in the world – probably not a good place to give that data to either.

This is a job for “DNS Crypt Proxy”!!!!  What this wonderful little open source gem does is take all of your DNS lookups and ship them (encrypted of course) to another country to be resolved.  It adds a tiny (read un-noticable) amount of overhead and gets that data out of the hands of your ISP.  You set this baby up on your APU2 box like I outlined in my blog post above, and your traffic is now significantly more private.

Now, if you want to take it one step further, you can use this wonderful little OpenBSD router you built to route all of your traffic through your VPN all of the time.  Here is a great blog post outlining how to do just that.  You probably want your VPN traffic to exit in a country that is more privacy friendly than the US or our allies.  Do a web search on “fourteen eyes” to understand where you can be more secure.  Some other things you can do are outlined in a website I reference frequently.  Installing the plugins recommended and switching to Firefox instead of Chrome is just good hygiene.

Finally, think about who you are using for web searches.  Don’t use Google if you can avoid it.  They are under no compunctions to protect your privacy.  Look into a search tool like DuckDuckGo (I know, goofy name but whatever).  You can configure your web browser to use that as your search tool by default and they don’t log your searches.  Your web searches can tell the world a heck of a lot about you…

So hopefully this gives you some good tips on how to better protect yourself from your own ISP.  Wait.  What?  Oh, you want to know what to do to help yourself with your smartphone.  Darn.  I was hoping we could skip that part because it’s very hard to do.  Oh well, I’ll give it a shot.

One thing you could do is to carry around a big battery for your APU2 box and duct tape that to your APU2 and your cell phone and only use WiFi to access it with a second cell phone taped to the APU2 to route your traffic to the ISP.  Hmm, while it would work, probably isn’t the most practical thing around.  However, you’d end up with great arm muscles from carrying around the additional weight…

Honestly, the problem with cell phones is that there isn’t a good solution.  For the most part on Android, you can’t install 100% of the plugins that I like for Firefox and there is zero way you can do anything like that on iOS.  The best you can do is to install an “OpenVPN” client “app” on your phone and use a service like AirVPN to route all of your traffic through the VPN.  You are on the hook for remembering to turn it on (because it turns off by itself if you aren’t using it for a period of time).  That’s about all I have right now.  I’ll probably learn more later so stay tuned if I have any better ideas down the road.

Thanks everyone, hope you enjoyed the read.

OpenBSD on Thinkpad X1 Carbon 4th Generation

In previous posts, I’ve detailed my movement away from all things Apple and my configuration of how to get OpenBSD installed on this laptop.  Now that I’ve been using it as my daily driver for a few months, I thought it would be worth revisiting the tweaks and configuration changes I’ve had to make to get really comfortable on this machine.  Given that, here we go…

For my particular model, I decided to turn all of the dials up to 11 in the configuration screens on the Lenovo website.  I have the 1TB NVMe drive, 16 GB of RAM and the 2560×1440 HiDPI display.  The latter has made things a bit interesting but since this is the future, I have been pressing on and solving the problems it has brought about!

The first thing I had to configure, based on some very nice work by Reyk was my /etc/xorg.conf file to turn on the efifb driver since we don’t have Skylake support yet in OpenBSD.  My particular configuration looks exactly like his:

Section "Monitor"
Identifier "default monitor"
DisplaySize 311 170
EndSection

Section “Device”
Identifier “default device”
Driver “wsfb”
EndSection

Section “Screen”
Identifier “default scren”
Device “default device”
Monitor “default monitor”
EndSection

Once you have that in place, you can do a “startx” from the command line to ensure that you have graphics working correctly.  After that, I installed my favorite BSD-licensed desktop environment, Lumina and configured my .xinitrc file to start it for me:

exec start-lumina-desktop

Now we have the fact that the trackpad gets a little wonky (a technical term) using the defaults.  Through some trial and error, I found the following line, added above the “exec start-lumina-desktop” in my .xinitrc gets things working the way I would like them to:

synclient VertScrollDelta=-107 HorizScrollDelta=-107 ClickFinger2=3 PalmDetect=1 PalmMinWidth=5 PalmMinZ=100 HorizHysteresis=50 VertHysteresis=50 ClickTime=50

Next, we have the HiDPI stuff to contend with.  I installed the xcursor-themes package and used the “whiteglass” theme.  By adding the following to your .profile, you will get the “desired” behavior (fonts big enough you can read, etc.) out of the HiDPI setup:

export GDK_SCALE=2
export XCURSOR_SIZE=36
export XCURSOR_PATH=/usr/local/lib/X11/icons
export XCURSOR_THEME=whiteglass

For Lumina, I configured the Lumina Glass color scheme and the Glass (System) theme with a 10 point font size.

With all of this done, you end up with a very usable desktop!

Say my Blog’s name!

Given that the title of this site is “functionally paranoid”, I noticed it had been a while since I had written anything focused on information security.  I thought about what would be a good topic and it hit me – how about my uber paranoid home networking setup?  So, put on your tinfoil hats my friends and buckle up – it’s going to be a wacky ride!

Physical Security

What’s the one thing that everyone forgets when thinking about information security?  Well, one is that there is always some guy named Bob who will mess things up (i.e. humans are the weak link because we can be socially engineered, etc.)  The other thing is that if you have physical access to anything, you can eventually brute force it.  However, there are things you can do to make things more difficult.  Here are some of the pros and cons:

  1.  BIOS passwords – Not something I’m typically impressed with.  Most can be avoided by opening up the machine, closing a jumper and powering it up to reset the NVRAM to factory defaults.  I don’t even bother with them.
  2. Full disk encryption – This one really rings my bell in a positive way.  If you can kill power to the box (either because the bad actor has to physically steal it and they aren’t carrying around a pile of car batteries and an inverter or because you can interrupt power to it some other way), then the disk will be encrypted.  The other beauty of this is that if a drive fails (and they all do eventually) you don’t have to have any privacy concerns about chucking it into an electronics recycler (or if you are a bad, bad person, into a landfill) because that data is effectively gibberish without the key (or without a long time to brute force it).
  3. Two factor auth for logins – I like this one as well.  I’m not a fan of biometrics because if your fingerprint is compromised (yes, it can happen – read about the department of defense background checks that were extracted by a bad agent – they included fingerprint images) you can’t exactly send off for a new finger.  Things like the YubiKey are pretty slick.  They require that you have the physical hardware key as well as the password so unless the bad actor lifted your physical key, they would have a much harder time with physical access to your hardware.

I use full disk encryption and two factor auth via a YubiKey on every piece of gear in my secure home network.

Network Segmentation

This is an area where I could get way more creative, because keeping network traffic isolated to particular subnets or VLANs is a great mechanism to prevent “island hopping”  (thank you Alan Jude and Chris Fisher for TechSnap!) when a bad actor gains access to one of your network resources; however, I’m keeping it simple for now.  The first thing I did was change the default password for administrative access to my ISP-supplied router.  There are some pretty interesting attacks that don’t compromise your machine, but instead use it to run Javascript to log into your non WAN exposed ISP router using the default userid and password and then do things like DNS hijacks, etc. to be able to maliciously impact every device on your network.

Just change the darned password to something else already!  Also, validate that your administrative interface for the thing isn’t exposed to the WAN side of the link.  Believe it or not, some of them ship that way.  I leave the ISP-supplied WiFi access point to provide a network I make available to my guests because it sits entirely outside of my secure network.

The next thing I do is connect the LAN side of the ISP-supplied router to the WAN side of my own router.  I use an APU2 box that I built for around $150 running OpenBSD as my router for my secure network.  I use the wonderful pf firewall that is supplied with OpenBSD to filter packets and map ports as I want to.  It has a very understandable syntax and has great performance.  The LAN side of this router goes to my managed switch (which I hope to replace with switchd running on an OpenBSD box at some point) which serves as the core backbone of my secure network.  Plugged also into this switch is my WiFi access point running in bridge mode for my network so that I can have WiFi access (with a very secure WPA2 password) to my secure network.

I know that any true security expert will scoff that I go to all of this trouble and then hook a WiFi access point up, but this is an area where I tilt the scales away from security towards convenience.  A good long WPA2 password isn’t ideal, but it isn’t crackable in a short period of time without a determined attacker who has lots of horsepower.  For example, a 16-character randomly generated password that uses uppercase, lowercase, numbers and special characters would have so many permutations, that it would take a freakishly long time to brute force.

Privacy

So security is good, but privacy is good as well.  You might not get hacked, but if you aren’t careful about privacy, you will be leaving breadcrumbs all over the internet – which is not good.  To that end, think about who probably knows the most about your online traffic – that’s right, your ISP.  Even if you do everything over HTTPS and use a VPN, you are probably still using them (or heaven forbid Google!) for your DNS resolution.  Enter DNSCrypt Proxy!  This little jewel takes all of your DNS traffic and egresses it to the Internet outside of your ISPs network.  I actually use a public resolver in Europe for mine.

I use Unbound as my DNS resolver on my local network (with all UDP port 53 traffic redirected to it by pf so I don’t have to configure anything on the clients) and then forward the traffic to DNSCrypt Proxy, caching the results in Unbound.  I notice ZERO performance penalty for this and it greatly enhances privacy.  This combination of Unbound and DNSCrypt Proxy works very well together.  You can even have redundancy by having multiple upstream resolvers running on different ports (basically run the DNSCrypt Proxy daemon multiple times pointing to different public resolvers).

I also use Firefox exclusively for my web browsing.  By leveraging the tips on this page, you can lock it down to do a great job of privacy protection.  The fact that your laptop’s battery drain rate can be used to fingerprint your browser completely trips me out but hey – that’s the world we live in.

For times I am not connected to my secure network, I use a non US-based VPN provider (I like AirVPN.org) and choose an endpoint that isn’t in a Fourteen Eyes country for egress.  I  have given thought at various times to doing that using pf on my home network but the performance hit is still a bit too much for me.  Likely that will improve over time.  The nice thing about AirVPN is that it uses the OpenVPN client meaning I can run it on my mobile devices as well.

You can get even more secure with things like the TOR browser bundle, but be careful how you configure it.  You can actually make yourself less secure than you think if you aren’t educated on how everything works.

The Cloud

This is where it starts to get weird for most folks.  They have pretty good security hygiene and maybe are fairly good about privacy, but they end up storing everything in a cloud service.  Remember, another way of saying “the cloud” is “some other guy’s computer”.  If you really to care about your privacy, then you should avoid cloud services like the plague.  However, that starts to get really hard and you have the good old balancing act between security and convenience.

I recently decided I would try to live a cloud-free life and I’ll give you a bit of a synopsis on it.  I discovered a wonderful OpenSource project called FreeNAS.  What this little gem does is allow you to install a FreeBSD/zfs file server appliance on amd64 hardware and have a slick administrative web interface for managing it.  I picked up a nice SuperMicro motherboard and chassis that has 4 hot swap drive bays (and two internal bays that I used to mirror the boot volume on) and am rocking the zfs lifestyle!  (Thanks Alan Jude!)

One of the nicest features of the FreeNAS is that it provides the ability to leverage the FreeBSD jail functionality in an easy to use way.  It also has plugins but the security on those is a bit sketchy (old versions of libraries, etc.) so I decided to roll my own.  I created two jails – one to run OwnCloud (yeah, I know about NextCloud and might switch at some point) and the other to run a full SMTP/IMAP email server stack.  I used Lets Encrypt to generate the SSL certificates and made sure I hit an A on SSLLabs before I did anything else.

OwnCloud includes the ability to install plugins and there are some dandy ones out there.  I installed a notes one that had iOS and Android clients (buh-bye Evernote with your attempted changed privacy policy which you rolled back but I see how you really are now).  I also installed a Calendar plugin that had CalDAV support (see ya’ Google Calendar!) and a CardDAV contacts plugin (later Google Contacts).  I got all of them up and running (along with the mail server Jail – use mxtoolbox to check your work here) and can now stop using the main cloud services I had previously been running.

How did I get around my ISP blocking common IMAP/SMTP ports you ask?  I set up a relay server on a VPS server that I rent for $8 per month.  I started out on one based in the US but my better thinking prevailed and I found a provider in a non fourteen eyes country.  Everything seems to be working OK once I got all of the right security / anti-spam bits in place but only time will tell.

This path isn’t for everyone because you are now fully in charge of the security and reliability of your mail / calendar / contact / note infrastructure.  Also, you just increased your attack surface so be extra careful there!

Backups

This is another fun one.  You go to all of this trouble and then you store your offsite data backup (you DO back up your data offsite, right?!??!?!!) in some cloud service like Carbonite or CrashPlan.  As convenient and cost-friendly as that is, you have no idea how many copies of your data they make in their datacenter operations and even if you leave them, they are probably still floating around in their cloud infrastructure somewhere.

Enter TarSnap – a company that advertises itself as “Online Backups for the Truly Paranoid”.  It brings a tear to my eye – a kindred spirit!  🙂  Thanks again to Alan Jude and Kris Moore from the BSD Now podcast for turning me onto this company.  It has a very easy command syntax (yes, it isn’t a GUI tool – suck it up buttercup, you wanted to learn the shell didn’t you?) and even allows you to compile the thing from source if you want to.

The way it works is that it encrypts the data on your machine before ever sending out to the network.  As long as you keep your private key on the encrypted drive on the machine you are backing up, it’s pretty darned safe although you might want to keep a copy of it somewhere that is physically secure and preferably offsite (perhaps a GPG encrypted file on your mail relay VPS).  The cost is based on the amount of data you store and the network traffic to get it to/from their service.  However, for the type of data I’m storing up there, it isn’t crazy expensive and it de-dupes the heck out of it to further reduce your costs.

Internet of Things

This is an easy one – avoid any “internet of things” device that connects to anything outside of your network!  That way you avoid these poorly implemented security nightmare devices becoming a public menace and joining a DDOS botnet.

OK, if you absolutely can’t live without your internet connected ice cube tray, the best way to secure these devices is to put them completely outside of your secure network.  Hook them up to the WiFi made available by your ISP supplied router (where I put my guest traffic).  It’s still (theoretically) sitting behind a “secure” router but it sure as heck isn’t on your secure network.

You could do other things like having a VLAN dedicated to them and locking them down by MAC address.  The problem is there is always some clever 8th grader in Germany who will still find a way to exploit them and island hop to your other machines.  Best keep them at arm’s length on the outside of your real router (you do run an OpenBSD router, right!?!??!?!)

Conclusion

There are other things that I’ll want to do eventually (perhaps putting my WiFi traffic on its own network segment with host isolation mode, switching to certificates for my WiFi access, or installing an IDS tool like snort), but for now, this is where I stand.  I’d appreciate any feedback or suggestions from folks out there with ideas on how to lock things down more.

Where the rubber meets the road (part two)

So in my last post, I detailed what it took to get Arch Linux running on my brand spanking new Thinkpad X1 Carbon (4th Generation) and was getting ready to install OpenBSD on the same disk.  In the intervening time, I banged my head against the wall over and over again trying to figure out what my problem was on OpenBSD with not seeing the partition structure correctly on the disk and have finally given up.  Time to NUKE AND PAVE baby!

I backed up my data from Arch to an external drive, downloaded the latest snapshot installer for OpenBSD, burned it to a thumb drive and booted the laptop from it.  My goal is to reverse the approach I took last time and install OpenBSD first.  I’m going to GPT partition the disk and then, once I have it working correctly, reboot into the Arch installer and install Arch second.

After booting the OpenBSD installer, I bailed out to the (s)hell at the prompt and rewrote the partition on my 1TB NVMe drive with:

# fdisk -gi -b 4096000 sd0

This repartitions the drive with GPT and creates a boot EFI volume that is 2G in size.  I then used:

# fdisk -e sd0

to edit the drive and resized the OpenBSD partition to 490 GB in size.  Next, I used disk label to create my encrypted partitions for swap and data:

# disklabel -E sd0

I created an “a” partition (don’t forget to make it RAID type)  hat took the rest of the OpenBSD area for my encrypted data drive.  I then used bioctl to create the encrypted volume:

# bioctl -c C -l /dev/sd0a softraid0

The crypto volume was now available as /dev/sd2 so I used “exit” to get back to the installer and installed OpenBSD on that volume as I normally do.  To be safe, I mounted the EFI partition (/dev/sd0i in my case) and copied the /efi/boot files into /efi/openbsd.  I vaguely recall a Linux installer overwriting the OpenBSD files at some point in my checkered past so it never hurts to be safe.

I then reformatted my thumb drive as FAT32 and grabbed the missing firmware tarballs I needed to, rebooted the new OpenBSD install, and untarred them into /etc/firmware.  After rebooting and verifying that everything was working fine, I downloaded the latest Arch installer, burned it to the thumb drive and booted from that.

I installed Arch as I detailed in my last post; however, when I fired up gdisk I got a weird error message:

“Warning! Disk size is smaller than the main header indicates! Loading secondary header from the last sector of the disk! You should use ‘v’ to verify disk integrity, and perhaps options on the expert’s menu to repair the disk.”

Immediately after this, I saw a second warning:

“Caution: Invalid backup GPT header, but valid main header; regenerating backup header from main header.”

And, not to be outdone, there was a third:

“Warning! Main and backup partition tables differ! Use the ‘c’ and ‘e’ options on the recovery & transformation menu to examine the two tables.”

Finally (not kidding), there was a fourth:

“Warning! One or more CRCs don’t match. You should repair the disk!”

Given all of that, I thought to myself, “This is probably why I couldn’t see the disk properly when I partitioned it under Linux on the OpenBSD side.  I’ll let it repair things and I should be good to go.”  I then followed the recommendation and repaired things, using the primary GPT table to recreate the backup one.  I then installed Arch and figured I was good to go.

Now I was ready to put on rEFInd as my boot manager for switching between the two.  I installed the “refind-efi” package using pacman:

$ sudo pacman -S refind-efi

I then ran refind-install and had what I needed working.  I rebooted to OpenBSD and was greeted with a kernel panic because it couldn’t find my root volume.  Uh oh.

I repeated this process a couple of additional times (starting from scratch, partitioning under OpenBSD, installing Arch, repairing the disk, etc.) and not surprisingly got exactly the same results.

OK.  What to do now.  I decided to rock this old school and dump the whole crazy GPT thing and do this MBR.  After all, I only need three partitions anyhow (the EFI one, the luks encrypted partition for Linux and the softraid encrypted partition for OpenBSD).

I booted up off of the OpenBSD installer thumb drive, jumped out to the (s)hell and did:

# fdisk -i -b 4096000 /dev/sd0

I then resized down the default OpenBSD partition (#3 by the way) to half its size using fdisk, and used disklabel to create a single “a” slice that took up the entire area (make it RAID for the type).  Then, bioctl and I created the encrypted softraid container and I rejoined the installer and installed normally on the new virtual device (sd2 in my case).

Installing Arch was slightly tricker this time because apparently very few people use EFI and MBR together, let alone with luks and lvm.  🙂

The trick was realizing that I couldn’t use systemd-boot as my boot manager because it doesn’t work with MBR nor do they apparently have plans to make it work (I get it, this is the true definition of a corner case).  I dropped back to GRUB for my boot manager with the only trick being that I changed the GRUB_CMDLINE_LINUX variable in /etc/default/grub to be:

GRUB_CMDLINE_LINUX=”cryptdevice=/dev/nvme0n1p2:lvmpool”

I then ran the configuration file generator:

# grub-mkconfig -o /boot/grub/grub.cfg

At this point, I crossed my fingers and booted back and forth (using the F12 boot menu on the Thinkpad and choosing the nvme drive booted OpenBSD and I could choose “grub” from the boot list to boot Linux) between OpenBSD and Arch multiple times to ensure there wasn’t any funny business going on.

Victory!!!!!!!

Now that I have everything working, I’ll restore my config and data to Arch, configure OpenBSD the way I like it and get moving.  I’ll take some time and drop a note on the tech@ mailing list for OpenBSD to see if they can figure out what the GPT problem was I was running into.  Hopefully it will make that part of the code stronger to get an edge-case bug report like this.

Where the rubber meets the road… (part one)

So, if you read my recent post on setting up a virtual machine multi-boot image with OpenBSD and Linux, you’ll be familiar with the background for this post.  Today, I’m going to detail the exact steps I took to set up my new work laptop, a Thinkpad X1 Carbon (4th Generation Skylake) laptop running Arch Linux and OpenBSD.  Since we don’t have solid support yet for Skylake, there are some interesting workarounds that others, far smarter than me, were kind enough to leave on the Internet.  I’ll detail those and give credit to the original authors below.

First off, let’s talk about why I’m doing this.  As some of you may know, I’ve been a bit of an Apple fanboi for quite some time.  So the concept of me moving off of Apple hardware to a <yuck> “PC” </yuck> was something that, quite frankly freaked out some of my friends.  One of them described it as one of the four signs of the apocalypse!

So, given that, why do I want to do it.  There is an old saying that you can throw a frog in a pot of boiling water and it will jump out to save itself, but if you put it in a pot of cool water and slowly raise the temperature, it will let itself be boiled to death because the change was so gradual.  Gross!  However, pretty apt to describe my situation.  Every Apple product that comes out is (read this in a Jony Ive accent) “x percent thinner and lighter than the previous one!”  Well, I’ve often wished that Apple would do certain things to the hardware – make it more standard, faster, more RAM, more battery life, etc.  However, I never asked for “thinner and lighter”.

As a result, when I saw the 12″ Macbook released I thought it would be a nice travel laptop and overlooked the fact that it had a single USB-C port (who doesn’t like dongles), in a stiff breeze it might blow over on a table, the CPU was massively under-powered and the NVMe hard drive in it is so weird it actually reports the wrong PCI device ID when queried by the bus.  But whatever, it was thinner and lighter.  Nevermind that I ended up carrying more dongles and adapters in my backpack to offset the weight “loss” and that USB-C can sometimes be finicky and a pain in the butt.  Oh and that extra thin keyboard.  What a pain to type on.  Oh and don’t get me started on the number of times I had to mash the power button for a hard reboot when it didn’t wake from sleep…

Then came the spate of bugs in iOS.  I can’t tell you how many times I’ve had to hard reboot my phone just to do things like make calls.  Oh, and the recurring one where my phone is ringing and the UI is locked up and won’t swipe to answer – awesome!  Then came macOS Sierra.  With kernel panics weekly (hadn’t seen one of those since Tiger), the 6-10 times per day where my external display blanks for 3-5 seconds and that Mail and Calendar crash daily for me.  I had hoped that the .1 release would fix the problems – it didn’t.

So I finally realized I was loving a glorified historical Apple and that I really wasn’t a happy user.  Then came the new Macbook Pro.  For the first time, I saw a new laptop that I didn’t actually want to buy.  As a VI user, the thought of having a fake escape key sounded awful and the little touch screen was little more than a gimmick.  Plus the under powered hardware and that “blessed” thin keyboard again?  No thanks.  So I made the hard decision to make the switch, did my research and settled on this setup.  Whew.  Some backstory!

OK, so I planned on getting the new laptop out of its box, shrinking the Windows partition, adding in Arch and OpenBSD (both with full disk encryption – remember the name of this blog).  Should be fun.

Shrinking the Windows partition was pretty simple.  I booted up, let it do it’s “first time you boot Windows” business, rebooted just to be sure I was in a clean state.  Rebooted and went into Disk Manager.  Resized the partition down to 200G, created three blank partitions – Arch /boot, Arch encrypted luks volume and OpenBSD “slice” (I chose to leave them unassigned relative to drive letters or paths and didn’t format them, I just wanted place-holders) and rebooted to be sure it all still worked.  It did.

Here’s where it got fun.  Time to boot the Arch install media and get started.  I based my install on this excellent blog post.  Also, based on this post, I took the recommendation and turned off secure boot.  The first thing I noticed was that the NVMe drive showed up as a different type of animal on Arch than I had expected.  It wasn’t a /dev/sd* device, it was /dev/nvme0n1 and my partitions were /dev/nvme0n1p1, etc.  So here is where I give you kids a PRO TIP.  When you are creating your encrypted luks volume, don’t get confused by the fact that there is a number in the base device name and think you are telling it to encrypt that partition and instead create a luks volume on the entire drive.  Yep.  That’s what I did.  Darnit! I could use the rescue partition – oh wait, I managed to whack that too.

Oh well, I don’t like Windows anyhow, so I’ll salve my pride by saying I intended for this to be a dual-boot install not a triple-boot one.  If I ever need to update the BIOS I’ll cross that bridge when I come to it.  So, starting from the top.  I booted the Arch install media, put a new GPT partition table on the disk and created three partitions.  A /boot partition that was type 8300, a luks container for my encrypted volume (type 8300 as well initially), and a type a600 partition for my OpenBSD slice.  I formatted the /boot partition as FAT32 and created a root, home and swapfile in the encrypted luks/lvm partition.  I did the install of Arch into it, rebooted and all was good.

I then set up a gnome desktop with gdm as the display manager (enabled on boot), created my default user (which I added to the wheel group and tweaked sudoers to allow me to run root commands when I need to) and installed my apps.  I rebooted and everything was swell.  The system handled my HiDPI display just fine, recognized all of the devices and even had hibernate / sleep working out of the box.  The only tweak I had to do was to disable Wayland because I intend to use Virtualbox to run my Windows VM for Skype for Business (the only reason I ever have to go to Windows these days).

In total, I ended up adding the following packages from the base Arch repository:

  • xorg-server
  • xorg-server-utils
  • xf86-video-intel (select libinput for the trackpad based on the earlier post I referenced on turning off secure boot)
  • mesa-libgl
  • gnome
  • gnome-shell
  • gnome-extra
  • gnome-initial-setup
  • gnome-software
  • gnome-tweak-tool
  • gnome-shell-extensions
  • evolution
  • evolution-ews (interestingly I had to first set up a non Exchange mail account before I could successfully set up an Exchange account)
  • firefox (follow this hardening guide for privacy optimization)
  • libreoffice (I installed stable because “fresh” seemed to oddly have some HiDPI problems)
  • aspell-en
  • chromium (for the rare times I have a site that doesn’t work on my locked down version of firefox)
  • openconnect and networkmanager-openconnect (we have a Cisco firewall at work)
  • vim (because it’s better than Emacs <grin>)
  • networkmanager
  • nmap
  • git
  • virtualbox (see this wiki entry on how to configure)
  • virtualbox-guest-iso
  • vlc and qt4 (so I can have the GUI)
  • ufw
  • openvpn and networkmanager-openvpn
  • freerdp
  • remmina
  • gstreamer

From the Arch User Repository (AUR), I installed the following:

  • slack-desktop
  • owncloud-client-service
  • skypeforlinux-bin
  • bluejeans
  • font-manager
  • hipchat4
  • tor-browser-en

OK.  Now for the fun part – installing OpenBSD.  I rebooted off of the latest snapshot install media, popped out to the shell and discovered to my dismay that it looks like my NVMe drive is not supported yet.  Dangit!  OK.  I think I know how to fix this, I’ll build a patched kernel and see if I can get around this – darned PCI device identifiers.  That will have to go in a separate blog post.

Making it real

In my last post, I talked about creating a multiboot Virtual Machine that had OpenBSD, FreeBSD and Linux on it.  In that post, I mentioned that my next step would be to actually set up a laptop on real hardware with this configuration and document the process.

In the intervening time, two big events and a lot of little ones occurred that are interesting to take note of here:

  1.  Apple released the new MacbookPro computers.  To say I am underwhelmed would be an understatement.  I recall a time when Apple not only was innovative and aggressive, but also built some of the best hardware I’ve ever had the privilege to use (more on that with #2 below). Unfortunately adding a tiny little OLED touch strip above the keyboard and further reducing the ports you have available to you doesn’t exactly align with that picture in my head.  But hey, it’s a free country (at least it still is, I’m writing this before the US Presidential election ) so YMMV.  You might really dig the new hardware from our friends in Cupertino.
  2. I bent my 12″ Macbook.  Yes, you read that right.  I bent the thing.  Not by dropping it off of a tall building or slamming it in a car door, but by simply carrying it in my backpack and putting it into the overhead bin on my flight.  It’s not really noticeable except for the fact that it now wobbles when I type on it.  And yes, I tested it on multiple flat surfaces.  And no, it isn’t that the little rubber foot fell off of it.
  3. I’m feeling like I’m being pecked to death by the numerous bugs I’m seeing in macOS Sierra.  There was a time a couple of releases ago where WiFi was a real problem.  Well guess what, that’s back.  I also am a big fan of Apple’s built-in mail and calendar programs.  Well, they randomly crash (or minimize when I click on another window) several times a week.  Oh, and did I mention the hibernation and resume from suspend problems I am having?  Oh, and the fact that while I’m typing this, the external monitor that is hooked up to my wobbly laptop via Apple’s own dongle has gone black for 1-3 seconds several times?

It just doesn’t feel like I’m the core demographic that they are targeting any more.  To that end, I’m seriously considering (and I know this will freak out anyone who knows me), stepping away from the Apple ecosystem.  Now the privacy nightmare malware application known as Windows 10 would not be where I land, so I begin to contemplate a world where I live on OpenBSD full time and work as well as at home.

Unfortunately, there are “corporate” things that I do that will require me to periodically use Windows (we have internal web-based tools that absolutely require Internet Explorer) and Microsoft Office is a necessary evil in the corporate world.  I know I could limp along with a remote desktop client to get to IE and possibly use the web tools in Office365, but I expect that I would run into pain that way.  This means that somewhere on this mythical machine, I would need a Windows VM.

So suddenly, my idea of doing a little experiment of dual booting a laptop got serious – as in deadly serious.  This “test” I’m conducting will be an experiment to see if I can walk away from macOS as my daily driver.  Now, ironically enough, the laptop I’m going to test this out on is an older MacbookPro 13 inch (an 11,1 model).  How is that for funny.  If I can live this way for a few weeks, however, I would be moving officially to a Lenovo Thinkpad X1 Carbon as it is the only small laptop I can find currently that has good open source OS support, a 1TB SSD and 16 GB of RAM.

OK, enough talking onto documenting what I did.  First, I started off by taking this poor old MacbookPro and running DBAN on it.  For those of you who aren’t familiar with DBAN, it stands for Darik’s Boot and Nuke.  Its a collection of utilities you can boot off of a thumb drive that will allow you to securely wipe a hard drive.  I erased the drive (including the EFI partition) on the MacbookPro so I could have a fresh start.

The next step was to download Ubuntu 16.10 onto a thumb drive and install it.  While I would have preferred to use Arch (I like rolling distros), I don’t want to ever be in a situation where I whack my work machine because I waited to long between updates, etc.  Ubuntu (or as I like to jokingly call it – Grandma’s Linux) is probably my better choice here.

Installing it was interesting though.  I couldn’t take the default and let it take the entire drive, plus I wanted to encrypt the disk so I did the manual configuration.  Understanding how to set things up took a couple of attempts and I’m still not totally satisfied with my end result.  In the end, I created an EFI system partition at the front of the disk that is 1,024MB in size, a /boot EXT4 partition that is the same size, and a 200GB encrypted volume that has in it a single EXT4 volume that is mounted as / to the filesystem.  What is missing is a swap drive.  I couldn’t create a “normal” swap partition because the Ubuntu installer informed me that Linux doesn’t encrypt its swap (that was a surprise to me as an OpenBSD user) and it wouldn’t even let me say “who cares do it anyhow.”  It just flat out refused to proceed until I did a swap-less install.  I got grumbled at by the installer but it let me proceed.

Once I rebooted (and the Ubuntu installer religiously hangs on me after installation so I did my typical wait 5 minutes and hold down on the power button – I think I am beginning to see why Bryan Cantrill calls Linux a “dumpster fire”) and rebooted.  I was prompted for my volume encryption password and was dropped very quickly at the desktop.  Now I created a swap FILE (not a partition):

$ sudo fallocate -l 20g /mnt/20GB.swap
$ sudo chmod 600 /mnt/20GB.swap
$ sudo mkswap /mnt/20GB.swap
$ sudo echo "/mnt/20GB.swap none swap sw 0 0" >> /etc/fstab

After doing that, I rebooted and confirmed that I had a working swapfile.

Next, I downloaded the latest snapshot of OpenBSD 6.0-current and dd’ed it to a thumb drive.  I rebooted, installed OpenBSD and promptly stepped all over the drive and made it unbootable.  Oops!

At that point, I decided I really wanted to better understand what I was doing and what better way to do that than to install Arch and do all of the LUKS stuff by hand.  I found a handy-dandy article that walked me through it.  I first downloaded the Arch install ISO and dd’ed it to a thumb drive which I booted from in UEFI mode on the Mac.

# wifi-menu

# timedatectl set-ntp true

# lsblk

From there I was able to find out that I needed to use the block device /dev/sda as my target (the internal drive on the laptop).  I then used gdisk to create the partitions.  I first issued the “o” command to wipe and recreate the GPT partition table.  I then created partition 1 as a 1,024M /boot and EFI system partition with a partition type of ef00 so that it was identified properly.  I then created partition 2 as a standard ext4 partition (this will be the LUKS partition) that consumed half the free space and partition 3 as an ext4 that consumed the rest (this will be where I will be installing OpenBSD).

# cryptsetup -c aes-xts-plain64 -s 512 -h sha512 -i 5000 -y luksFormat /dev/sda2
# cryptsetup luksOpen /dev/sda2 crypt
# lvm pvcreate /dev/mapper/crypt
# lvm vgcreate lvmpool /dev/mapper/crypt
# lvcreate -L 35GB -n root lvmpool
# lvcreate -L 16GB -n swap lvmpool
# lvcreate -l 100%FREE -n home lvmpool
# mkfs.fat -F32 /dev/sda1
# mkfs.ext4 /dev/mapper/lvmpool-root
# mkfs.ext4 /dev/mapper/lvmpool-home
# mkswap /dev/mapper/lvmpool-swap
# swapon /dev/mapper/lvmpool-swap
# mount /dev/mapper/lvmpool-root /mnt
# mkdir -p /mnt/boot
# mount /dev/sdX1 /mnt/boot
# mkdir -p /mnt/home
# mount /dev/mapper/lvmpool-home /mnt/home

Now that I have the volumes created and mounted, I installed Arch as I normally would.  By the way, how weird is it that Linux numbers its devices starting at 1?  I would have expected the first partition to be /dev/sda0 (which of course to an OpenBSD guy such as myself seems dyslexic as it should be /dev/sd0a like we do it).  🙂

Before rebooting, you have to enable the crypto in the kernel by adding “encrypt” and “lvm2” between “block” and “filesystems” in the /etc/mkinitcpio.conf file and then regenerate the initramfs by issuing the “mkinitcpio -p linux” command.  Install the simple bootloader via “bootctl install”.  Finally in /boot/loader/entries/arch.conf file, add the following:

linux /vmlinuz-linux
initrd /initramfs-linux.img
options cryptdevice=/dev/sda2:crypt ro root=/dev/mapper/lvmpool-root rw

Set your hostname in /etc/hostname, install the necessary wifi software with “pacman -S iw wpa_supplicant dialog” and change the root password with “passwd”.  Press CTRL+D to get out of your chroot, unmount /mnt (where you just installed Arch) and reboot.  You should be presented with a prompt to enter your crypto password and then you boot to the console just as you normally would.

Now to install OpenBSD.  I boot from the install60.fs USB stick as I normally would and immediately drop to the shell from the installer prompt.  Use fdisk to change the ext4 partition into an OpenBSD one, then use disklabel on that partition to create a swap file and the installer space.  Following this handy guide, I then created my softraid crypto filesystem and installed as I normally do.

Upon reboot, I am taken into Arch where I can install rEFInd from the console.  Another reboot and (although the icons and naming is pretty goofy by default on rEFInd in this case), I can safely switch between the two operating systems.

Mission accomplished!  I’ll clean things up with the rEFInd configuration, add puffy as an icon and get to it configuring my environments on both sides.  While it ended up being a long journey, I learned a lot on this and ended up having a test machine to now see if I can make the big switch away from macOS and Apple hardware.

Helping out an Internet Friend…

While I know everyone is probably waiting impatiently for my Fortran follow-up (I assure you all, I do have a working Fortran-based web application development framework – I just want to pretty it up a bit more <grin>), I thought I’d take a moment and diverge to talk about something I’m doing to help a fellow BSD aficionado out.

I’m in the process of standing up a BSD Users Group (BUG) in Indianapolis, IN (https://indybug.org) and in the process, met someone who was interested in dual booting a BSD and Linux on his HP laptop.  In the past, I cheated with my Thinkpad x220 (which is now an x230 but that’s another story) by just using two drives and leveraging the boot menu to decide which one to boot.  My primary SSD is OpenBSD-current and my m.SATA drive runs Arch for those times that I need something from the Linux world.

Well, this individual only has the ability to have a single drive in his laptop and would like to do the hard thing of dual booting.  Also, it turns out this is a UEFI machine.  Given that, I decided to roll up my sleeves and see what I could make happen.  Since we have EFI support in OpenBSD 6.0 now, I thought I’d give it a whirl in a Virtual Machine under VMWare Fusion on my Mac.

First things first, I needed to figure out how to set up a UEFI VM.  There were no obvious checkboxes to be found in the UI so, after some Googling, I found out that I just needed to add a line to the *.vmx file that contains the settings for the VM.  I created a generic 64-bit VM with 2 cores, a 40 GB IDE drive and 2GB of RAM, added firmware = “efi” to the VMX file and fired it up to see what happens.  Sure enough, I got an EFI boot screen from VMWare instead of the traditional BIOS one.  Looks like that first step was easier than I thought.

Next, I downloaded the OpenBSD 6.0 install60.iso image and booted.  No joy.  I fiddled around with it for some time and was unable to boot that image from the EFI version of VMWare Fusion.  Given that there is more than one way to skin a cat, I downloaded the latest Ubuntu 16.04 desktop ISO and was able to successfully boot that.  Therefore, I decided I’d install Ubuntu first and then layer on the OpenBSD install from that.

The only non default settings I chose were:

  • Install 3rd party software
  • Installation type of “something else” so I could control the partitioning
    • I created a 64MB EFI partition at the front of the disk
    • Next, I created a 20GB primary partition at the beginning of the space, mounted as the root (/) filesystem
    • I then added a 4096MB swap partition for Ubuntu
    • Finally, I used the rest of the free space to create a Reserved BIOS Boot AreaFAT32 partition that was not associated with a mount point – this is where I will be installing OpenBSD
      • You will get a warning about the FAT32 partition not being used and that’s fine.  We don’t want Linux to use it.  That will be where we’ll install OpenBSD
      • If you are asked about “forcing the system to be UEFI only” that’s what you want to do.  Remember, our goal is to create a UEFI dual booting system.
  • Everything else should be the defaults.  Just create your user and sit back and wait for the install to finish.

So I got that done and got ready to boot off of the OpenBSD installation ISO.  And that’s when it hit me.  Nothing has changed since I tried to boot off of it last time.  Doh!  OK, so I downloaded the install60.fs file and dd’ed it to a USB drive, mapped the USB drive to the VM and rebooted.  Hitting <ESC> quickly and repeatedly, I ended up at the UEFI boot device selection screen in VMWare and was able to boot off of the USB drive.  So I ended up at the initial install prompt.

I mashed through the defaults in the OpenBSD installer until I got to the disk partitioning.  Since I told VMWare to make my hard drive an IDE one, I knew I was playing around with wd0 and not sd0 (my USB key).  I dumped into fdisk by selecting to (E)dit the partition scheme and saw my setup from Linux.  First was the EFI partition (I am guessing I’ll have to copy my bootx64.efi file to that at some point), second was the Linux etx4 partition, third was my Linux swap partition and fourth was a weird looking one that is the “Reserved BIOS Boot” partition.  That’s the one I’ll fiddle with.

Issuing the command “edit 3” allowed me to fiddle with that partition #3 (remember, we start counting at zero).  I set it’s type to “A6” (OpenBSD) and then took the defaults with the exception of naming it “OpenBSD”.  A quick “write” followed by a “quit” allowed me to update my new partition and get back to the installer.

I took the (a)uto layout for the filesystem setup and let the installer create all of the filesystems.  Once that was done, I continued mashing defaults for the rest of the install and just let it do a full install of everything (including X) on the system with the files coming from the USB drive.  When I got dumped to the prompt at the end, that’s where I knew I needed to play around a bit.

I mounted the EFI partition (mount /mnt /dev/wd0i) and poked around.  It looks like the OpenBSD installer already created an /EFI/BOOT folder and stuck our boot loaders in it.  This might mean I can just reboot from the hard drive and it will just work.  I’ll give it a shot and see.  Sure enough, mashing <ESC> at the boot screen allowed me to choose between Ubuntu (the default) and an unlabeled hot mess that turned out to be OpenBSD.  Now I’m going to push the envelope and try to get rEFInd on this thing to have a pretty boot menu.

I rebooted to Ubuntu and hit the rEFInd web page so I could download the Linux installer.  I installed the .deb file and the ran “sudo refind-install”.  It installed just fine.  I rebooted and sure enough I could switch between Ubuntu and a weird icon in the middle that turned out to be OpenBSD.

I guess I have achieved my goal of dual booting Linux and OpenBSD on a UEFI install.  Now to see if my new friend can do the same by following this.