Functionally Paranoid

As I work on some ports for OpenBSD, one of the issues that has been mentally nagging at me is the idea that I might have a great AMD64 port but it won’t actually work on other processor architectures.  To me, one of the big draws of OpenBSD is the fact that you can take hardware that others might consider “obsolete”, put a modern operating system on it and then have a useful machine again.

With this in mind, I decided to see what I could do about getting some “other” processors into my stable.

Right off the bat, I realized I had an old G4 Mac Mini in a closet that would make a nice start.  I installed 5.7 from my release media (anyone running OpenBSD should buy the media to support the project – even if you don’t use it the stickers are always cool <g>).  I immediately ran into problems because the approach I took to partitioning was not creating the MSDOS partition with the bootloader.  Turns out that the path of least resistance is to edit the default partition scheme and go from there.  The bad idea (that I kept bumping my head into the wall over) was to either start from scratch or immediately do a “d *” from disklabel in the installer.

Once I had my PowerPC 32-bit system up and running, I realized I had another processor architecture I could try.  Sitting in that same closet was an old Thinkpad T21.  I pulled it out, spent 20 minutes looking for the power supply and was able to load the system pretty quickly.  Now I have three processor architectures to compile and test on natively.

Well, those were some pretty tame ones.  I wanted to try my hand at something more exotic so I read the docs carefully, looking for ideal machines that would be powerful enough to be useful, wouldn’t take up a lot of space and would also be affordable on the second-hand market.  I found a Sparc64 based machine, the Sun Blade 100.  This is not a “blade” server but instead is a workstation with a funny name.  I found one for less than $30 on eBay and went for it.

Setting up OpenBSD on this hardware was surprisingly simple.  I read through the documentation and then followed it pretty carefully.  It was surprisingly similar to setting up a PowerPC machine and before too much longer I had it up and running.

I’m eyeing an SGI machine and possibly an Alpha.  It seems that the PA-RISC machines are surprisingly expensive for even older boxes.  Apparently the resale market for them is much stronger or the supply is more constrained.  I’m not sure there.

At the end of the day, building and testing on these other platforms is useful to the community and the “exotic” nature of them in today’s homogeneous computing landscape lends a little spice to working on things.  I’m glad my path took this turn.

OK.  At this point, I have a working router for my internal network.  I’m using pf as my firewall, have a heavily (in my mind) secured ssh path into things and I’m having fun watching port scanners hit me and try to log into my ssh (which doesn’t accept passwords).  I think it’s time to see if I can break everything by setting up a mail server!  I mean, how hard can it be?

And that’s how all of this started.  I figured it would be as simple as finding a good tutorial on the internet, standing up the right config changes on my machine and presto – I’ll have my own email server with the data being stored on my own hardware, not some cloud server that gets mined for ways to better market to me.  Unfortunately, the state of US based ISPs made this much more difficult.

After some stumbling around, I discovered that inbound traffic on port 25 (smtp) was blocked at the perimeter by my ISP.  I would need to get more creative.  Later I discovered that they also block all outbound traffic on that port as well.  More creativity needed to solve that problem.

Fortunately as a listener of many fine podcasts on the Jupiter Broadcasting Network (http://jupiterbroadcasting.com) including BSDNow (http://bsdnow.tv), I had heard of a company called Digital Ocean (http://digitalocean.com).  This is a cloud hosting provider where all storage is SSD and you have gigabit networking right up to your VM.  The best part is that you can stand up their low-end server for as little as $5.00 US per month.

The reason I need that server is that I need to be able to have a mail relay that doesn’t have the port restrictions on it to send and receive email on my behalf and forward traffic both ways.  For example, someone sends me an email and the internet deilvers it on port 25 of my relay server.  It then turns around and forwards it to a high-numbered port on my internal server (actually my router that I described in the last blog post).  When I send mail from my server, it has to go out on a high numbered port and get relayed to my DigitalOcean server so that it can turn around and deliver it on port 25 for me.

Conceiving all of this in my mind as a design was one thing.  Actually building it was another.  I decided to use the OpenSMTP implementation that is part of the base operating system in OpenBSD and (of course) run the relay server on OpenBSD.  And there was the first hiccup.  While DigitalOcean recently added support for FreeBSD, they don’t officially support OpenBSD yet.  Fortunately for me, the folks at the BSDNow podcast had already solved this for me.  They pointed me at a nice blog post where someone set up a FreeBSD server on DigitalOcean and used the swapfile to load the miniroot filesystem from OpenBSD to run the install from (http://www.tubsta.com/2015/04/openbsd-on-digital-ocean/).

Cool!  The one trick you have to know is that OpenBSD 5.7 does not work to install this way.  You have to be on a newer release so I’m running 5.8-current on the server and its working just fine.  Once I had that in place, I needed to open the appropriate ports on pf to allow traffic into the right ports but keep everything else locked down.

#       $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
#
# See pf.conf(5) and /etc/examples/pf.conf

set skip on lo

block return    # block stateless traffic
pass            # establish keep-state

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

block in log all

pass in on egress proto tcp to any port 22
pass in on egress proto tcp to any port smtp
pass in on egress proto tcp to any port 2525
pass in on egress proto tcp to any port submission

As you can see, it’s a pretty vanilla configuration but I’m opening port 25 (smtp), port 587 (submission) and port 2525 (my high numbered port that I will receive email from my internal server on to relay to the ultimate destination over port 25 on the public internet).

The next step was to actually stand up OpenSMTP on the relay server.  I first needed to enable it to start at boot by adding:

smtpd_flags=""

to my /etc/rc.conf.local file.  Then I followed the tutorial on this brilliant blog post (http://technoquarter.blogspot.com/p/series.html) to do most of the configuration of my relay server.  I did have to modify the resulting /etc/mail/smtpd.conf for my relay use case as follows:

pki mail.mydomain.com certificate "/etc/ssl/mail.mydomain.com.crt"
pki mail.mydomain.com key "/etc/ssl/private/mail.mydomain.com.key"

listen on lo0
listen on egress tls pki mail.mydomain.com auth-optional
listen on egress port submission tls-require pki mail.mydomain.com auth

listen on egress port 2525 tls pki mail.mydomain.com auth-optional

table relays file:/etc/mail/relays
table aliases db:/etc/mail/aliases.db
table vusers file:/etc/mail/vusers
table vdomains file:/etc/mail/vdomains

accept from source <relays> for any relay as “@mydomain.com”
accept from any for domain <vdomains> relay via “tls://11.11.11.11:2525”

As you can see, the key changes were to add a relays table to identify the specific IP addresses I will relay mail to and from, the listen on port 2525 to ensure that I capture the outbound mail from my internal server, and the accept from lines at the bottom to identify who I am receiving email from and where I am relaying it to.

Now, I needed to configure my internal server.  I modified /etc/rc.conf.local to allow the smtpd daemon to auto-start at boot, and used the same basic configuration of this server.  It’s /etc/mail/smtpd.conf file, however, is the mirror image of the relay server’s:

pki mail.mydomain.com certificate “/etc/ssl/mail.mydomain.com.crt”
pki mail.mydomain.com key “/etc/ssl/private/mail.mydomain.com.key”

listen on lo0
listen on egress port 2525 tls pki mail.mydomain.com auth-optional
listen on egress port submission tls-require pki mail.mydomain.com auth

table aliases db:/etc/mail/aliases.db
table vusers file:/etc/mail/vusers
table vdomains file:/etc/mail/vdomains
table relays file:/etc/mail/relays

accept for local alias <aliases> deliver to mbox

accept from source <relays> for domain <vdomains> virtual <vusers> deliver to mbox
accept from local for any relay via “tls://mail.mydomain.com:2525” as “@mydomain.com”

As you can see, the key difference is that we are only listening on port 2525 (not port 25 and 2525 like we do on the relay server) and we are relaying via TLS to the DigitalOcean VM on port 2525 for outbound communication.

I opened up ports 587 and 2525 in my local /etc/pf.conf file, reloaded pf, started smtpd on both servers and… it didn’t work.  Took me a bit of thinking but I had forgotten to port map those ports from my ISP’s cablemodem/router to my internal box.  Once I did that, eveything worked… sort of.

It turns out that today, you are a spammer to the internet until proven otherwise.  Fortunately, there is a great service called mxtoolbox.com that will scan your setup for you and let you know what needed to be done.

1. I needed to make sure that my relay server’s reverse lookup on its IP address matched the name I was sending mail from.  On DigitalOcean they take the name of your VM in the dashboard and use that.  Easy enough to change.
2. I needed to create an SPF record for the domain.  This is a TXT record in DNS that tells senders which servers will accept email.  I went with a simple one that just said “use any MX record you see in my DNS” to keep it easy.  In the end, the record looks like “v=spf1 mx -all”.
3. My internal server needed to have an MX record in my DNS for the domain.  I could not get the relay server to successfully relay to the internal server without it.  Not sure why, but this did solve the problem for me.  I used a 10 priority for my relay server and a 100 priority for my internal server to ensure that mail would be directed at the relay by senders.
4. I needed to add a DMARC record to tell senders where to send undeliverable reports.  This ends up being another TXT record on the domain that looks like “v=DMARC1; p=none; rua=mailto:postmaster@mydomain.com”.
5. I created an alias of “postmaster” to ensure that these emails make it to my account.

Once I had all of this working, I could successfully send email from Google to my new account on my mail server, it would arrive on port 25 of my relay, get passed along on port 2525 to my internal server, would be accepted and stored in my local account’s mbox.  I could then read the message and reply to it, going out of port 2525 on my internal server to the relay server, then it would be passed along on port 25 to the destination.

I’m still showing up as spam in a lot of mail services so I clearly have more work to do, but I get a clean bill of health on mboxtools.com (including a detailed search for my relay server’s IP address in all of the blacklists, etc.)  I’ll continue to research this and edit this post in the future when I figure it out.  However, what I have now is working well enough (and was hard enough to figure out) that I wanted to share what I had learned.

Now that I have my laptop working well, it’s time to address one of the main reasons I wanted to explore OpenBSD – securing my home network.  Many people might question the need for this.  After all, isn’t that the job of your Internet Service Provider?

Case in point – I received a notice from my ISP that they would be sending me a new cablemodem/router that I had to install by a particular date or I would possibly not be able to access the Internet.  After setting it up (turns out they were migrating to IPv6 which was why I needed new hardware), I ran into some problems.  I spent 2+ hours on the phone with their support team that followed a pattern:

• On hold for 20 minutes
• Speak to an agent
• They tell me to reboot the modem (which I’ve done)
• I tell them the problem is deeper than that
• They tell me they will escalate this to a level 2 technician, please hold
• They hang up on me

After a couple of iterations of this, I got frustrated.  I did some Googling and found the default admin userid and password for the modem.  Surely that wouldn’t work…  Yep it did.  So on the one hand, I was able to fix my problem (yay) on the other, what an appalling security setup.  I needed to have my own firewall that fronted to the ISPs router and treat their network as hostile.  As I’m fond of saying, the only thing more dangerous than running an insecure network is thinking you are running a secure network.

Fortunately, our friends at BSDNow.TV had a great tutorial for solving this problem (http://www.bsdnow.tv/tutorials/openbsd-router).  However, there was the question of what hardware to use.  I wanted something that would be quiet and not consume a lot of power (i.e. not generate a lot of heat) and yet be powerful enough to serve multiple purposes on my network.  I also wanted hardware (especially network hardware) that was mainstream for OpenBSD.  Oh, and it needed two NICs.

I ended up selecting a fanless ASRock motherboard (http://www.newegg.com/Product/Product.aspx?Item=N82E16813157417) that had a dual core 1.86 GHz Atom processor and two Intel gigabit NICs onboard.  I purchased a replacement fan (http://www.newegg.com/Product/Product.aspx?Item=N82E16835608055) for the power supply in the case (http://www.newegg.com/Product/Product.aspx?Item=N82E16811108196) I ordered to keep the noise levels down.  Throw in a couple of sticks of RAM and a small SSD and I was good to go.

When the hardware arrived, I quickly assembled it, booted from the OpenBSD 5.7 install image (I figured I wouldn’t run current on this box because it needed to be more of an appliance) from a thumb drive (install57.fs) and configured my base system.  After that, I followed the BSDNow tutorial and soon had a router up and running.

I’ve been running the router for some time now and the performance is great.  The only issue I have had to date is with dnscrypt and the server I chose to use in the Netherlands.  There was a recent Amazon outage and that server went dark on me which caused me to chase my tail a bit to find out why I couldn’t resolve names any more on the network.  After that, it seemed to have an issue where, after 7 days of uptime, it would randomly stop resolving names.  I switched to a server in Sweden and it looks like that problem is resolved.  I could run my own recursive server internally but I liked the fact that my DNS requests were totally unknown to my ISP and being serviced outside of the United States.

On my “todo” list going forward, I would like to:

• Set up an internal caching proxy server to improve my local network’s performance to frequently visited sites
• Set up my own internal mail server that stores all of my mail data locally on an encrypted volume
• Set up network monitoring (likely nagios given my experience with it) using the new httpd daemon in OpenBSD (no more Apache for me)
• Set up snort as an Intrusion Detection System (IDS) on the internal network

In the “just for convenience” category, I set up an ssh server, poked a wall through the pf firewall and port forwarded the port from the ISP’s modem to my box.  I set it up as securely as I could think to (no password logins allowed, root cannot login) and limited it to using certificate based authentication with only one user on the box having access.  For fun (I know, I have strange hobbies) I created a cron job to look at who was trying to remotely log into the open port and was astonished at the frequency of attacks.

For those interested, here is the quick & dirty script I cooked up to generate a daily report from the router that I send to my gmail address each morning (triggered by cron) so that I can keep an eye on what’s going on:

#! /bin/sh
echo 'Uptime:' > /tmp/network-report
uptime >> /tmp/network-report
echo ' ' >> /tmp/network-report
echo 'Hardware sensors' >> /tmp/network-report
sysctl hw.sensors >> /tmp/network-report
echo ' ' >> /tmp/network-report
echo 'Egress (em0) network statistics' >> /tmp/network-report
vnstat -i em0 >> /tmp/network-report
echo ' ' >> /tmp/network-report
echo 'Interal (em1) network statistics' >> /tmp/network-report
vnstat -i em1 >> /tmp/network-report
echo ' ' >> /tmp/network-report
echo 'List of firewall rules' >> /tmp/network-report
pfctl -g -s rules| grep '^@' >> /tmp/network-report
echo ' ' >> /tmp/network-report
echo 'Tcpdump of egress (em0) network' >> /tmp/network-report
tcpdump -n -e -ttt -r /var/log/pflog inbound and host 11.11.11.11 >> /tmp/network-report
echo ' ' >> /tmp/network-report

echo ‘ ‘ >> /tmp/network-report
echo ‘Failed ssh login attempts’ >> /tmp/network-report
cat /var/log/authlog | grep ‘sshd’ | grep ‘Invalid’ >> /tmp/network-report
cat /var/log/authlog | grep ‘sshd’ | grep ‘Invalid’ | grep -o ‘[0-9]\{1,3\}\.[0-
9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}’ | sort | uniq -u | while read in; do /usr
/local/bin/country.sh “$in”; done >> /tmp/network-report

cat /tmp/network-report | mail -s ‘Daily router activity’ me@mydomain.com

The “country.sh” was a little quick & dirty script I cooked up to further “gild the lilly”.  I was curious as to what the country of origin was for the attempted intrusions.  Again, I have odd hobbies.  Here’s that script too if you are interested:


! /bin/sh

echo >/tmp/lookup “curl -silent /dev/null https://restcountries.eu/rest/v1/alpha/”
curl -silent /dev/null ipinfo.io/$1 | grep country | sed s/’  “country”:’//g |sed s/'”‘//g | sed s/,//g | awk ‘{print tolower($0)}’ >> /tmp/lookup

sed ‘N;s/\n//’ /tmp/lookup > /tmp/lookup.sh
chmod +x /tmp/lookup.sh

echo $1 ‘ -> ‘ >/tmp/result.out
exec /tmp/lookup.sh | perl -pe ‘s/,/\n/g’ | grep name | sed ‘s/{“name”:”//g’| sed ‘s/”//g’ >>/tmp/result.out

cat /tmp/result.out | sed ‘N;s/\n//’

To continue the security / privacy theme, I discovered some really nice tools on a site called http://privacytools.io and incorporated them into my setup.

I’m now running Firefox full-time and am using LastPass as a password manager so that I can have a unique, strong password on all sites that are password protected.  In addition, I’m using:

• HTTPS Everywhere (EFF) – Forces sites to use HTTPS everywhere it can so that more of your traffic is encrypted by default
• Random Agent Spoofer – Allows you to randomize your user-agent and also block other things that the browser passes back
• Self-destructing cookies – Deletes a site’s cookies when you close the tab or close the browser
• uBlock Origin – A very impressive ad and tracker blocking tool

In addition, at the recommendation of PrivacyToolsIO, I purchased a VPN subscription from a non US-based provider who doesn’t retain logs (AirVPN).  It certainly does slow down my network access but the fact that the OpenVPN client encrypts data before it ever leaves my laptop is very nice.

Finally, for backups, I found an excellent provider called TarSnap.  The backup client is open source and it encrypts your data with a key that you generate and control before it leaves your machine.  The encrypted, de-duplicated data is stored in the cloud at a very inexpensive price.  My estimate is that my encrypted backups will cost in the neighborhood of $10 per year.