The Most Metal Thing I’ve Done Today

As a middle-aged electric bass player, the “metal moments” of my life have been coming with less frequency than they did when I was younger. As a result, I tend to look for opportunities to be “metal” on any given day. To that end, I want to explore Canonical’s Metal as a Service or MaaS. Yeah, I know, I went for the cheap pun!

For those of you who aren’t familiar with this awesome piece of software, it essentially allows you to take a collection of physical servers on a private network and turn them into a cluster that allows you to pass out physical or virtual servers to users and then reclaim them when you are done. It does all of this using standard protocols that make life very, very easy. For example, the MaaS servers boot off of DHCP/PXE from an image hosted on the controller so that the OS image doesn’t live on the physical disk of the machine, freeing its built-in storage up for use by the cluster. Additionally, the software supports things like the Intel Active Management Technology (AMT) and its ability to allow remote power on / power off of machines that have this capability (along with many other more enterprise-y technologies for similar control).

For the purpose of this post, I’m going to create a MaaS cluster out of six machines that I have dedicated to the purpose and will be using them to host various projects in my home lab. As long-time readers of this blog know, I am a fan of the Lenovo Thinkpad family of laptops so as a result (like many in my cult) I have quite a stack of them lying around at any given time. For the purpose of this, I will be harnessing the power of my W520, W530 and W541 machines – all of which support the AMT (and more importantly I haven’t CoreBoot-ed yet so it still is enabled).

In addition, I have what I call my “Beast”, a tower machine with a Threadripper CPU that has 32 virtual cores, my NAS box (another AMD cpu machine that has a bunch of spinning physical disks) and finally the machine I’m using for my controller. For that purpose, I dragged out an old Dell laptop I had lying around. It only has one NIC (a WiFi card that I used to attach to my home network) but I picked up a USB-3 gigabit Ethernet adapter that is well supported by Linux to use to run the private network.

The controller machine connects to my home network (10.0.0.0/24) as well as to a small 8-port managed Gigabit switch that all five of the worker nodes will be solely attached to (192.168.100.0/24). That’s the physical network layout. Pretty simple. I also took the time to put a proper AMT password on the machines that support this technology which the MaaS controller will use to reboot them as needed. For the two AMD machines, I have to physically press the power button – at some point I might get an IP enabled power strip that is supported by MaaS and use it to allow them to be “remote controlled” as well but this works just fine for the time being. You might also want to check that virtualization is turned on in the BIOS for any of the machines you are using.

I’m using Ubuntu 22.04 Server for the controller machine and am running it pretty much vanilla except for some network configuration to allow it to serve as a packet router from the private network to my home network so that machines in the cluster can download packages as needed. I could work around that by hosting a mirror on my controller with the packages I needed (I think) but this was easier. For most of this post, I’m basing my configuration on the MaaS 3.2 documentation.

I downloaded the latest 22.04 server from the Ubuntu website and then used the “Startup Disk Creator” application that ships as part of the base OS on my laptop to create a bootable USB drive. After booting from the USB drive on the Dell laptop, the only configuration change I made to the default install was to enable an SSH server on the machine so I can remote in and do everything I need to from my laptop (except for pressing the power buttons a few times on the worker nodes).

Once the controller is installed and booted up, I have to make some network configuration changes to allow it to have a static IP address on both the home network side (WiFi) as well as on the private network that it will be managing. To do this, I edit the /etc/netplan/00-installer-config.yaml file to look like the following:

network:
  ethernets:
    enx000ec6306fb8:
      dhcp4: false
      optional: true
      addresses: [192.168.100.1/24]
    wifis:
      wlp1s0:
        dhcp4: false
        optional: true
        addresses: [10.0.0.5/24]
        nameservers: 
          addresses: [8.8.8.8]
        routes:
          - to: default
            via: 10.0.0.1
        access-points:
          "my_ssid_name":
            password: "********"
  version: 2

After saving these changes, I ran “sudo netplan try” to test the configuration and ensure that everything is working the way I wanted it to. Once I was satisfied with the network, I updated the machine (“sudo apt update” and then “sudo apt upgrade”). After that, I reboot the machine to pick up the new kernel I downloaded in the updates.

I want my machines on the private network to be able to reach the Internet through the MaaS controller. To make things simple, I’m just going to set up a basic router on this machine using a guide I found here:

# echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
# sysctl net.ipv4.ip_forward=1
# iptables -A FORWARD -i enx000ec6306fb8 -o wlp1s0 -j ACCEPT
# iptables -A FORWARD -i wlp1s0 -o enx000ec6306fb8 -m state --state RELATED,ESTABLISHED -j ACCEPT
# iptables -t nat -A POSTROUTING -o wlp1s0 -j MASQUERADE
# iptables -t nat -A POSTROUTING -o enx000ec6306fb8 -j MASQUERADE
# apt install iptables-persistent

After running the “apt install…” command, make sure you tell it to persist the IPV4 and IPV6 rules and they will be stored in /etc/iptables under files called “rules.v4” and “rules.v6”. At this point, because I’m old-school, I do a reboot.

For my lab, I want to be as close to a “production” environment as I can get. Therefore, I’m opting for a “region+rack” configuration. Using snaps, installing MaaS is… well… a snap:

$ sudo snap install --channel=3.2 maas

The next thing we need to do is set up a PostgreSQL database for this instance of MaaS:

$ sudo snap install maas-test-db

At this point, it is time to initialize your instance of MaaS:

$ sudo maas init region+rack --database-uri maas-test-db:///

I took the default for my MaaS URL (http://10.0.0.5:5240/MAAS). I then ran the command “$ sudo maas createadmin” and provided my admin credentials and my Launchpad user for my ssh keys.

At this point, I logged into my MaaS instance from that URL and did some configuration. First, I named my instance and set the DNS forwarder to one that I liked. Next, we need to enable DHCP for the private network so that it can PXE boot new machines on the network. To do this, navigate to the Subnets tab and click on the hyperlink in the “vlan” column that corresponds to the private network. Click “Configure DHCP” and then fill in the Subnet dropdown to correspond to the IP address range of your private network then save the change. You should now notice the warning about DHCP not being configured has gone away from the MaaS user interface.

The next thing we need to do is set up the default gateway that is shared by the MaaS DHCP server to the machines. To do this, navigate to the “Subnets” tab and click on the hyperlink in the “subnet” column for your private network. Click “Edit” and fill in the Default Gateway IP address and the DNS address if you’d like. After clicking “Save” your machines will be automatically configured to use the default gateway you provided (in my case, the private network IP address of my MaaS controller).

I first boot up the Thinkpads (that have Intel AMT) on the private network and they PXE boots off of the MaaS controller and eventually show up under the “Machines” tab of the MaaS user interface. I click on each of them in the MaaS user interface and configure their names and their power setup to be Intel AMT and provide my passwords and IP addresses that I set up in the firmware on each of them. I then booted up the AMD machines and in their configuration, just set their power type to “Manual.

At this point, you will need to get the machines into a “usable” state for MaaS so to do that, check the box next to each one on the “Machines” tab and select “Commission” from the actions menu. You’ll have to physically power on any machines that don’t have Intel’s AMT and then they will go through the commissioning process. When done, they will show up as “Ready” on the “Machines” tab.

Now I need to get the default gateway working for each of the machines. There might be an easier way of doing this; however, I haven’t figured it out yet so I’m following part of a guide found here. For each machine, click on it and then navigate to the network tab. When there, check the box next to the network interface that is connected to the private network’s switch and press the “Create Bridge” button. Name the bridge “br-ex”, the type is “Open vSwitch”, select the fabric and subnet corresponding to your private network and pick “auto assign” for the ip mode.

Now, check the boxes next to your “Ready” machines and select “Deploy” from the actions menu. Be sure to check the “Auto Assign as KVM host” to make them available to host virtual machines for you. Press the “Start deployment…” button and be sure to power on any that don’t have Intel AMT technology to control their power state. At this point you should be done with the power button pushing business unless you need to do maintenance on the machines.

This seemed as good a time as any to create a MaaS user for myself. To do this, I navigated to the “Settings” tab and selected “Users” and then clicked “Add User”. I filled in the details (by the way, MaaS enforces no duplicate email addresses among its users so if you are like me and want an admin account and a separate user account, you’ll have to use two email addresses) and clicked “Save” and I was good to go. I logged in as that user and supplied my SSH key from Launchpad.

If you now switch to the main MaaS “KVM” tab, you should see your machines available and be able to add virtual machines. You do this by clicking on one of the hosts and then clicking the “Add Virtual Machine” button. It then shows up as a “New” machine in the MaaS user interface.

I then log in as my “user” account in MaaS and deploy the “New” virtual machines. Once they are completely deployed, you can then ssh into them from a machine that has connectivity to the private network. The only trick I discovered is that you have to log in as the “ubuntu” user, NOT the user you have set up in MaaS.

At this point, I have a working MaaS home lab that I can use for a variety of projects. I hope that you found this post helpful!