Code of Connor

go-nbd release 0.3.0

Wed, 04 Mar 2026 17:53:13 -0800

I’m happy to announce that a new release of the github.com/digitalocean/go-nbd module, version 0.3.0, is available here:

https://github.com/digitalocean/go-nbd/releases/tag/v0.3.0

Getting Started With Kubevirt

Mon, 25 Aug 2025 18:55:40 -0700

When I first got into homelab as a hobby, I had a single server. Of course, that’s how it usually starts for those of us in the homelab community; but for most of us, it doesn’t end there. I know this, because I am most of us.

Instead of a single server, my homelab now consists of 1-3 Minisforum UM480XT mini PCs. I say “1-3” because I switch things up a lot depending on my mood and what I feel like maintaining.

My interests eventually led me to learning about containerization and orchestration, the latter being an essential concept in providing highly available infrastructure for applications in spite of hardware failure.

I opted to use my homelab to get hands on experience with the orchestration platform I wanted to learn, Kubernetes. Maybe you’ve already heard of it.

Almost everything I currently run in my homelab is well-suited to containerization and is a good fit for deploying with Kubernetes. There are a couple of exceptions to this.

My managed switches are UniFi switches, so I run the UniFi controller application on the network management VLAN. I use Glenn R’s UniFi installation scripts for this, since the UniFi controller application is all packaged up for Ubuntu.
I occasionally spin up VMs for the traditional VPS experience, for learning in a sandbox, experimenting with operating systems, or for running stateful components like databases (or anything that comes packaged with a database.)

Those two exceptions don’t make up the bulk of my software deployments, so I’d prefer to optimize for the common case which is deploying stateless applications on Kubernetes. To that end, I’ve chosen to deploy Kubernetes on one of my mini PCs, and I figured I’d bring the virtual machines along for the ride in case I settle on Kubernetes for the longer term in my homelab.

Up until now, I’ve relied on the traditional Linux KVM virtualization stack, which is KVM, QEMU, libvirt, and Virt Tools.

Luckily, I don’t have to leave it behind! KubeVirt bridges the traditional Linux KVM stack to an orchestration-based paradigm using Kubernetes.

Requirements

Virtual machines need to be on their designated VLANs (especially the UniFi controller, as mentioned above.)
There shouldn’t be a network stack inbetween the virtual machine and its storage devices. I don’t want the CPU, memory, or network overhead of highly-available CSI implementations and if any migrations need to happen, offline migrations are perfectly acceptable (KubeVirt only supports live migration if the PersistentVolume’s access mode is ReadWriteMany, which local NVMe storage won’t be.)

Host networking

To start, I did a fresh installation of Ubuntu 24.04 with LVM on one of my idle mini PCs.

For the VLAN requirement, let’s start with setting up networking on the host. I always do this step first because I don’t have a remote management interface for these mini PCs and messing up the networking over SSH means I have to get up and plug the computer into a monitor and keyboard anyway.

# /etc/netplan/net.yaml
network:
  version: 2
  ethernets:
    eno1: {}
  vlans:
    eno1.10:
      id: 10
      link: eno1
  bridges:
    br1:
      interfaces:
        - eno1
      parameters:
        stp: false
        forward-delay: 0
    br10:
      interfaces:
        - eno1.10
      parameters:
        stp: false
        forward-delay: 0
      addresses:
        - 192.168.88.147/24
      routes:
        - to: default
          via: 192.168.88.1
      nameservers:
        addresses:
          - 192.168.88.1

This is fairly straightforward. “vlan1” is my network management VLAN. “vlan10” is my regular LAN. br1 is only there for the UniFi controller VM to be on the network management VLAN, which is the untagged, default VLAN.

At this point, I was able to do the rest of the setup from the comfort of my couch over SSH.

Host storage

Now for the storage requirement. LVM doesn’t consume the entire VG for the initial installation of Ubuntu, so it offers a lot of flexibility for what I want to do with the rest of the space even though I just have the single NVMe drive installed.

$ sudo lvcreate --type thin-pool -L 250G ubuntu-vg -n workload

It’s not necessary to create an LV for each base image, especially if you use the KubeVirt Containerized Data Importer, but I’m not currently using it.

$ sudo lvcreate -V 5G -T ubuntu-vg/workload -n ubuntu-24.04_amd64
$ curl -LkO https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
$ sudo qemu-img convert -O raw noble-server-cloudimg-amd64.img /dev/ubuntu-vg/ubuntu-24.04_amd64

The qemu-img step is just because Ubuntu offers the image as a QCOW2 image, but the LV is a raw block device and so QCOW2 won’t work here.

Kubernetes

At this point, I’m ready to install Kubernetes. I’ll be installing K3s, which is a lightweight Kubernetes distribution. I might move onto something like kubeadm or something more manual in the future, but I was excited to get started on this project so K3s it is!

I won’t be overly opinionated here, the only exception is disabling the local path provisioner and passing an argument to the Kubelet so that it’s possible to deploy privileged pods (which is necessary for the KubeVirt pods to have access to /dev/kvm for hardware accelerated virtualization.)

# /etc/rancher/k3s/config.yaml
---
write-kubeconfig-mode: "0644"
tls-san:
  - "metalk8s.local.connorkuehl.net"
disable:
  - local-storage
kubelet-arg:
  - "allow-privileged=true"
node-ip: "192.168.88.147"
bind-address: "192.168.88.147"
cluster-init: true

Note that the mode I chose for the write-kubeconfig-mode is overly permissive and is not acceptable for production.

I chose to use Multus to connect the virtual machine pods to their respective bridge device on the host.

Something I really like about K3s is that if you lay down Kubernetes manifests under a well-known directory, a K3s controller will apply them. Even Helm charts:

# /var/lib/rancher/k3s/server/manifests/multus-cni.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: multus
  namespace: kube-system
spec:
  repo: https://rke2-charts.rancher.io
  chart: rke2-multus
  targetNamespace: kube-system
  valuesContent: |-
    config:
      fullnameOverride: multus
      cni_conf:
        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
        binDir: /var/lib/rancher/k3s/data/cni/
        kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
        multusAutoconfigDir: /var/lib/rancher/k3s/agent/etc/cni/net.d

With Multus installed, let’s teach it about the host bridges:

# /var/lib/rancher/k3s/server/manifests/vlan1.yaml
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: "vlan1"
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "bridge",
    "bridge": "br1",
    "ipam": {}
  }'

# /var/lib/rancher/k3s/server/manifests/vlan10.yaml
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: "vlan10"
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "bridge",
    "bridge": "br10",
    "ipam": {}
  }'

Note 1: on a multiple node setup, this will not work unless the host networking is configured identically. For example, if br10 bridges to VLAN 10 on node A, but VLAN 11 (or doesn’t exist) on node B, that would be a problem.

Note 2: the IPAM block is disabled because I will only use these NetworkAttachmentDefinitions with KubeVirt VMs whose guest operating systems will DHCP to get an IP address from my router.

Alright, what about the LVM thin pool I set up?

I chose to use local-static-provisioner to expose the LVs that I precreate on the node automatically. All I have to do is create the LV and then drop a symlink to it in a discovery directory and the controller will automatically create the PersistentVolume object in the kube-apiserver for me. The alternative would be writing a bunch of YAML like this after setting up the LV:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-pv-9c25bcdb
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  local:
    path: /mnt/disks/nvme/hellokubevirt
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-nvme
  volumeMode: Block

I’d rather not do so much typing.

Let’s set up the discovery directory:

$ sudo mkdir -p /mnt/disks/nvme

I’ll install local-static-provisioner with Helm, so here are the values for the chart:

# local-static-provisioner.values.yaml
classes:
  - name: local-nvme
    hostDir: /mnt/disks/nvme
    volumeMode: Block
    storageClass:
      reclaimPolicy: Retain

$ helm template \
	--debug local-static-provisioner/local-static-provisioner \
	--version v2.8.0 \
	--namespace local-static-provisioner \
	-f local-static-provisioner.values.yaml | sudo tee /var/lib/rancher/k3s/server/manifests/local-static-provisioner.generated.yaml

For some reason, I couldn’t get this working as a HelmChart Addon like I did with Multus above.

Lastly, I followed the KubeVirt installation guide.

Creating a test VM

First, I’ll create a copy of the base image:

$ sudo lvcreate –V 10G -T ubuntu-vg/workload -n hellokubevirt
$ sudo qemu-img convert -O raw /dev/ubuntu-vg/ubuntu-24.04_amd64 /dev/ubuntu-vg/hellokubevirt

And expose it to the local-static-provisioner

$ sudo ln -s /dev/ubuntu-vg/hellokubevirt /mnt/disks/nvme/

And finally… a VM.

I’ll concede it’s a bit of a shock in its final form all at once, but I’m too lazy to type up the progression that resulted from running kubectl explain ... for just about everything.

# hellokubevirt.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hellokubevirt
  namespace: default
spec:
  volumeName: local-pv-9c25bcdb
  volumeMode: Block
  accessModes:
    - ReadWriteOnce
  storageClassName: local-nvme
  resources:
    requests:
      storage: 10G
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: hellokubevirt
  namespace: default
spec:
  runStrategy: Always
  template:
    spec:
      domain:
        resources:
          requests:
            memory: 1024M
        cpu:
          cores: 1
        devices:
          disks:
            - name: disk0
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
            - name: net0
              bridge: {}
      networks:
        - name: net0
          multus:
            networkName: default/vlan10
      volumes:
        - name: disk0
          persistentVolumeClaim:
            claimName: hellokubevirt
        - name: cloudinitdisk
          cloudInitNoCloud:
            userData: |-
              #cloud-config

              hostname: hellokubevirt
              users:
              - name: root
                ssh_import_id: ['gh:connorkuehl']

$ kubectl create -f hellokubevirt.yaml
$ virtctl console hellokubevirt
# ...
# Output omitted for brevity.
[  OK  ] Reached target cloud-init.target - Cloud-init target.

Ubuntu 24.04.3 LTS hellokubevirt ttyS0

hellokubevirt login:

One last wrinkle

If I stop then start the VM, networking breaks. Astute readers might have noticed I did not specify a MAC address in the VirtualMachine spec. Each time I stop then start the VM, KubeVirt generates a new VMI from the VirtualMachine spec, including a MAC address if one was omitted. This makes netplan very upset in the guest OS.

To fix, I just include a MAC address (or use the one that was generated first):

$ kubectl get vmi hellokubevirt -o=jsonpath='{.spec.domain.devices.interfaces[0].macAddress}'
b6:12:29:88:50:59
$ kubectl edit vm hellokubevirt

         interfaces:
         - bridge: {}
           macAddress: b6:12:29:88:50:59
           name: net0

That’s sufficient for me.

I did find an alternative workaround on Xe Iaso’s blog post, but I chose not to use it because I need a stable MAC address anyway as I prefer to configure static DHCP leases at my router.

Conclusion

Overall, I’m quite pleased with how easy it was to get started with KubeVirt.

Will I stick with it? Only time will tell. I do miss virt-install and I’m a big fan of how simple plain old Linux with libvirt is.

Effortless Static Sites With Terraform and Caddy

Sat, 22 Oct 2022 19:29:32 -0500

My blog has moved around several times since its inception. I first started it in college using Automattic’s hosted wordpress.com services, then I eventually moved it to a $5 VPS where I self-hosted my own copy of WordPress. After some time of that, its final form has taken the shape of a Hugo-generated static site on a $5 Linux VPS.

I’ll concede that it doesn’t take a substantial amount of toil to deploy a server and configure it to run a basic WordPress blog or static HTML pages for that matter. I’ve developed some sentimental feelings toward this blog though, and taking the proper steps to ensure that it is properly backed up or easily restored in case of tragedy adds a bit more complexity.

There are a couple of well-understood solutions that constitute a valid backup/restore/recovery plan.

The first would be to simply implement a traditional backup solution. Many cloud providers, including the one I use for my VPS, have disk-based snapshot and backup solutions. I could also integrate a file-based backup solution from inside the VPS where the site could be backed up to other locations.

Both of these scenarios would likely require me to spend more money on my blog, which I am not strictly averse to because of its importance to me. The file-based backup strategy would also require me to implement a restore strategy.

Obviously, I’d need to come up with automation for either strategy, because it’d be foolish not to. And well, that’s just more work.

I decided to go a different route. I chose to apply some of the immutable infrastructure as code principles that I’ve been learning to implement a turnkey solution that is entirely reproducible from source, no matter what. It also doesn’t require paying for my cloud provider to host snapshots or blob storage buckets.

At the heart of it, is a Terraform module that I’ve written. Terraform allows you to declare the infrastructure that you want. It will read this declaration and examine the state of your infrastructure, and it will carry out whatever actions are necessary to ensure that your infrastructure has taken the shape that you want.

In this case, I started off by declaring that I want a single DigitalOcean droplet¹.

resource "digitalocean_droplet" "www" {
  image  = "ubuntu-20-04-x64"
  name   = "www"
  region = "sfo3"
  size   = "s-1vcpu-1gb"

  ssh_keys = var.ssh_keys
}

Now, this hasn’t really saved me much work at all. If anything, it saves me the effort of clicking around on the cloud console to make a droplet and/or typing the droplet create commands at the command line.

I still have to SSH in, configure a web server, and rsync the static site over to the served directory.

Luckily, I can also specify a cloud-init script that the droplet will run after it is created and powered on.

For example, I can now automate the installation of a web server, all from the comfort of Terraform²:

resource "digitalocean_droplet" "www" {
  image  = "ubuntu-20-04-x64"
  name   = "www"
  region = "sfo3"
  size   = "s-1vcpu-1gb"

  ssh_keys  = var.ssh_keys
  user_data = <<-EOF
#!/usr/bin/env bash
  
mkdir -p /var/www-data/html

export DEBIAN_FRONTEND=noninteractive
apt-get install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list

apt-get update            
apt-get install -y caddy  
systemctl enable --now caddy
cat > /etc/caddy/Caddyfile <<EOM
codeofconnor.com {
    root * /var/www-data/html
    file_server
}
EOM

chown -R caddy:caddy /var/www-data/html
systemctl reload caddy
  EOF
}

Ah, that’s much nicer. Now all that’s required is rsync’ing the files over for Caddy to serve, which is no big deal, because I already have a simple Makefile target for that in the git repo that has all of the source markdown for my blog.

Egads! This won’t work as is, because when I rsync my files over, the owner and group is set to my user and group, and so Caddy can’t actually display them because it does not have permission to.

I can just make a quick tweak to the cloud-init script so that the caddy group is automatically applied to files added to the html directory and caddy will be able to read them:

chown -R caddy:caddy /var/www-data/html
+chmod g+s /var/www-data/html

Alas, now Terraform will have to re-create this droplet so it can create one that will have the updated cloud-init script run as part of the provisioning process. This means that the version of the site I uploaded earlier will be lost, and I’ll have to upload it again.

It’s not the end of the world, but still requires me to remember to run the deploy target from my site’s Makefile again. Not a great experience.

In most cases, this is the beauty of immutable infrastructure as code. There is almost always no doubt at all what the state of the system is in because it’s written here as code and Terraform will alert you to any configuration drift and show what’s needed to correct it.

This is also easily fixed while still practicing immutable IaC techniques.

In this case, we just have to isolate the mutable part of the infrastructure. We can just attach a block volume and populate it with the contents of the site.

resource "digitalocean_volume" "www_data" {
  region                  = "sfo3"
  name                    = "www-data"
  size                    = 5
  initial_filesystem_type = "ext4"
  description             = "Content for the webserver to serve"
}

resource "digitalocean_volume_attachment" "attach_www_data_to_www" {
  droplet_id = digitalocean_droplet.www.id
  volume_id  = digitalocean_volume.www_data.id
}

And before we forget, we want to make sure our cloud-init data mounts it to the right place so Caddy can serve files from it:

#!/usr/bin/env bash
    
export DEBIAN_FRONTEND=noninteractive
      
mkdir -p /var/www-data
+mount -o discard,defaults,noatime /dev/disk/by-id/scsi-0DO_Volume_${volume_name} /var/www-data
+echo '/dev/disk/by-id/scsi-0DO_Volume_${volume_name} /var/www-data ext4 defaults,nofail,discard 0 0' >> /etc/fstab
+
+mkdir -p /var/www-data/html
  
apt-get install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
  
apt-get update            
apt-get install -y caddy  
systemctl enable --now caddy
  
cat > /etc/caddy/Caddyfile <<EOF
${domain_name} {
    root * /var/www-data/html
    file_server
} 
EOF
  
chown -R caddy:caddy /var/www-data/html
chmod g+s /var/www-data/html
  
systemctl restart caddy

Astute readers may have noticed I moved the cloud-init script out of the heredoc in the Terraform module. For completeness, the Terraform module is updated to look like this now:

  image  = "ubuntu-20-04-x64"
  name   = "www"
  region = "sfo3"
  size   = "s-1vcpu-1gb"

  ssh_keys  = var.ssh_keys
  user_data = templatefile("${path.module}/user_data.sh", {
    domain_name = var.domain_name
    volume_name = digitalocean_volume.www_data.name
  })

Nice! Now I just have to run terraform apply and then deploy the site one last time since the old droplet with the site on it was blown away.

This time, the deploy will deposit the files into the block volume since I mounted it at the same place that the deploy script is used to copying files to.

Now I can happily tear down and recreate the droplet, and I won’t have to run the deploy to refurnish it with the site data every time.

The finishing touch, of course, is DNS. I can easily point codeofconnor.com at the correct droplet, no matter how many times I destroy and recreate it, like so:

resource "digitalocean_domain" "a_record" {
  name       = var.domain_name
  ip_address = digitalocean_droplet.www.ipv4_address
}

Now, this basic configuration does result in downtime if the base droplet is destroyed. At least, until DNS propagates to point at the new droplet.

If the day ever comes that I want a zero-downtime redeploy of the base droplet, I’ll be sure to write about it here!

Now anything can happen to my infrastructure, and I can be back up and running with exactly these steps:

Make sure I have a valid DigitalOcean API token
terraform apply
make deploy
Wait for DNS to propagate

Disclaimer: as of this writing, I am a DigitalOcean employee. ↩︎
It’s unlikely that every code snippet here can be copy-and-pasted into your own Terraform configuration and “just work.” I ended up copying and pasting snippets from the final product, and simplifying them down to their earlier renditions because I did not have a git history with many intermediate steps. The final form has more details in it that I feel would distract from the main goals of this post. ↩︎

Applying Cloud Native Patterns to My Discord Bot

Sun, 31 Jul 2022 12:05:46 -0500

I recently read through Cloud Native Patterns by Cornelia Davis. I thought it was a great book. This post is not a review of Cloud Native Patterns, but instead it’s a journal of how I’ve applied some of what I’ve learned.

There have been many times that I’ve declared Popple to be a completed project, but it’s continued to prove itself a valuable playground for experimenting and learning new-to-me ideas in both software development and ops. So, naturally, I chose to hack on Popple to get some hands-on experience with some of the concepts described in the book.

At the risk of oversimplifying things, I’ve found these two guiding questions to be a good rule of thumb for what needs to change:

What prevents me from running this as multiple processes?
What happens if $dependency is malfunctioning?

The first question has to do with identifying which parts of the application would prevent horizontal scaling to eliminate single points of failure. The second question is a barometer for deciding what kinds of reliability patterns I should bake in and where to meet any kinds of desired consistency or availability goals.

DISCLAIMER: I realize this rearchitecting was completely overkill for workloads that my current Discord bot deployment handles. This was just for academic purposes.

I had no issues with the reliability or performance of a single process with a SQLite database.

Previous Architecture

A quick look at this might uncover some answers to the guiding questions above.

The following aspects of this deployment prevents horizontal scaling:

This deployment assumes a SQLite database on the local filesystem. Deploying an additional instance to a separate node would need access to that database somehow. Pulling it out to a remote filesystem is possible I suppose, but there may be concurrency issues and this would require additional provisioning on nodes where it’s deployed.
It’s not obvious from looking at the diagram, but the process identifies itself to Discord using a special token. The Discord API does allow for bots to introduce themselves as a shard of that token, but my bot does not currently do that. So I’d only want one of these deployed until the bot supports sharding.
Another thing that needs to change is how Popple is configured. Previously, it would read a config file from the filesystem. Rather than adding those to nodes that are provisioned, I’ll change it to be configured purely through environment variables. There will be a couple of secrets that need to be configured, but it’s not going to be in scope for this blog post since I only deploy Popple to my homelab. I’ll address this at a later date.

Because of this, my Discord bot is a stateful service, which inherently makes horizontal scaling much harder. The first step is going to be isolating the stateful parts of the application.

Isolating state

Let’s deal with the SQLite issue first. There are a couple of things that immediately come to mind:

Deploy a relational database (E.g., MariaDB, Postgres, MySQL, etc).

It would look like this:

Alternatively, put a CRUD service in front of the SQLite database. Like this:

I’d rather do the first one, since it simplifies development because I won’t have to maintain the CRUD API that stands in front of the SQLite database.

The second issue is Discord sharding.

It looks like the Discord library I am using has a way to shard, but it seems to require a shard ID and the total number of shards a priori. Off the top of my head, I’m not sure how I could horizontally scale the processes up and down without having to coordinate across all of them to make sure their shard ID and max shard counts are correct, so until I have an idea of how to approach that, I will work around it by splitting the actual Discord presence out into its own process, like so:

I drew an arrow between bot and popple because the above diagram implies there will have to be some sort of client-server communication between those two processes.

However, dear reader, I am writing this after the fact, and while I don’t mean to spoil the rest of the post, I can tell you for sure that client-server is not the direction I’m taking this, because it couples the two services.

Keeping things decoupled

Now that I’ve factored out the stateful services (the Discord bot presence and the MySQL database), the business-logic component can now scale horizontally.

A client-server relationship between the bot and the business-logic service would work, but in my opinion, it would add some unnecessary complexity.

First, I’d need to expose an API for the popple service so that the bot can hit those endpoints when something happens.

Second, from a development perspective, I would probably have to scale the bot internally with concurrency for multiple operations from from any number of servers that the bot is overseeing.

Third, there’s another model that fits nicely with how a chat bot operates. I’ve already alluded to this in a previous paragraph:

First, I’d need to expose an API for the popple service so that the bot can hit those endpoints when something happens.

Emphasis mine.

Yep. We’re making this bad bear event-driven.

All the bot has to do is read incoming messages, determine if it’s a control command or if the message has anything that indicates someone or something’s karma is incremented or decremented.

For example, if a control message comes in changing a bot value, the bot could emit an setting-changed event. Or if a message comes in that changes karma, it could emit a karma-changed event.

I’m not going to write a message broker, so I’ll be recruiting some help:

I like this model a lot, and not just because a chat bot lends itself well to an event-driven paradigm. It also allows for the bot and business service components to be developed independently of one another.

For example, someone could contribute to the project without needing a Discord bot token if the area of the codebase they’re concerned with only exists in the service component.

Lastly, something I noticed is that the top-level driver functions for each of these services become very simple. They wait for an event and process it.

If you’re into reading Go code, then you can see for yourself what I’m talking about:

popplebot
- The requestLoop that reads Discord messages and emits “request” events that are handled by popplesvc.
- The eventLoop that processes events that popplesvc emits (such as “karma changed” or “control message processed”)
popplesvc
- The eventLoop that processes request events that popplebot submitted to a work queue and emits events when it processes a control message or a “karma change” request.

Conclusion

I’ll be the first to admit that what I’ve done here is make the deployment much more complicated.

The previous architecture essentially allowed me to copy a binary over and start it. If Discord’s up and I have a path through the Internet to Discord then we’re good to go.

Aside: to be fair, all it takes to deploy to my homelab is to push my git repo there and run docker-compose up -d.

Now I have two binaries to deploy, and I need to make sure that they have a RabbitMQ instance and a MySQL database up for them to function properly.

Despite this, I like the spot that the project is in a lot more than I did previously.

Now,

It’s easier to develop components in isolation. For example, drive-by contributors no longer need their own Discord bot token to run the application since they can just develop and test the popplesvc component.
It’s more suitable for a cloud-native deployment. The popplesvc component could be added to a Kubernetes replica set that can expand and contract freely because it no longer has a file system dependency on the SQLite database.
Either component could now go down briefly and when it comes back up, RabbitMQ will be able to furnish it with any events that it missed.

Monitoring My Discord Bot

Thu, 14 Jul 2022 20:34:49 -0500

I decided to implement a health check capability into my Discord bot. Its uptime is totally unimportant, but it was still fun to do.

In terms of monitoring the Discord bot, there were a few options:

Build my own with cron, bash, and sendmail; install it on a separate host
Use cron to send pings to healthchecks.io
Use either updown.io or uptimerobot.com.

I chose the second option. I found it compelling because the model of pushing a ping from my host to the monitoring API didn’t require me to punch a hole in my host’s firewall so that the monitoring API could scrape the health check endpoint. It’s also free, but it’s worth noting that when I played around with updown.io’s price estimators that I found it to be very inexpensive (on the order of cents per year for my use case).

Instead, I have a cron job that executes a bash script, like this:

#!/usr/bin/env bash

details=$(curl -sSf http://localhost:8000/health)
curl -m 10 --retry 5 --data-raw "$details" https://hc-ping.com/<redacted>/$?

The way it works is the cron job runs on the same host that my Discord bot runs on. It hits the bot’s health check endpoint and stores the output in the details variable. The output is just JSON that describes the health of the bot, like this:

{
  "discord_connection":"connected",
  "discord_heartbeat_latency":"63 ms"
}

healthchecks.io can persist information that you send in your ping, and I wanted it to persist the health details with each check-in because I figure there’d be some forensic value in having that information.

The second curl invocation is the actual healthchecks.io ping. I include the exit status of the first curl invocation because the healthchecks.io API allows you to specify that the monitored service was able to check-in but is experiencing failures.

To take advantage of that, my curl invocation uses the -f flag so that its exit code will be non-zero if the healthcheck endpoint doesn’t return an HTTP 200 code.

The end result of this monitoring goes a little something like this:

My cronjob runs every 5 minutes. I configured the healthchecks.io monitor to expect a ping every 5 minutes.
If healthchecks.io does not receive a ping within that 5 minute interval, I configured it to have a 15 minute grace period. If the grace period elapses and healthchecks.io does not receive a ping, then healthchecks.io sends me an email.
If healthchecks.io receives a ping with the non-zero exit code supplied from my cronjob, then it sends me an email.

Otherwise, no news is good news! Setting all of this up took approximately 5 minutes.

So what happens if the bot goes down and I get an email? Probably nothing, heh. It’ll be cool though. If anything, I’m guessing this will end up inadvertently tracking my ISP’s uptime since my Discord bot runs in my homelab.

I’m not in any way affiliated with healthchecks.io, and I’ve never spoken with them.

Type Safe Enums in Go

Wed, 11 May 2022 17:13:19 -0500

I’m willing to bet that many Go programmers have seen or used this strategy for enumerating things in code before:

type Profession int

const (
	Unknown Profession = iota
	Warrior
	Cleric
	Hunter
	Mage
)

In practice, this is probably fine. In fact, I can’t think of a time this has caused a (known) bug in one of my programs (yet.) Famous last words.

However, there are times where I wish the solution was a little more bulletproof at compile-time.

For example, I dislike having an Unknown variant. This is mainly to avoid ambiguity when client code introduces a zero-value for this enum and perhaps forgets to set it, like so:

var prof Profession

_ = whatIsYourProfession(prof)

This means that most code that cares about the enum type will have to account for the Unknown variant, like so:

func whatIsYourProfession(p Profession) error {
	switch p {
	case Unknown:
		return errors.New("unknown profession")
	case Warrior:
		fmt.Println("Couldn't beat you in an arm-wrestling match.")
	case Cleric:
		fmt.Println("Divine!")
	case Hunter:
		fmt.Println("Shoot an arrow over them mountains.")
	case Mage:
		fmt.Println("I'll pick a card, any card.")
	}
	return nil
}

Even though I accounted for all of the professions, there’s still a bug here.

Technically, client code could do this:

if err := whatIsYourProfession(Profession(-1)); err == nil {
	fmt.Println("fooled ya!")
}

Again, in practice, it’s unlikely that client code would pull the pin on that grenade and hand it to us, but I wish the compiler would stop them from doing so.

So I guess we could add a default case:

func whatIsYourProfession(p Profession) error {
	switch p {
	case Unknown:
		return errors.New("unknown profession")
	case Warrior:
		fmt.Println("Couldn't beat you in an arm-wrestling match.")
	case Cleric:
		fmt.Println("Divine!")
	case Hunter:
		fmt.Println("Shoot an arrow over them mountains.")
	case Mage:
		fmt.Println("I'll pick a card, any card.")
	default:
		return errors.New("how could you betray me like this?")
	}
	return nil
}

We could prevent the zero-value situation by sealing the type within our package so client code can’t zero-value construct it without using specific constructor functions:

package profession

import (
	"errors"
)

var (
	ErrUnknown error = errors.New("unknown profession")
)

type Profession func() profession

func (p Profession) String() string {
	var s string
	switch {
	case Is(p, Warrior):
		s = "warrior"
	case Is(p, Cleric):
		s = "cleric"
	case Is(p, Hunter):
		s = "hunter"
	case Is(p, Mage):
		s = "mage"
	}
	return s
}

type profession int

const (
	warrior profession = iota
	cleric
	hunter
	mage
)

func FromString(s string) (Profession, error) {
	switch s {
	case "warrior":
		return Warrior, nil
	case "cleric":
		return Cleric, nil
	case "hunter":
		return Hunter, nil
	case "mage":
		return Mage, nil
	}
	return nil, ErrUnknown
}

func Warrior() profession {
	return warrior
}

func Cleric() profession {
	return cleric
}

func Hunter() profession {
	return hunter
}

func Mage() profession {
	return mage
}

func Is(r, target Profession) bool {
	return r() == target()
}

Now there’s no such thing as an unknown zero-value profession. Furthermore, because the underlying type is sealed in the package and only accessible via constructor functions, the worst thing client code can do is pass a nil function pointer through if they create a zero-value (nil pointer) profession.Profession:

var prof profession.Profession // nil

Whether or not that offends you, is up to you. Representing an invalid state is a bug, and I’d rather segfault than limp along and hope for the best despite garbage state flowing through my program.

Let’s tie it all together with a complete example program, where we can now have peace of mind that garbage enum values won’t be able to quietly sneak through.

package main

import (
	"bufio"
	"errors"
	"fmt"
	"os"
	"strings"

	"codeofconnor.com/profession/profession"
)

func main() {
	scanner := bufio.NewScanner(os.Stdin)
	for scanner.Scan() {
		line := strings.TrimSpace(scanner.Text())

		prof, err := profession.FromString(line)
		if errors.Is(err, profession.ErrUnknown) {
			fmt.Println("I've never heard of a", line)
			continue
		}

		whatIsYourProfession(prof)
	}
}

func whatIsYourProfession(prof profession.Profession) {
	var message string
	switch {
	case profession.Is(prof, profession.Warrior):
		message = "Couldn't beat you in an arm-wrestling match."
	case profession.Is(prof, profession.Cleric):
		message = "Divine!"
	case profession.Is(prof, profession.Hunter):
		message = "Shoot an arrow over them mountains."
	case profession.Is(prof, profession.Mage):
		message = "I'll pick a card, any card."
	default:
		message = "I've never met a " + prof.String() + " before."
	}
	fmt.Println(message)
}

% ./p
warrior
Couldn't beat you in an arm-wrestling match.
hunter
Shoot an arrow over them mountains.
mage
I'll pick a card, any card.
cleric
Divine!
potato
I've never heard of a potato

My New Virtualization Homelab

Fri, 12 Nov 2021 22:03:41 -0600

Behold!

The specs

Processor: Intel Core i5 11400 6C/12T
Motherboard: ASUS Prime 560M-A
RAM: 32GiB DDR4 @ 3200MHz (Corsair Vengeance)
SSD0: Samsung Pro 970 256GB
SSD1: Tcsungbow 1TB ¯\_(ツ)_/¯
Case: CoolerMaster N200 MicroATX
Hypervisor/host OS: Debian 11 Bullseye

Why?

I like virtualization and I want an always-on server to continue to learn and experiment with the KVM virtualization ecosystem.

Why not used enterprise gear?

Looks big and loud. Most of the used gear at my price point seemed kind of old, too. And besides, if this ever stops being a hobby for me, I can just throw a video card in this (if such a mythical thing can ever be found again) and find it a new home.

Running an ARM64 OpenBSD Virtual Machine on Apple Silicon

Sun, 07 Nov 2021 11:08:48 -0600

Install an HVF-equipped build of QEMU

The QEMU developers recently merged Apple Silicon support for Apple’s Hypervisor.Framework virtualization layer. This means that barring any complications or removals, the next release tag for QEMU should include this support.

For more information, see the 6.2 planning page on the QEMU wiki.

Alas, if you are reading this before the next release of QEMU that contains this code is generally available, you will need to build QEMU from source.

If you use the Homebrew package manager, then this means you already have the Xcode build tools installed, so this can be as easy as:

$ git clone https://gitlab.com/qemu-project/qemu.git
$ cd qemu
$ brew install meson
$ brew deps qemu | xargs brew install
$ mkdir builddir
$ cd builddir
$ ../configure --target-list=aarch64-softmmu
$ meson compile
$ sudo meson install

If the idea of building and installing the bleeding edge of QEMU is offensive to you, then you may wish to check out the latest tag and apply the Apple Silicon Hypervisor.Framework patches on top of it.

Download the OpenBSD 7 Install Image

Create a directory for stashing install media:

$ mkdir -p $HOME/virt/images/base

Download the install image and the SHA256 info:

$ cd $HOME/virt/images/base
$ curl -LkO https://cdn.openbsd.org/pub/OpenBSD/7.0/arm64/SHA256
$ curl -LkO https://cdn.openbsd.org/pub/OpenBSD/7.0/arm64/install70.img

Always check the checksums:

$ shasum -a 256 -c SHA256 --ignore-missing
install70.img: OK

Create a new virtual disk

$ qemu-img create -f qcow2 openbsd.qcow2 15G

Set up the launch script

cat > run_openbsd.sh <<EOF
#!/usr/bin/env sh

qemu-system-aarch64 \
    -M virt \
    -accel hvf \
    -m 2048 \
    -cpu cortex-a57 -M virt,highmem=off \
    -drive file=/usr/local/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on \
    -drive file=install70.img \
    -drive file=openbsd.qcow2,format=qcow2 \
    -nographic \
    -serial tcp::4444,server,telnet,wait
EOF

Don’t forget to replace the path to your openbsd.qcow2.

Mark it as executable:

$ chmod u+x run_openbsd.sh

In the next step you will launch the VM, but won’t see anything. That is because QEMU is waiting for you to connect over telnet before it proceeds.

So, in one window:

$ ./run_openbsd.sh

And in another window:

$ telnet localhost 4444

Wait for the early boot messages to finish and eventually you should see something resembling an OpenBSD install prompt.

Install OpenBSD

This step is up to you! Just remember that once it’s finished installing, remove the -drive line in your run_openbsd.sh that contains the install70.img. You don’t need it anymore!

Enjoy your super-fast OpenBSD VM on your super-fast Apple Silicon machine!

Set up host to guest SSH access

You don’t have to live your life exclusively over telnet. If you’d prefer to SSH in at this point, go ahead and add these lines to your run_openbsd.sh launcher script:

    -net user,hostfwd=tcp::2222-:22 \
    -net nic \

This will forward port 2222 on your host to port 22 in the guest (which is the default port that sshd will listen on).

In the guest, enable sshd:

# rcctl enable sshd
# rcctl start sshd

On the host, copy your public key over (unless you’d rather type the password for the user in your guest)

$ ssh-copy-id -i ~/.ssh/id_ed25519 -p 2222 ckuehl@localhost

Finally, now you can simply SSH into your guest whenever you want:

$ ssh -p 2222 ckuehl@localhost

A Faster Way to Create Virtual Machines with Cloud Images and virt-manager

Fri, 09 Jul 2021 10:30:05 +0000

I’ve written previously about Booting Cloud Images with QEMU. However, I’ve since graduated to a more convenient method of spawning virtual machines. This method is also much faster and is more cohesive with the rest of the virtualization stack that you’ll find on your Linux distribution. As someone who creates and tears down tons of virtual machines for testing things, this method appeals to me more than the previous. Let’s get into it.

Download a Cloud Image

Ideally, you’ll find a download link for a QCOW2 VM image. Otherwise, you’ll need to convert it with qemu-img convert. As before, I will use a Fedora cloud image.

# mkdir /var/lib/libvirt/images/base
# cd /var/lib/libvirt/images/base
# curl -LkO https://download.fedoraproject.org/pub/fedora/linux/releases/34/Cloud/x86_64/images/Fedora-Cloud-Base-34-1.2.x86_64.qcow2

Generate a cloud-init bootstrap image

This can be a one-time thing, if you don’t mind each of your new VMs taking on these settings.

# mkdir cloud-init-bootstrap
# cd cloud-init-bootstrap
# touch meta-data
# cat > user-data <<EOF
#cloud-config

system_info:
  default_user:
    name: fedora

chpasswd:
  list: |
    fedora:password
  expire: False

resize_rootfs: True
EOF
# genisoimage -output seedci.iso -volid cidata -joliet -rock user-data meta-data

Obviously, if this machine is going to be accessible via the Internet, you’ll want to follow some better security practices. For testing kernel changes, it’s probably fine.

Create a disk for the new VM

You can just use exactly what was downloaded, however, I like to create a new image which is based on the original download. That way I can create new VM disks without having to re-download or remember to copy and paste the original disks. It’s also convenient to make a larger disk than the base image, as cloud-init will take care of resizing the file system for you.

# qemu-img create -f qcow2 \
  -b /var/lib/libvirt/images/base/Fedora-Cloud-Base-34-1.2.x86_64.qcow2 \
  -F qcow2 \
  /var/lib/libvirt/images/fedora-34.qcow2 15G

I tend to name the disk what I intend to name the virtual machine for identification purposes.

Create the new domain (VM)

virt-manager comes with a helpful tool called virt-install to provision a new virtual machine. Note that the shell prompt has switched from root to a user session.

$ virt-install --name fedora-34 --os-variant fedora --vcpus 2 --memory 2048 \
  --graphics vnc --virt-type kvm --disk /var/lib/libvirt/images/fedora-34.qcow2 \
  --cdrom /var/lib/libvirt/images/base/cloud-init-bootstrap/seedci.iso

Enjoy your new virtual machine

The VM may now be managed on the command line with virsh or with the virt-manager GUI. You could leave this one in its current state and simply clone new VMs from it whenever you’d like a brand-new one, or follow these same steps again for new VMs.

Implementing a Continuous Delivery Pipeline for my Discord Bot with GitHub Actions, podman, and systemd

Sun, 04 Jul 2021 15:04:19 +0000

I’ve been having a lot of fun lately refining a weekend project I started a few months ago. I basically threw this bot over the wall back in early April. About a month ago, I started getting serious about learning the Go programming language, so I thought I’d just revisit my Discord bot with a more “learned” eye and find ways to polish it up a bit.

Popple is a Discord bot that I made for myself and my friends, and it has been my playground for practicing everything I was learning in a project with an extremely small blast radius. Actually, the blast radius is both small and sympathetic, since most of my friends in that server are software developers too; so it was easy to laugh about whatever bugs that had made it into the running version of the bot.

In any case, I’ve been pushing commits to Popple consistently each week (actually, much to my pleasant surprise, a few developers have contributed to closing some of my “good first issues” on the repo too!) This rate of change started to become cumbersome, especially since early on, I didn’t have any automation in place for this.

My deployments were all manual, and looked like this:

Push changes to the branch I wanted them on
SSH into my VPS
Run go install github.com/connorkuehl/popple@latest
Get annoyed that go-install seems to be caching the @latest, and re-run the command pasting in the latest SHA instead of the latest keyword…

Yuck!

Lucky for me, I was learning about Ansible, so I thought it’d be fun to write a playbook. The playbook would fetch the latest changes for the Git ref I passed in as a variable, it would build & install the binary to my PATH, and restart a systemd unit so that the new binary could “go live.”

This playbook was actually an enormous improvement to the manual steps; but there was still a lot to be desired here.

I didn’t want to keep the Go toolchain or git installed on my server
I didn’t want the server to have to build the code from source

Containers to the rescue

Long story short, I wrote a Dockerfile, I made an account on hub.docker.com, and I built and pushed images of the tip of my master branch as well as the commits I had already tagged as releases.

My Dockerfile looks like this:

FROM golang:1.15-alpine AS build
RUN apk add build-base      # for gcc
RUN mkdir /popple
ADD . /popple
WORKDIR /popple
RUN go build .

FROM alpine:latest
WORKDIR /root/
COPY --from=build /popple/popple .

# docker image run --rm -v path/to/db:/root/popple.sqlite \
#                       -v path/to/token:/root/bot.token \
#                       image_name
ENTRYPOINT ["/root/popple", "-db", "popple.sqlite", "-token", "bot.token"]

This Dockerfile takes advantage of what are known as “multi-stage builds.” The first image can balloon up with all of your build dependencies, but can hand off the final binary to a much slimmer image for actual use.

You can see my first image is called “build” and the second image has a COPY --from=build ... statement which copies the binary over. Go’s static linking is what buys us this convenient one-line copy. The binary is all we need.

Then, on my VPS, I could simply: podman run --name popple_bot -d ... docker.io/conkue/popple

It was great! Still rather manual, but this removed the need to keep a Go toolchain and git installed on my VPS. Furthermore, we haven’t exactly removed me from the equation here. I still have to build the new image, push it to Docker Hub, SSH in, pull the latest image and bounce the container.

Actually, this all sounds more involved than what the process was like before… but not for long! (And yes, I probably could have just updated the Ansible playbook but I knew we could automate even that step and that’s where I was moving towards… can’t stop now!)

GitHub Actions wasn’t far behind

Lucky for me, automatically pushing a container image from GitHub Actions is already a solved problem.

I found a tutorial on Docker’s website for configuring a GitHub Action that will build and push a container image from the Dockerfile in your git repo.

My action looks exactly like this (at least, at the time of this writing):

name: Deploy to Docker Hub

on:
  push:
    branches: [ master ]

jobs:

  build:
    runs-on: ubuntu-latest
    steps:
    - name: Set up caching
      uses: actions/cache@v2
      with:
        path: /tmp/.buildx-cache
        key: ${{ runner.os }}-buildx-${{ github.sha }}
        restore-keys: |
          ${{ runner.os }}-buildx-

    - name: Check out source code
      uses: actions/checkout@v2

    - name: Login to Docker Hub
      uses: docker/login-action@v1
      with:
        username: ${{ secrets.DOCKER_HUB_USERNAME }}
        password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}

    - name: Set up Docker BuildX
      id: buildx
      uses: docker/setup-buildx-action@v1

    - name: Ship it
      id: docker_build
      uses: docker/build-push-action@v2
      with:
        context: ./
        file: ./Dockerfile
        push: true
        tags: ${{ secrets.DOCKER_HUB_USERNAME }}/popple:latest
        platforms: linux/amd64,linux/arm64
        cache-from: type=local,src=https://codeofconnor.com/tmp/.buildx-cache
        cache-to: type=local,dest=/tmp/.buildx-cache

Okay, so far, this is fantastic, but I’ve only removed the need for me to build and push the container images from my local machine, but that’s a step in the right direction.

podman-auto-update would like to join the party

Actually, before I even knew about podman-auto-update(1), I had generated a systemd unit for my Podman container, but I am going to reorder the events so that I look like a capable and highly observant blog author.

Turns out, podman will check to see if there’s a new version of your container image for you. All you have to do is ask. The podman version shipped in the Enterprise Linux distribution that runs on my VPS doesn’t seem new enough to support the registry value for the io.containers.autoupdate label; which appears to be favored by the documentation. I would have used that if it did. It does support the image value, so:

$ podman run --label="io.containers.autoupdate=image" -d --name popple_bot \
  ...snipped \
  docker.io/conkue/popple:latest

podman-auto-update(1) is aware of the fact that many might choose to have systemd start or stop their containers, and so it will behave nicely, especially once you see the next step where we generate a systemd unit for this bot.

But first, let’s enable and start the systemd timer that podman ships, so that auto-updates are indeed automatic:

$ systemctl enable podman-auto-update
$ systemctl start podman-auto-update

The timer will automatically check for updates once per day at midnight, but this can be tweaked according to the podman-auto-update(1) man page. I’ll probably leave it at the default setting to see how it feels, especially since I’ve pretty much added everything I want to add to the bot.

You can also just SSH in and run podman auto-update if you want to trigger this early.

Giving a daemon the reigns

I like setting up systemd units so that when my VPS is restarted everything comes back up nicely. podman makes this easy too:

$ podman generate systemd --new --name popple_bot > /etc/systemd/system/popple.service
$ systemctl daemon-reload
$ systemctl enable popple
$ systemctl start popple

Conclusion

Let’s take a look at my to do list:

~~I don’t want to manually SSH in to the VPS to deploy~~* podman-auto-update will automatically pull the latest bot container image
~~I don’t want to manually build and push container images~~* GitHub Actions will automatically push a new image of the latest commit to master when I push
~~I don’t want my VPS to have git or the Go toolchain installed~~* Just needs podman!
~~I want my bot to automatically come back up when the server restarts~~* podman generated a systemd unit for me!

Sweet! All in a morning’s work. Now I can just kick back, push some commits, and watch the changes roll out automatically.

How I use Thunderbird to Write Emails and Review Patches

Wed, 21 Apr 2021 18:17:29 +0000

Regardless of how one might feel about patches-over-email software development, the reality is that a lot of exciting open source projects are developed on mailing lists. Configuring a pleasant plaintext-oriented e-mail environment may not be obvious for those of us who come from a primarily git forge style background. At least, it wasn’t for me. In any case, I’ve finally arrived at a productive setup and I’d like to write it down here for posterity.

Note: This is not a guide to sending git patches with Thunderbird. I use git-send-email(1) exclusively for that.

Initial Configuration

The first thing I do after installing Thunderbird is apply Linux kernel’s recommended settings. After that, I select “Edit” > “Account Settings” and modify the following:

“Copies and Folders” tab: Select “Place replies in the folder of the message being replied to”
“Composition & Addressing” tab: De-select “compose messages in HTML format; and for the “When quoting,” dropdown menu, select “start my reply below the quote”

Note: As of this writing, the recommended “exteditor” Thunderbird addon is not compatible with the latest version of Thunderbird. I describe how I cope with this later on in the post.

Reviewing Patches

For a one-time setup, install the Colored Diffs Thunderbird addon. It will apply git-diff coloring to email messages if it detects a diff in the body of the message.

Depending on the scope of the change, I may choose to review the patch exclusively within Thunderbird by reading the diff, or I may choose to apply the patches locally to my git tree to play around with them. The aforementioned Colored Diffs addon makes the former much more pleasant.

If there’s just a few patches, I might open a terminal window and navigate to my git tree. Once I’ve done that, I’ll right-click the first patch > Select “Copy to Clipboard” > Select “Message”. Then I can apply the patch:

$ git am -
<PASTE>
^D

I’ll repeat that for each patch.

This doesn’t scale well for more than 2 to 3 patches though. If there are multiple patches, I will select each patch and then right-click > “Save selected messages” > “As mbox file (new)” > Save. Then, in a terminal window, I can navigate to my git tree and just git am /tmp/patch/to/patches.mbox.

Late to the party?

If you subscribed to a mailing list and you stumble upon some patches or messages that occurred prior to your subscription but you still want to participate in, not all is lost. Many mailing list archives let you download previous emails, usually bundled by the month they were sent.

I use the ImportExportTools NG Thunderbird addon to import emails from the archives into Thunderbird. Once they’ve been imported, you can reply or read them at your leisure. I’ve found this to be a good way to get caught up on mailing lists. For reading a short thread or two, the HTML archive view is fine; for anything more involved than that, it’s more comfortable to read inside of Thunderbird, in my opinion.

Composing Emails

For a while, I would use the regular editor that comes with Thunderbird. However, it can be hard to consistently indent things or keep the width of my paragraphs to a consistent level in the editor since it doesn’t include line or column information. I also prefer using Vim or Vim keybindings to edit text.

Since the exteditor addon is incompatible with newer versions of Thunderbird, I just open a terminal emulator and type my messages up in Vim. If I am replying to an email and I want to quote parts of it, I will just cut and paste the contents of the Thunderbird editor into Vim, then I put Vim into email mode:

:set filetype=mail colorcolumn=72

Or, to spend less time typing:

:set ft=mail cc=72

This way the syntax highlighting is much nicer with quoted snippets and I can keep the widths of my paragraphs consistent with the color column or column information that Vim shows me. Furthermore, with the filetype set to “mail”, Vim will automatically break to a new line when I threaten to exceed the conventional 72 character width of plaintext emails.

When I am ready to send my email, I just copy the contents of my Vim buffer and paste it into the Thunderbird editor and hit “Send”.

The xclip(1) package is very useful for this. To copy the entirety of my Vim buffer, all I have to do is type:

:w !xclip -sel clip

Then select the Thunderbird editor and hit ctrl+V!

Note: I don’t have a suggestion for a Windows or macOS equivalent to the xclip(1) tool. Sorry! If you know of one, let me know and I can update this post.

Conclusion

That just about sums up everything about my Thunderbird workflow that makes me feel productive.

There is no reward for reading a blog post as banal as this. 🙂

Tips for Improving the Quality of Your Code Reviews

Mon, 19 Apr 2021 08:16:00 +0000

Check out the changes locally

For everything but the most trivial of patches, check change out locally. Not only is this a technical prerequisite for some of the other tips in this article, but I’ve found it is easier to remain focused on the review when it takes place outside of my email inbox/GitHub/Gitlab/etc.

Use more context when viewing changes

The default context for a diff is rather narrow. It will show lines added and removed next to only a few other lines of the code. This can reduce the efficacy of the review unless you would consider yourself already intimately familiar with the surrounding codebase without having it in front of you.

If you’re on the command line, the --unified=<n> or -U<n> flag can be passed to git-diff(1) and git-show(1) to show more context for the diff. Some graphical git tools show the entire context by default. I do almost everything git-related on the command line, but lately I have been using the gitk GUI client when reviewing patches because it shows the entire context for each patch and makes it easy for me to jump around from patch to patch.

This lets me see the entire contents of the files that the diff modifies. Expanding the context makes it particularly easy to see:

if the change is stylistically consistent with the rest of the code;
if there are any existing abstractions (types, functions, constants, etc) that could have been leveraged to better accomplish this change, but weren’t;
if some other part of the code should have been updated to take advantage of this change, but wasn’t;
if there’s a better home for this change

The list above is a great way to better familiarize yourself with the code while performing code reviews.

Exercise it yourself

Hitting the code paths that the patches modify is especially critical if the change didn’t come without any automated unit, integration, or end-to-end tests. If this is the case, should it have come with automated tests? Probably. This is very dependent on the thing you’re working on.

There are a number of benefits to figuring out how to exercise these code paths:

It allows you to learn or verify what you know about the “flow” of the code. What modules/types/subsystems are crossed from start to finish to reach this code path? How do they communicate?
It may help you spot potential improvements to the patch or confirm its correctness.
If it is a customer/user visible change, try to empathize with them. What is the experience like for a non-technical user compared to a technical user?

Determine the change’s discoverability (if applicable)

Should this change include documentation? If so, does it already? If it’s a new subcommand to a command line tool, is it added to the tool’s usage or --help output? Is this an addition to some API that is meant to be consumed by other services or developers? Where will they learn about its existence?

sevctl available soon in Fedora 34

Thu, 08 Apr 2021 08:25:00 +0000

I am pleased to announce that sevctl will be available in the Fedora repositories starting with Fedora 34. Fedora is the first distribution to include sevctl in its repositories 🎉.

sevctl is an administrative utility for managing the AMD Secure Encrypted Virtualization (SEV) platform, which is available on AMD’s EPYC processors. It makes many routine AMD SEV tasks quite easy, such as:

Generating, exporting, and verifying a certificate chain
Displaying information about the SEV platform
Resetting the platform’s persistent state

As of this writing, Fedora 34 is entering its final freeze, but sevctl is queued for inclusion once Fedora 34 thaws. sevctl is already available in Fedora Rawhide for immediate use.

Please submit all bug reports, patches, and feature requests to sevctl’s upstream repository on GitHub.

Binary RPMs can be found in the rust-sevctl packaging repo.

Booting Cloud Images with QEMU

Sun, 24 Jan 2021 10:31:46 +0000

Do you ever get frustrated with waiting for a heavy VM image to download or with installing operating systems onto virtual machines manually? It can start to feel cumbersome after a while, especially if you bring up and tear down lots of virtual machines as part of your workflow. It’d be nice if spawning a ready-to-use VM was as quick and as easy as it is when using a public cloud.

Good news! It can be. Many Linux distributions (and BSDs) publish what are known as “cloud images” for use in virtual machines. These images are purposefully tiny and contain the bare minimum for what is required to provision a virtual machine.

Create a directory structure for your VMs

This is up to you, however, here’s mine (I like to keep it simple):

$ mkdir -p $HOME/VM/images/base

Download a cloud base image

I’ll use a Fedora Cloud Base Image for the virtual machines, but like the previous step, you can find your own cloud image from your favorite OS distribution. Distributions typically do a good job of explicitly pointing out which download is a cloud image, so I don’t think you’ll have much trouble finding one.

$ curl -Lk https://download.fedoraproject.org/pub/fedora/linux/releases/33/Cloud/x86_64/images/Fedora-Cloud-Base-33-1.2.x86_64.qcow2 > $HOME/VM/images/base/fedora_cloud_base_33.qcow2

The important thing here is that the cloud image is a qcow2 image. If it’s a raw image, you’ll have to convert it like so:

$ qemu-img convert -f raw -O qcow2 path_to_raw_image.raw new_filename_for_qcow2_image.qcow2

Create a new image from the cloud base image

I usually make a new folder where I can put artifacts related to my individual VMs as needed:

$ mkdir $HOME/VM/fedora_33
$ qemu-img create -f qcow2 -b $HOME/VM/images/base/fedora_cloud_base_33.qcow2 $HOME/VM/fedora_33/hdd.qcow2 8G

Deriving a new image from the base image is a neat trick because it allows us to create new VM images from the base image without having to download a new image again.

Create a cloud-init bootstrap image

The cloud image that we’re using requires cloud-init to perform some first-time setup. I’m trying to use only the bare minimum configuration to bring up a virtual machine here, but curious readers should head over to cloud-init’s documentation if they’d like to learn other ways that it can be used to customize their virtual machine.

$ cd $HOME/VM/fedora_33
$ touch meta-data
$ cat > user-data << EOF
#cloud-config

system_info:
   default_user:
     name: fedora

chpasswd:
  list: |
    fedora:password
  expire: False
 

resize_rootfs: True
ssh_authorized_keys:
   - ssh-rsa AAAAB3Nza...
EOF
$ genisoimage -output seedci.iso -volid cidata -joliet -rock user-data meta-data

Launch your virtual machine

Here is a super minimal QEMU command line for launching your new VM! You’ll want to supplement this with more arguments to fit your needs (i.e., -accel, -cpu, etc).

$ qemu-system-x86_64 -m 512 -drive file=hdd.qcow2 -cdrom seedci.iso

Log in with the user account that was described in the user-data file. In the case of this article, the username is fedora and the password is password. Obviously this configuration is mainly for cheap, disposable VMs that will be used for local development/testing. This configuration is unsuitable for a production environment.

You can remove the -cdrom argument from the command line for subsequent launches because cloud-init will only use it for the instance’s first boot to configure things.

How Rust's Type Checker Helped Find a Bug in a Linux Kernel ioctl Definition

Sat, 05 Dec 2020 11:45:27 +0000

Don’t you love it when your compiler thinks hard so you don’t have to? Rust’s built-in static analysis is praised for providing all kinds of safety guarantees for your code. Today, it’s not about your code, or even my code; it’s about how calling Linux ioctls through a type-safe abstraction layer exposed a bug in an ioctl definition and Rust’s type-checker was the first one to bark about it!

iocuddle is a library for improving the safety of ioctl calls from Rust. But what’s so unsafe about ioctls that we need a crate for it in the first place? The Linux kernel’s ioctl mechanism is a minimal interface that allows Linux module developers to provide APIs to userspace that don’t necessarily fit the mold of the primary module classes: char, block, and net. To this end, the ioctl function definition must be broad enough to avoid constraining the interfaces that module authors can expose.

Let’s take a look:

#include <sys/ioctl.h>

int ioctl(int fd, unsigned long request, ...);

Most readers will probably agree that this minimal interface allows for a large number of valid inputs. In fact, there’s not much out there that this function won’t accept at compile-time or at run-time. As a result, the kernel only needs one syscall instead of allowing driver developers to add zillions for their specific module.

Unfortunately for userspace programmers, this means a great deal of care must be taken for each ioctl call site to ensure the correct data are passed in. If a mistake is made, it won’t be discovered until run-time.

Readers who are already familiar with the Rust programming language may already know that, among other things, Rust ensures type-safety with the ferocity of one thousand suns. Furthermore, most programmers in general would probably agree that they’d rather be notified of an error at compile-time rather than run-time.

iocuddle exists to help provide these guarantees to Rust applications and libraries. It shifts the care that must be taken from each call site and funnels it into one place: the initial iocuddle::Ioctl definition. It is here that the crate author must ensure they’ve found the correct ioctl type from the Linux headers and that they understand the ioctl’s “direction” (though this is also found in the Linux headers). After that, the Rust definition of the ioctl is complete, and client code can enjoy compile-time type-checking of what they’re sending in to an ioctl call.

And yet… it wasn’t working for me! The internet lied to me! Rust is no panacea!1!1 (joking, joking…)

Despite quadruple checking that I had the right ioctl type:

const KVM: Group = Group::new(0xAE);

against the Linux kernel’s:

uapi/linux/kvm.h:#define KVMIO_0xAE

as well as checking that I had the right ioctl number:

pub const PIN: Ioctl<Write, &Command<Pin>> = unsafe { ENC_OP.write(0xbb) };

against the Linux kernel’s:

uapi/linux/kvm.h:KVM_MEMORY_ENCRYPT_REG_REGION   _IOR(KVMIO, 0xbb, struct kvm_enc_region)

My ioctl call was failing with ENOTTY, no appropriate ioctl.

Have you spotted the bug? Oops. I expressed the iocuddle::Ioctl as a Write direction (_IOW) ioctl, not _IOR as it’s shown here in the Linux kernel source. Silly me! Under the covers, iocuddle performs the same ioctl construction as the ioctl constructor macros. So indeed, mislabeling the ioctl direction will result in an ioctl with ENOTTY.

I had been debugging for a while, and was getting tired, so I just quickly changed the direction in my iocuddle::Ioctl definition and found my program fails to compile. More specifically, I’m unable to define this Ioctl such that I can send in my own data that the kernel needs to fulfill my request! That can’t be right, according to the documentation for this ioctl, I’m supposed to supply information:

4.111 KVM_MEMORY_ENCRYPT_REG_REGION

Capability: basic
Architectures: x86
Type: system
Parameters: struct kvm_enc_region (in)
Returns: 0 on success; -1 on error

Readers who are already familiar with Linux ioctls may already agree that iocuddle’s types weren’t wrong, they just weren’t letting me express an illegal state. Why is this an illegal state? Let’s look into ioctl directions a little more closely:

The Linux kernel ioctl definitions are built using macros that provide hints as to the direction that data flows during the course of the ioctl call:

_IOR indicates that userspace will read data from the kernel. Generally, this means that userspace provides some buffer or struct for the kernel to write to, and if the ioctl succeeded, then userspace can inspect the contents of that buffer or struct for whatever information the ioctl promised to give them.
_IOW indicates that userspace will supply data to the kernel. If the ioctl succeeds, then userspace can assume that the kernel took whatever actions were described in the ioctl and its relevant buffers.
_IOWR indicates that data can flow in both directions for this ioctl. This is quite common, as one ioctl may be used to multiplex different commands. Often times, the data passed in to the kernel is a struct that describes what subcommand to execute during the course of the ioctl.

So really, iocuddle’s types were forcing me to define types that make sense for the semantics of the ioctl. A Read direction iocuddle::Ioctl has no business allowing the caller to send in its own buffer. The iocuddle::Ioctl will zero out its own buffer and let the kernel scribble on it before handing that to the caller.

Indeed, everything about the documentation for this ioctl suggests that data flows from userspace into the kernel, and while Rust’s type-checker didn’t point that out in so many words, it’s the only thing at compile-time that pointed out something is wrong with this ioctl.

While it is a great emotional victory to know that this ioctl should have been constructed with _IOW, it is a futile one, for the ioctl has already been etched into the stone tablet that is the Linux syscall ABI and it so it shall be… forever.

Hacktoberfest 2020 Was Not All Bad

Sat, 31 Oct 2020 18:02:53 +0000

Hacktoberfest 2020 had a rocky start. I’m not here to argue against any of the criticisms brought up by other members of the community. Their feedback is not unfounded. However, I don’t believe it was all bad.

I signed one of my weekend projects up for Hacktoberfest to gain some more experience in a maintainer role rather than an individual contributor role. In this regard, I believe Hacktoberfest 2020 was a successful experience for myself and for the contributors who spent their time and energy submitting patches to my project.

What follows are some of the lessons I’ve learned through this experience.

Disclaimer

Before getting into it, I’ll just throw a disclaimer up. I’m perfectly aware that my sample size of 1 does not reflect the state of all open source projects. Furthermore, this experiment was entirely fueled on an event that offers contributors a chance at getting a free T-shirt.

However, a maintainer can’t control the reasons why contributors to contribute. There are a number of reasons why people might contribute:

They are paid to contribute
They want to fix something they’re experiencing
They want to sharpen their own skills on a project that uses a language or tools or some underlying concept they want to learn
They want to win a free T-shirt

To a maintainer, the experience is more or less the same regardless of what motivates a contributor to contribute. At least, for legitimate contributions.

More momentum = more contributions

What I mean by momentum is the activity levels that an open source project produces. For very large projects, it almost looks like this bus drives itself. Ubiquitous projects, either through popularity or practical use, see hundreds if not thousands of issues filed against them. Patches/pull requests/merge requests may meet or exceed even the number of issues filed against them.

Regardless, the more popular or ubiquitous a project is, the more work it is generating. These are all opportunities for contributions.

For a smaller project–much less a project that no one uses or cares about (not trying to diss myself, but really, who uses Trivial File Transfer Protocol in 2020?)–contribution opportunities are not as visible.

The maintainer must make these tasks obvious by filing issues themselves.

I found that the more issues I filed, the more pull requests I would get. In fact, a quick look through all pull requests not authored by me reveals that 100% of the pull requests from other people are associated with an issue that has already been filed.

Furthermore, I’ve noticed that when I stopped filing issues, the new contributor count started waning.

My takeaway from this is that it is encouraging to new contributors to see visible work as a jumping-off point for them to get started with making contributions.

Generally, this will be issues/bugs filed in some kind of project tracker. I also consider reviewing pull requests as contributing; however, during my experiment, only one contributor reviewed someone else’s work on the repo and they only did so after having some of their own patches merged.

Removing ambiguity increases contributions from new/inexperienced contributors

Speaking of visible work, I’ve found it is helpful to remove ambiguity from bugs/issues filed in the tracker. Some might argue this is almost to the point of spoonfeeding. I don’t think so.

Clearly stating the acceptance criteria and providing a loose outline of hints for how one might orient themselves to the task and the part of the codebase that it pertains to resulted in a large number of well-defined issues being closed on my repo during Hacktoberfest.

It can be tempting to write the bare minimum that you believe will jog your memory of what must be done while working on a project with a close-knit team of people who are responsible for different parts of the codebase (or just by working by yourself).

Fight that urge. If it will take a significant amount of time to decompose the task in this way when you’re filing the issue, then the issue can be marked as a meta or a large-scoped issue. I’d encourage you to revisit the issue and break it down further when your schedule allows. The advanatages of this are two-fold:

A little bit of time designing can save a lot of time coding
Someone else might do the work for you if they have some clue what a way forward might look like

It takes up a lot of your time

I get it now. Before, I was sympathizing. Now, I’m empathizing. I know from my day job that reviewing code, especially for larger changes, takes a lot of time and energy. I maintained a much smaller project during Hacktoberfest than I do for work. I’m also paid to do those things at work; but no one is paying me for my toy TFTP project :-)

Encouraging contributions, reviewing them, and shepherding them all the way through to be merged can take a lot of time. Some pull requests might take 5, 10, 15 minutes to review. Others might take 30 minutes or an hour. They might also require more than one pass over while you cogitate over it!

Sometimes this time investment just isn’t palatable if you’ve had a tough week at work, or if you want to invest your spare time in some of your other hobbies, or spend time with people you care about, etc.

Overall

To a lot of the internet, it seems like Hacktoberfest as a net negative. It was worth it to me, though. I got to learn a lot about maintaining a project in a controlled environment.

In hindsight, much of this article reads like common sense. I’ve found that after spending too much time in the weeds with one workflow or another, sometimes we lose sight of the fundamentals. A little reflection is always a breath of fresh air.

My hope is that some of the contributors, especially those that struck me as newer to the software field, took something away from this process, too.

Test Driven Development in the Clang Compiler

Thu, 25 Jul 2019 15:48:13 +0000

A while back, I participated in a software engineering capstone with a group of other computer science students to complete my degree. Our project was to create a from-scratch implementation of grsecurity’s “randstruct” GCC plugin for the Clang compiler. Long story short, we ended up sending out a request for comments (RFC) on the initial draft that we produced during the capstone. A number of Clang/LLVM contributors took the time to review what we made and kindly suggested some changes for a future revision.

Since then, not much has happened with the Clang Randstruct RFC. We (the members of the capstone team) have all gone our separate ways and have been busy settling into post-university life.

Over the past week or so I’ve been re-familiarizing myself with the project and working on a new revision that addresses all of the comments from the RFC. The primary concern with the RFC was test coverage — something that we kind of scratched our heads over during the implementation of our RFC draft.

The entire time we had been compiling source code with our feature and without the feature and manually inspecting structure layouts with either pahole, gdb, or even just a test program that calculated and printed the offset of a structure’s field(s). The expectation of that last manual test method being that it is–of course–a different offset than the unmodified compiler produces.

Can we even automate that? What do we do? Do we compile something and compare a diff of the assembly? A diff of the pahole output? Is that even accurate? What kinds of inconsistencies or “noise” would throw that off? I mean, we could certainly compare the outputs of the offset calculation test programs, but what kind of coverage would you call that? Toy programs don’t even scratch the surface of complexity of many “real” software projects.

It turns out we were overcomplicating it. I don’t mean that in a rude way. Often times when I’m introduced to a problem area for the first time I’ve found that my first solutions often do overcomplicate things. This is a normal learning experience.

We felt slightly lost in the massive, complex project that is the Clang compiler. We weren’t completely in the dark, but we were holding a candle, not a lighthouse (granted, one could never hold a lighthouse, the analogy here is about illumination). Clang is huge and complex, but it also exposes powerful interfaces into its internal machinery.

Last week I started poking around the unittests/ folder to get a feeling for how the tests are written in the first place. After a little bit of searching, I found this:

AST0 = tooling::buildASTFromCodeWithArgs(Code0, Args, InputFileName);

You can’t see me, but I’m wiping a tear from my eye. It’s beautiful. This function will compile the code that lives in the string called Code0 with the compiler arguments stored in Args (the InputFileName is unused for my purposes, since the code is in the string). It returns the abstract syntax tree that results from compiling the code. It is the star of this blog post.

Our entire project is predicated around manipulating the representation of a RecordDecl (Structure) in Clang’s Abstract Syntax Tree. This is a huge win.

I like test driven development. I also know that test coverage is primarily what needs to be addressed for the next revision of this if there’s any hope of getting this code upstream. So I’m thinking: why not outline all of the tests that were suggested by the reviewers and try to “rewrite” Clang Randstruct following the principles of test driven development? This way I can more easily address and iterate on all of the concerns outlined for this next revision while increasing test coverage the whole time! (And maybe fix a bug or two along the way.)

So, without further ado, I made a file in the unittests/AST/ directory and wrote these two functions:

static std::unique_ptr<ASTUnit> MakeAST(const std::string& SourceCode, Language Lang)
{
    auto Args = getBasicRunOptionsForLanguage(Lang);
    auto AST = tooling::buildASTFromCodeWithArgs(SourceCode, Args, "input.cc");
    return AST;
}

static RecordDecl* GetRecordDeclFromAST(const ASTContext& C, const std::string& Name)
{
    return FirstDeclMatcher<RecordDecl>().match(C.getTranslationUnitDecl(), recordDecl(hasName(Name)));
}

The first function taps into Clang’s tooling library to produce an AST for some code. The second function simply fetches the first RecordDecl from the AST with the matching name.

Here’s an example. This is one of the first tests I wrote when I was ready to begin porting over the randomization code from our RFC (since this is a TDD rewrite, after all):

TEST(StructureLayoutRandomization,
StructuresLayoutFieldLocationsCanBeRandomized)
{
    std::string Code =
        R"(
        struct test_struct {
            int a;
            int b;
            int c;
            int d;
            int e;
            int f;
        };
        )";

    auto AST = MakeAST(Code, Lang_C);
    auto RD = GetRecordDeclFromAST(AST->getASTContext(), "test_struct");
    RandomizeStructureLayout(AST->getASTContext(), RD);
    std::vector<std::string> before = {"a", "b", "c", "d", "e", "f"};
    std::vector<std::string> after = GetFieldNamesFromRecord(RD);

    ASSERT_NE(before, after);
}

Obviously, this won’t compile. I don’t want to belabor the principles of test driven development here, but the test is written first, then we write just enough implementation code to get the test to compile and pass.

To build enough of Clang just for our unit tests to run, make -j4 ASTTests will suffice.

The Clang test suite is massive, so running make clang-test is a great way to lose an afternoon to compilation times.

Similarly, running all of ASTTests will age you. We can run just our tests by passing a gtest filter:

$ ./tools/clang/unittests/AST/ASTTests --gtest_filter=StructureLayoutRandomization*

Now, rinse and repeat.

This has resulted in what I consider to be a much tighter and more responsive development cycle than before. It is much faster to compile the ASTTests than it is to compile Clang and point it at a test program manually. It has eliminated manual structure layout inspection. Failing an assertion results in a more human-friendly “this does not equal that” type message. Since each change was motivated by the appearance of a unit test it’s much easier to track down where a bug was introduced.

Practicing test driven development inside Clang has been a really motivating experience. I’ve rewritten most of the functionality with automated test coverage executing every code path (which is a sharp increase from 0%). Not only that, but adding some of the unit tests suggested by the upstream contributors has helped guide the implementation and verify correctness. I was able to refactor some of the randomization algorithm without breaking a sweat since I already had test coverage to help verify my assumptions about how everything should work!

Not to mention, the test suite provides a much shorter feedback time. It’s also no longer required to maintain small test programs and compile one with the feature enabled and one version with Randstruct disabled. Since it’s automated, there is no manual structure layout inspection required, which has helped me iterate on new features much quicker. This is an important factor that helps me stay motivated.

Every large, unfamiliar code base seems nebulous at first and many of its helpful internal features are hard to discover sometimes. This post is an ode to Clang’s internals. I’m that happy to have stumbled upon this foothold for helping me work on this next revision.

How the Linux Kernel Detects PCI Devices and Pairs Them With Their Drivers

Sat, 01 Jun 2019 18:45:42 +0000

Have you ever wondered how Linux knows what PCI devices are plugged in? How does Linux know what driver to associate with the device when it detects it?

In short, here’s what happens:

During the kernel’s init process (init/main.c), various subsystems are brought up according to their “init levels.” Among these early subsystems are the ACPI subsystem and the PCI bus driver.
The ACPI subsystem probes the system bus. This “probe” is actually a recursive scan since there can be other devices that act as “bridges” from that main system bus.
Each bus is probed, that is, asked to enumerate the devices that are connected to them. It’s at this point we’ll start seeing their sysfs entries.
For each device that the bus sees, it will attempt to associate a device driver to it. How does it know to do this? Well, it’s actually up to the device driver. It’s the driver’s responsibility to export a table of devices that it will support when it registers itself to the PCI subsystem. This is used by the hotplug system to map modules to the PCI devices they support. It’s basically a phone book of who we need to call when dealing with a given PCI device.
Assuming a match, the kernel will (eventually) call the driver’s probe() function, and the device driver can decide whether or not it claims the device. Yes, the kernel basically takes the device and walks up to the driver(s) that claim they can handle a certain device and then asks, “is this your kid?”
Remember, a device driver whose module_init equivalent has been called is included in this roll-call (built-in or module) so long as this device is compatible with their supported module device table. Built-in modules are asked first (according to the order that they’re linked into the kernel image).

Finally, let’s take a look at a stack trace from a kernel running in QEMU. We’ll start from the bottom, and work our way up. (I’ve removed the function addresses in the stack trace to reduce clutter)

#28 ?? ()
        at arch/x86/entry/entry_64.S:352
#27 ret_from_fork () 
        at init/main.c:1087
#26 kernel_init (unused=<optimized out>)
        at init/main.c:1169
#25 kernel_init_freeable ()
        at init/main.c:1009
#24 do_basic_setup () 
        at init/main.c:991
#23 do_initcalls () 
        at ./include/linux/compiler.h:305
#22 do_initcall_level (level=<optimized out>)
        at init/main.c:915
#21 do_one_initcall (fn=0xffffffff828d6101 <piix_init>)
        at drivers/ata/ata_piix.c:1773

As part of initialization, the kernel brings itself up gradually. This gradual bring-up is described by the kernel’s init levels. Here’s the order defined in init/main.c:

pure
core
postcore
arch
subsys
fs
device
late

On our qemu-system-x86_64 system, it looks like piix_init is being invoked. This driver’s entry point is declared using the module_init macro, meaning that it belongs to the device init level. So at this point, the ACPI subsystem has already been brought up and the PCI bus has been scanned for devices.

#20 piix_init () 
        at drivers/pci/pci-driver.c:1402
#19 __pci_register_driver (drv=<optimized out>, owner=<optimized out>, mod_name=<optimized out>)
    at drivers/base/driver.c:170
#18 driver_register (drv=0xffffffff827a33b0 <piix_pci_driver+112>)
        at drivers/base/bus.c:645
#17 bus_add_driver (drv=0xffffffff827a33b0 <piix_pci_driver+112>)
        at drivers/base/dd.c:1037
#16 driver_attach (drv=<optimized out>)
        at drivers/base/bus.c:304
#15 bus_for_each_dev (bus=<optimized out>, start=<optimized out>, data=0x0 <fixed_percpu_data>, fn=0x0 <fixed_percpu_data>) 
        at drivers/base/dd.c:1021

This module registers itself as a PCI driver in its init function. This means the driver is associated with the PCI bus and is now a valid candidate for driving PCI devices. Since it’s being registered now, we can carry on and see if we can find the physical device that it should be paired with. So, we iterate through all of the eligible devices connected to the bus in bus_for_each_dev and begin searching.

#14 __driver_attach (dev=0xffff88801b14f0b0, data=0xffffffff827a33b0 
<piix_pci_driver+112>)
        at drivers/base/dd.c:944
#13 device_driver_attach ( drv=0xffffffff827a33b0 <piix_pci_driver+112>, 
dev=0xffff88801b14f0b0)
        at drivers/base/dd.c:670
#12 driver_probe_device ( drv=0xffffffff827a33b0 <piix_pci_driver+112>, 
dev=0xffff88801b14f0b0)
        at drivers/base/dd.c:509
#11 really_probe (dev=0xffff88801b14f000, drv=0x1f <fixed_percpu_data+31>)
        at drivers/pci/pci-driver.c:425
#10 pci_device_probe (dev=0xffff88801b14f0b0)
        at drivers/pci/pci-driver.c:385
#9  __pci_device_probe (pci_dev=<optimized out>, drv=<optimized out>)
        at drivers/pci/pci-driver.c:360
#8  pci_call_probe (id=<optimized out>, dev=<optimized out>, drv=<optimized 
out>) 
        at drivers/pci/pci-driver.c:306
#7  local_pci_probe (_ddi=0xffffc900000dbc50)
        at drivers/ata/ata_piix.c:1672
#6  piix_init_one (pdev=0xffff88801b14f000, ent=0x1f <fixed_percpu_data+31>)
        at drivers/pci/pci.c:1806

Wow, that looks like a mouthful at first glance. Rest assured, all that’s happening here is that for each eligible device connected to the PCI bus, the bus attempts to attach the driver to the device. This ultimately ends up calling the driver’s probe() callback, which in this case is piix_init_one().

This callback is registered drivers/ata/ata_piix.c line 1760. We also see in piix_init_one() the driver checks the PCI device ID information to determine if it should reject the device. The device is not claimed if this probe function returns an error.

#5  pcim_enable_device (pdev=0xffff88801b14f000)
        at drivers/pci/pci.c:1806
#4  pci_enable_device (dev=<optimized out>)
        at drivers/pci/pci.c:1677
#3  pci_enable_device_flags (dev=0xffff88801b14f000, flags=768)
        at drivers/pci/pci.c:1588
#2  do_pci_enable_device (dev=0xffff88801b14f000, bars=31)
        at arch/x86/pci/common.c:709
#1  pcibios_enable_device (dev=0xffff88801b14f000, mask=<optimized out>) 
        at drivers/pci/setup-res.c:456
#0  pci_enable_resources (dev=0xffff88801b14f000, mask=31)

However, if the device driver remains happy during its probe() function, it will ultimately enable the PCI device and return success.

And that’s how the Linux kernel detects PCI devices and pairs them with their device driver!

Macromancer's Spellbook: Using Macros to Stay Dry

Thu, 31 May 2018 03:42:59 +0000

It is sometimes hard to see past the thick haze of caution surrounding the use of C Preprocessor macros in your code. Indeed, the dangers of Macromancy are well-stated in many corners of the internet. You would do well to heed them.

However, there are times where a judicious use of macros can help reduce duplication in your code and make it tidier and easier to reason with. Every time a block of code is duplicated by hand, it is likely to become yet another maintenance burden to contend with. If it just so happens that some of these blocks of code contains a bug… well… you get to go fix it in more than one place.

I know! Why don’t we just extract the duplicated code into a function? Good instinct. In many cases, this may be the perfect solution to your problem. In my case, it wasn’t.

I recently upgraded the scheduling algorithm for my distribution of xv6 (a learning OS). I abstracted away the process table (an array of processes) by building multiple circular-linked state lists on top of it. Now, we have the infinitely preferable solution of not having to iterate over the entire process table when we want to find and manipulate some active processes.

Shortly into the implementation, I met my DRY monster for the project. There are many areas in the scheduling and process-handling routines where I need to traverse these state lists the exact same way, but interact with the processes differently while I’m traversing.

In my situation, the redundant code was the traversal of one or more state lists. However, embedded in those traversals were routine-specific blocks of code that operated on one or more unique local variables to that routine. As you can see, I don’t have a one-size fits all operation that could be neatly extracted into a function. I have a control structure and context-specific behavior to do within.

I could have certainly just copied and pasted my state list traversal logic into the four different routines where I needed it, but it felt so painful. What if I made a mistake? If four system-critical process routines were compromised with some stupid bug it would be was an absolutely hellish nightmare to debug. Trust me.

Here’s what it looked like at first:

struct proc **lists[] = {
  &ptable.plists.ready,
  &ptable.plists.run,
  &ptable.plists.sleep,
  // ... snipped..
};

for (int i = 0; i < NELEM(lists); ++i) {
  struct proc *rear = *(lists[i]);
  if (rear) {
  p = rear->next;
  do {

    // routine specific stuff goes here!

    p = p->next;
  } while (p != rear);
 }
}

It seems simple enough. It takes an array of circular-linked lists for it to traverse, and then inside the inner do-while is the routine-specific logic.

Copying and pasting it four times is too painful. C Preprocessor macros are just token expansions. Let’s have the compiler copy and paste it for us. The control structure for looping over an array and traversing circular-linked lists therein is the same for all four functions.

#define STATELIST_TRAVERSE(lists, cursor, code) for (int i = 0; i < NELEM(lists); ++i) { \
  if (*(lists[i])) { \
    cursor = (*(lists[i])); \
    struct proc *next = cursor->next; \
    do { \
      { \
        cursor = next; \
        next = cursor->next; \
        code \
      } \
    } while (*(lists[i]) && cursor != *(lists[i])); \
  } \
} \

It’s just a little bit hideous at first, but I got used to it as soon as I simply used that macro four times and everything just worked.

Let’s dissect this a little bit.

The backward slashes on the end of each line are so that the C Preprocessor knows that the macro substitution I’m defining will span multiple lines.

The macro takes three arguments, lists, cursor, and code. I designed it this way because I want the contract to be very clear regarding the macro’s usage. Whoever wants to use the macro must provide an array of circular-linked lists for me to traverse, they must provide their pointer by which the lists will be traversed (I call this the “cursor”), and they must supply the code they want executed with each iteration. The macro requires the caller to supply the “cursor” to the macro so that they may interact directly with whichever process the loop is currently visiting through that cursor. Basically, they can use the cursor like a really simple iterator for access to the element.

By doing this, I believe that the macro is transparent and does not obfuscate or try to take control from whoever is using it. It does not heavily pollute the routine with variable declarations within. It is a reliable scaffolding for its one purpose: traversing through a number of circular linked lists while exposing the elements to the programmer for whatever purposes they need.

Let’s see it in action. Here’s a part of the exit() routine for when a process is terminating:

// ...snipped
  struct proc **lists[] = {
    &ptable.plists.ready,
    &ptable.plists.run,
    &ptable.plists.zombie,
    &ptable.plists.sleep,
  };

  // Pass abandoned children to init.
  STATELIST_TRAVERSE(lists, p, {
      if (p->parent == curproc) {
        p->parent = initproc;
        if (p->state == ZOMBIE) {
          wakeup1(initproc);
        }
      }
  });
// ...snipped

Granted, it does look a little weird at first. As you can see, I create an array of circular-linked lists that I want the macro to traverse. Different routines will need to visit different numbers of lists, so this set-up is required for each call-site.

I pass in my pointer that I declared earlier in the routine. This will be used as my “cursor” to traverse the circular-linked list of processes.

The third argument, which I wrap in curly braces, is the routine-specific code block that I want executed at each process in the list. As you can see, I use the pointer that I passed in as the “cursor” so that I may interact directly with the element within the list.

I did it the hard way without the macro and it was painful. I abstracted it away into a macro and it trimmed down the redundant lines of code significantly across my process management routines.

Using a macro shouldn’t be your first choice for everything. If function extraction is not possible, or you are working in an environment that cannot support the overhead of function calls, etc. a macro can certainly help.

Remember to be cautious about what your macros are doing, what variables they introduce, and how they operate on arguments passed in to them.

Bonus story: I also created control commands for printing the contents of my various state lists with different formatting. Guess what really came in handy for that? The macro you just read about.