My blog has moved around several times since its inception. I first started it in college using Automattic’s hosted wordpress.com services, then I eventually moved it to a $5 VPS where I self-hosted my own copy of WordPress. After some time of that, its final form has taken the shape of a Hugo-generated static site on a $5 Linux VPS.

I’ll concede that it doesn’t take a substantial amount of toil to deploy a server and configure it to run a basic WordPress blog or static HTML pages for that matter. I’ve developed some sentimental feelings toward this blog though, and taking the proper steps to ensure that it is properly backed up or easily restored in case of tragedy adds a bit more complexity.

There are a couple of well-understood solutions that constitute a valid backup/restore/recovery plan.

The first would be to simply implement a traditional backup solution. Many cloud providers, including the one I use for my VPS, have disk-based snapshot and backup solutions. I could also integrate a file-based backup solution from inside the VPS where the site could be backed up to other locations.

Both of these scenarios would likely require me to spend more money on my blog, which I am not strictly averse to because of its importance to me. The file-based backup strategy would also require me to implement a restore strategy.

Obviously, I’d need to come up with automation for either strategy, because it’d be foolish not to. And well, that’s just more work.

I decided to go a different route. I chose to apply some of the immutable infrastructure as code principles that I’ve been learning to implement a turnkey solution that is entirely reproducible from source, no matter what. It also doesn’t require paying for my cloud provider to host snapshots or blob storage buckets.

At the heart of it, is a Terraform module that I’ve written. Terraform allows you to declare the infrastructure that you want. It will read this declaration and examine the state of your infrastructure, and it will carry out whatever actions are necessary to ensure that your infrastructure has taken the shape that you want.

In this case, I started off by declaring that I want a single DigitalOcean droplet¹.

resource "digitalocean_droplet" "www" {
  image  = "ubuntu-20-04-x64"
  name   = "www"
  region = "sfo3"
  size   = "s-1vcpu-1gb"

  ssh_keys = var.ssh_keys
}

Now, this hasn’t really saved me much work at all. If anything, it saves me the effort of clicking around on the cloud console to make a droplet and/or typing the droplet create commands at the command line.

I still have to SSH in, configure a web server, and rsync the static site over to the served directory.

Luckily, I can also specify a cloud-init script that the droplet will run after it is created and powered on.

For example, I can now automate the installation of a web server, all from the comfort of Terraform²:

resource "digitalocean_droplet" "www" {
  image  = "ubuntu-20-04-x64"
  name   = "www"
  region = "sfo3"
  size   = "s-1vcpu-1gb"

  ssh_keys  = var.ssh_keys
  user_data = <<-EOF
#!/usr/bin/env bash
  
mkdir -p /var/www-data/html

export DEBIAN_FRONTEND=noninteractive
apt-get install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list

apt-get update            
apt-get install -y caddy  
systemctl enable --now caddy
cat > /etc/caddy/Caddyfile <<EOM
codeofconnor.com {
    root * /var/www-data/html
    file_server
}
EOM

chown -R caddy:caddy /var/www-data/html
systemctl reload caddy
  EOF
}

Ah, that’s much nicer. Now all that’s required is rsync’ing the files over for Caddy to serve, which is no big deal, because I already have a simple Makefile target for that in the git repo that has all of the source markdown for my blog.

Egads! This won’t work as is, because when I rsync my files over, the owner and group is set to my user and group, and so Caddy can’t actually display them because it does not have permission to.

I can just make a quick tweak to the cloud-init script so that the caddy group is automatically applied to files added to the html directory and caddy will be able to read them:

chown -R caddy:caddy /var/www-data/html
+chmod g+s /var/www-data/html

Alas, now Terraform will have to re-create this droplet so it can create one that will have the updated cloud-init script run as part of the provisioning process. This means that the version of the site I uploaded earlier will be lost, and I’ll have to upload it again.

It’s not the end of the world, but still requires me to remember to run the deploy target from my site’s Makefile again. Not a great experience.

In most cases, this is the beauty of immutable infrastructure as code. There is almost always no doubt at all what the state of the system is in because it’s written here as code and Terraform will alert you to any configuration drift and show what’s needed to correct it.

This is also easily fixed while still practicing immutable IaC techniques.

In this case, we just have to isolate the mutable part of the infrastructure. We can just attach a block volume and populate it with the contents of the site.

resource "digitalocean_volume" "www_data" {
  region                  = "sfo3"
  name                    = "www-data"
  size                    = 5
  initial_filesystem_type = "ext4"
  description             = "Content for the webserver to serve"
}

resource "digitalocean_volume_attachment" "attach_www_data_to_www" {
  droplet_id = digitalocean_droplet.www.id
  volume_id  = digitalocean_volume.www_data.id
}

And before we forget, we want to make sure our cloud-init data mounts it to the right place so Caddy can serve files from it:

#!/usr/bin/env bash
    
export DEBIAN_FRONTEND=noninteractive
      
mkdir -p /var/www-data
+mount -o discard,defaults,noatime /dev/disk/by-id/scsi-0DO_Volume_${volume_name} /var/www-data
+echo '/dev/disk/by-id/scsi-0DO_Volume_${volume_name} /var/www-data ext4 defaults,nofail,discard 0 0' >> /etc/fstab
+
+mkdir -p /var/www-data/html
  
apt-get install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
  
apt-get update            
apt-get install -y caddy  
systemctl enable --now caddy
  
cat > /etc/caddy/Caddyfile <<EOF
${domain_name} {
    root * /var/www-data/html
    file_server
} 
EOF
  
chown -R caddy:caddy /var/www-data/html
chmod g+s /var/www-data/html
  
systemctl restart caddy

Astute readers may have noticed I moved the cloud-init script out of the heredoc in the Terraform module. For completeness, the Terraform module is updated to look like this now:

  image  = "ubuntu-20-04-x64"
  name   = "www"
  region = "sfo3"
  size   = "s-1vcpu-1gb"

  ssh_keys  = var.ssh_keys
  user_data = templatefile("${path.module}/user_data.sh", {
    domain_name = var.domain_name
    volume_name = digitalocean_volume.www_data.name
  })

Nice! Now I just have to run terraform apply and then deploy the site one last time since the old droplet with the site on it was blown away.

This time, the deploy will deposit the files into the block volume since I mounted it at the same place that the deploy script is used to copying files to.

Now I can happily tear down and recreate the droplet, and I won’t have to run the deploy to refurnish it with the site data every time.

The finishing touch, of course, is DNS. I can easily point codeofconnor.com at the correct droplet, no matter how many times I destroy and recreate it, like so:

resource "digitalocean_domain" "a_record" {
  name       = var.domain_name
  ip_address = digitalocean_droplet.www.ipv4_address
}

Now, this basic configuration does result in downtime if the base droplet is destroyed. At least, until DNS propagates to point at the new droplet.

If the day ever comes that I want a zero-downtime redeploy of the base droplet, I’ll be sure to write about it here!

Now anything can happen to my infrastructure, and I can be back up and running with exactly these steps:

1. Make sure I have a valid DigitalOcean API token
2. terraform apply
3. make deploy
4. Wait for DNS to propagate