Hosting this blog in Kubernetes

As of about a week or so ago, this blog is now hosted in Kubernetes.

Obligatory:

Seriously though, it was a great learning experience, bringing together a number of different moving parts into an automated, repeatable deployment, using Terraform to orchestrate the whole smash. This blog runs on Ghost. My goal here was to wrap everything up into automation and separate the persistent storage from the containerised front end, so that the Ghost instance(s) could be completely destroyed and recreated without affecting the site content itself. Briefly, this comprises:

• Azure MariaDB service (for site content)
• Azure blob storage (for images and other static assets)
• Azure DNS
• Azure Kubernetes Service
• Bitnami Ghost Helm chart
• Azure Storage integration for Ghost (for storing static assets)
• Kubernetes Nginx ingress controller
• Kubernetes ExternalDNS controller (updates Azure DNS to create an A record for the site, pointing at the public IP of the Ingress Controller)
• Kubernetes cert-manager controller (for automatically requesting and renewing LetsEncrypt TLS certificates).

Ok, that wasn't so brief. Suddenly that Twitter post doesn't seem so much like a derisive dig in the ribs at k8s, more a representation of reality. It really looks like overkill doesn't it? Maybe it is, but it's also fun. The gory details of how this all hangs together is probably something that will span a few blog posts (if I ever get around to writing them), but let's begin with something nice and simple, but which originally stumped me for a good while trying to get it working...

As previously mentioned, a key goal was to ensure that there was no persistent storage required in AKS itself. Pointing the Ghost container at an external instance for its database was easy enough - the Helm chart has parameters exposed for this. Terraform creates the MariaDB instance, and injects the hostname, username and password into the module that goes ahead and sets up the Ghost database and ultimately passes the connection into into the Ghost Helm chart itself. The thing that you don't get out of the box is the storage adapters that allow you to put static content elsewhere. Without this, images that are uploaded into blog posts are stored directly on the filesystem of the application server, and therefore would be lost if the container were recreated (and would appear to be intermittently unavailable if the service were ever scaled out to multiple instances).

After some head scratching over how to automate the installation of the Azure Storage Adapter (which was solved by @FraPazGal over in this issue that I opened), I had a Ghost container with the Azure Storage Adapter included. The Dockerfile is available here and it looks like this:

FROM bitnami/ghost:latest

USER root
RUN npm install ghost-azure-storage --unsafe-perm
RUN useradd ghost
RUN echo "ghost ALL=(ALL:ALL) NOPASSWD: ALL" | tee /etc/sudoers.d/ghost
RUN cp -vR node_modules/ghost-azure-storage ./current/core/server/adapters/storage

The one thing that this Dockerfile does not do is update the config.production.json file to include the configuration options there. Doing this seemed a little pointless to me, since, in true twelve-factor app style, Ghost can pull any configuration from Environment variables, and the Ghost Helm chart can supply additional Environment variables if required. So, I chose to do it this way instead. The Azure Storage Adapter uses configuration that would look like this in JSON:

  "storage": {
    "active": "ghost-azure-storage",
    "ghost-azure-storage": {
      "connectionString": "YourConnectionStringHere",
      "container": "YourOptionalContainerName",
      "cdnUrl": "YourCDNEndpointDomain",
      "useHttps": "true"
    }
  },

To pass these in as environment variables, you need to use a double underscore as a nesting separator, so the storage/active key becomes storage__active when passed in as an environment variable. Since I'm using the Terraform Helm Provider to deploy the chart, passing these options in looks like this in Terraform:

  set {
    name  = "extraEnvVars"
    value = <<-EOT
            - name: storage__active
              value: ghost-azure-storage
            - name: storage__ghost-azure-storage__connectionString
              value: ${var.azureStorageConnectionString}
            EOT
  }

Once this was done, it was possible to completely separate the site data and assets from the app itself, meaning that I could tear down and redeploy the Ghost container as often as I wished, without affecting the site itself.

That's all for this one. There's plenty more in here to unpack, so I'll probably write another post or two on some of the other challenges that this presented and how they were solved.

See you in 2021!