Moving the Mastodon Helm Chart over to Bitnami
This article walks you through switching your Mastodon Helm Chart deployment to the chart packaged by Bitnami.
Some months ago, we ran Mastodon on Kubernetes with the official Mastodon Helm Chart. Since then, the Mastodon community has moved their Helm Chart to another repository. Sadly, that Helm Chart is poorly maintained, and our attempt to use it ran into several blockers. As a result, we decided to go with the Mastodon Helm Chart from Bitnami, a more dependable and better-maintained option.
Bitnami provides a large collection of Helm Charts for well-known applications. Many of them are updated regularly, yet working with the Mastodon Helm Chart still posed some challenges for us. While migrating, we ran into a number of problems, some of them caused by templates that were defined incorrectly.
You may ask whether moving the Helm Chart to another, better-maintained repository is really needed. It is. Mastodon recently disclosed several vulnerabilities that affect older versions. Unfortunately, Mastodon’s official Helm Chart does not include these critical security patches. Luckily, the Bitnami team does what is needed to keep their Helm Chart current.
So, below is a tutorial on moving from the Mastodon Helm Chart to the one packaged by Bitnami. Let’s get started!
Get the Mastodon instance ready for the migration
If you read my earlier article on deploying Mastodon on Kubernetes, you may recall that I deploy the Helm Chart with Flux CD, following the GitOps approach. For this migration, we want to change our current configuration as little as possible. These are the steps we need:
- Update the HelmRelease and tweak the Helm Chart values.
- Update the HelmRepository.
- Create a new secret for Redis.
Be aware that the migration will cause some temporary downtime. Prepare your values.yaml file with that in mind. Below, I go through a few sections that can cause trouble. You can find the Mastodon Helm Chart in the Bitnami repository under this link.
Apache Helm Chart and Ingress
Ingress requires the Apache Helm Chart, a dependency, to be enabled. Note that, as of October 25th, Mastodon Helm Chart version 2.1.0 depends on the Apache Helm Chart in version 9.x.x.
Setting the values for the Apache Helm Chart was somewhat tricky too – version 9.6.5 marks ingress.certManager as deprecated, but the Helm Chart templates show that this value is still needed to enable TLS in ingress. There was also an issue with the Apache ConfigMap, though thankfully it got merged and closed. Do I even need to say that the migration was rather challenging?
The Apache Helm Chart 10.0.0 release reworked the entire Ingress logic. Bear in mind that if you follow this article later on, you might have to adapt the Ingress configuration described here.
initJob
The init job runs the database and elasticsearch migrations, verifies the connection to S3, and precompiles the assets.
The initJob.createAdmin flag has to be set to false – if it isn’t, the init job cannot complete, and its completion is required for the Mastodon pods to start.
WebSocket connection
useSecureWebSocket has to be set to true – we hit a problem where the website could not open a WebSocket connection to streaming. The issue remains open here.
Storage
We use Google Cloud Storage for storage. During the migration, the S3 bucket attempted to use the http protocol, even though externalS3.protocol had the value https. On top of that, after Mastodon started, the pods were unable to find the right bucket name, which is why the configuration below was needed.
forceHttpsS3Protocol: true
s3AliasHost: storage.googleapis.com/<bucket-name>
Enable metrics
Further on in this tutorial, I’ll also show the configuration needed to gather metrics with Prometheus Statsd Exporter and display them on a Grafana dashboard. Put the name of your Prometheus Statsd Exporter service here.
extraConfig:
STATSD_ADDR: <name of statds exporter service>.<namespace>:9125
Redis secret
At the moment, the Mastodon Helm Chart logic lets you point to existing Redis secrets only when you use an external Redis. When Redis is used as a dependency, the Helm Chart templates make the secret name identical to the Redis Helm Chart release name.
Move your current password into a new secret called <your-helm-release>-redis, for instance mastodon-bitnami-redis.
Edit your Helm Chart values
Once everything is in place, your values file should look like the one below:
# Default values for the Helm Chart are from: https://github.com/bitnami/charts/blob/main/bitnami/mastodon/values.yaml
adminUser: admin
adminEmail: admin@email.com
existingSecret: existing-secret-name
extraConfig:
STATSD_ADDR: <name of statds exporter service>.<namespace>:9125
localDomain: your.domain
useSecureWebSocket: true
enableS3: true
forceHttpsS3Protocol: true
s3AliasHost: storage.googleapis.com/your-bucket-name
externalS3:
host: storage.googleapis.com
existingSecret: existing-secret-name
protocol: https
bucket: your-bucket-name
region: region-name
existingSecretAccessKeyIDKey: AWS_ACCESS_KEY_ID
existingSecretKeySecretKey: AWS_SECRET_ACCESS_KEY
smtp:
server: smtp-server
from_address: your-smtp-email@com
domain: your.domain
delivery_method: smtp
openssl_verify_mode: peer
enable_starttls_auto: true
auth_method: plain
existingSecret: existing-secret-name
web:
replicaCount: 1
resources:
limits:
cpu: 1
memory: 1280Mi
requests:
cpu: 250m
memory: 768Mi
sidekiq:
replicaCount: 1
resources:
limits:
cpu: 1
memory: 768Mi
requests:
cpu: 250m
memory: 512Mi
streaming:
replicaCount: 1
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 128Mi
initJob:
createAdmin: false
externalDatabase:
user: mastodon
database: mastodon
existingSecret: existing-postgres-secret-name
existingSecretPasswordKey: password
redis:
enabled: true
architecture: standalone
existingSecret: mastodon-bitnami-redis
master:
persistence:
size: 2Gi
resources:
requests:
cpu: 100m
memory: 1Gi
limits:
cpu: 1
memory: 2Gi
auth:
existingSecret: mastodon-bitnami-redis
postgresql:
enabled: false
minio:
enabled: false
elasticsearch:
master:
masterOnly: false
replicaCount: 1
resources:
requests:
cpu: 100m
memory: 1Gi
limits:
cpu: 1
memory: 2Gi
heapSize: 250m
persistence:
size: 4Gi
coordinating:
replicaCount: 0
data:
replicaCount: 0
ingest:
replicaCount: 0
apache:
enabled: true
ingress:
enabled: true
hostname: your.domain
path: /
ingressClassName: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-issuer
tls:
- hosts:
- your.domain
secretName: your.domain-tls
certManager: true
Keep in mind that the resource requests and limits are only examples – they suit my setup, but you may need to tune them.
Set up the HelmRepository:
Change your HelmRepository file:
kind: HelmRepository
metadata:
name: mastodon
namespace: default
spec:
type: oci
interval: 5m
url: oci://registry-1.docker.io/bitnamicharts
Set up the HelmRelease:
Change your HelmRelease file:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: mastodon-bitnami
namespace: default
spec:
interval: 5m
chart:
spec:
chart: mastodon
version: 3.0.5
sourceRef:
kind: HelmRepository
name: mastodon
interval: 5m
valuesFrom:
- kind: ConfigMap
name: mastodon-values
- kind: ConfigMap
name: pgsql-14-configuration
valuesKey: DB_HOST
targetPath: externalDatabase.host
- kind: ConfigMap
name: pgsql-14-configuration
valuesKey: DB_PORT
targetPath: externalDatabase.port
Notice that I don’t put the values straight into the HelmRelease file – I reference a ConfigMap generated with kustomize instead. The relevant Flux CD documentation is available here.
Run the migration
Create a new HelmRelease under a different name, e.g. mastodon-bitnami. This matters – simply updating the existing Mastodon HelmRelease won’t work, a separate one is required. The InitJob described earlier has post-install and pre-upgrade hooks. The pre-upgrade hook means the InitJob runs before any resources get updated. As a result, changes made to existing configmaps would be ignored. The InitJob would keep using stale vars and fail over and over.
Suspend the old Mastodon HelmRelease. Scale the old Mastodon deployments down to 0 replicas – the idea is to shut the server down while keeping the current configuration and Redis setup, just to be safe..
Add the new HelmRelease and watch the initJob. Once it succeeds, the Streaming, Web, and Sidekiq pods should come up together with Redis, Elastichsearch, and Apache. Check your ingress – by this point cert-manager should have issued the new certificate.
Monitoring Mastodon with Prometheus and Grafana
Mastodon exposes StatsD metrics. Before Prometheus can scrape them and Grafana can display them, they need to be converted. The Prometheus Community’s Statsd-exporter turns StatsD metrics into Prometheus metrics according to configured mapping rules.
Install the Statsd-exporter Helm Chart in your monitoring namespace.
Add the HelmRepository:
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: prometheus-community
namespace: flux-system
spec:
interval: 5m0s
url: https://prometheus-community.github.io/helm-charts
Add the HelmRelease:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: prometheus-statsd-exporter
spec:
chart:
spec:
chart: prometheus-statsd-exporter
sourceRef:
kind: HelmRepository
name: prometheus-community
namespace: flux-system
version: 0.9.1
interval: 5m0s
releaseName: prometheus-statsd-exporter
valuesFrom:
- kind: ConfigMap
name: prometheus-statsd-exporter-values
valuesKey: prometheus-statsd-exporter-values.yaml
Reference the values from a ConfigMap generated with Kustomize – that way our HelmRelease stays neat and tidy.
Converting metrics from the Statsd format into one Prometheus can read requires mappings. I came across a really helpful article about monitoring Mastodon. Its author [publishes his mapping configuration](https://ipng.ch/s/articles/2022/11/27/mastodon-3.html#:~:text=The Statsd Exporter mapping configuration file: [/etc/prometheus/statsd-mapping.yaml]) along with a Grafana Dashboard that does the job perfectly.
Put the mappings into values.yaml:
serviceMonitor:
enabled: true
namespace: monitoring
endpoint: "/metrics"
statsd:
mappingConfig: |-
# add here mappings
Remember to put the name of your Prometheus Statsd exporter service into the Mastodon Helm Chart values, under the STATS_ADDR env. Import your dashboard into Grafana, and you should be able to gather statistics and keep an eye on your instance.
Summary
At this point, your Mastodon instance should be successfully migrated with minimal downtime. This Helm Chart may still need a few pull requests and fixes, but it is comforting that the Bitnami Github repository is well-maintained and updated regularly. Have fun running your Mastodon server without hiccups!