Estimate project
← All posts

Moving the Mastodon Helm Chart over to Bitnami

This article walks you through switching your Mastodon Helm Chart deployment to the chart packaged by Bitnami.

7 min read
DevOpsKubernetesgitopsmastodontwittersocial mediaCloud

Some months ago, we ran Mastodon on Kubernetes with the official Mastodon Helm Chart. Since then, the Mastodon community has moved their Helm Chart to another repository. Sadly, that Helm Chart is poorly maintained, and our attempt to use it ran into several blockers. As a result, we decided to go with the Mastodon Helm Chart from Bitnami, a more dependable and better-maintained option.

Bitnami provides a large collection of Helm Charts for well-known applications. Many of them are updated regularly, yet working with the Mastodon Helm Chart still posed some challenges for us. While migrating, we ran into a number of problems, some of them caused by templates that were defined incorrectly.

You may ask whether moving the Helm Chart to another, better-maintained repository is really needed. It is. Mastodon recently disclosed several vulnerabilities that affect older versions. Unfortunately, Mastodon’s official Helm Chart does not include these critical security patches. Luckily, the Bitnami team does what is needed to keep their Helm Chart current.

So, below is a tutorial on moving from the Mastodon Helm Chart to the one packaged by Bitnami. Let’s get started!

Get the Mastodon instance ready for the migration

If you read my earlier article on deploying Mastodon on Kubernetes, you may recall that I deploy the Helm Chart with Flux CD, following the GitOps approach. For this migration, we want to change our current configuration as little as possible. These are the steps we need:

  • Update the HelmRelease and tweak the Helm Chart values.
  • Update the HelmRepository.
  • Create a new secret for Redis.

Be aware that the migration will cause some temporary downtime. Prepare your values.yaml file with that in mind. Below, I go through a few sections that can cause trouble. You can find the Mastodon Helm Chart in the Bitnami repository under this link.

Apache Helm Chart and Ingress

Ingress requires the Apache Helm Chart, a dependency, to be enabled. Note that, as of October 25th, Mastodon Helm Chart version 2.1.0 depends on the Apache Helm Chart in version 9.x.x.

Setting the values for the Apache Helm Chart was somewhat tricky too – version 9.6.5 marks ingress.certManager as deprecated, but the Helm Chart templates show that this value is still needed to enable TLS in ingress. There was also an issue with the Apache ConfigMap, though thankfully it got merged and closed. Do I even need to say that the migration was rather challenging?

The Apache Helm Chart 10.0.0 release reworked the entire Ingress logic. Bear in mind that if you follow this article later on, you might have to adapt the Ingress configuration described here.

initJob

The init job runs the database and elasticsearch migrations, verifies the connection to S3, and precompiles the assets.

The initJob.createAdmin flag has to be set to false – if it isn’t, the init job cannot complete, and its completion is required for the Mastodon pods to start.

WebSocket connection

useSecureWebSocket has to be set to true – we hit a problem where the website could not open a WebSocket connection to streaming. The issue remains open here.

Storage

We use Google Cloud Storage for storage. During the migration, the S3 bucket attempted to use the http protocol, even though externalS3.protocol had the value https. On top of that, after Mastodon started, the pods were unable to find the right bucket name, which is why the configuration below was needed.

forceHttpsS3Protocol: true
s3AliasHost: storage.googleapis.com/<bucket-name>

Enable metrics

Further on in this tutorial, I’ll also show the configuration needed to gather metrics with Prometheus Statsd Exporter and display them on a Grafana dashboard. Put the name of your Prometheus Statsd Exporter service here.

extraConfig:
  STATSD_ADDR: <name of statds exporter service>.<namespace>:9125

Redis secret

At the moment, the Mastodon Helm Chart logic lets you point to existing Redis secrets only when you use an external Redis. When Redis is used as a dependency, the Helm Chart templates make the secret name identical to the Redis Helm Chart release name.

Move your current password into a new secret called <your-helm-release>-redis, for instance mastodon-bitnami-redis.

Edit your Helm Chart values

Once everything is in place, your values file should look like the one below:

# Default values for the Helm Chart are from: https://github.com/bitnami/charts/blob/main/bitnami/mastodon/values.yaml

adminUser: admin
adminEmail: admin@email.com
existingSecret: existing-secret-name
extraConfig:
  STATSD_ADDR: <name of statds exporter service>.<namespace>:9125

localDomain: your.domain
useSecureWebSocket: true

enableS3: true
forceHttpsS3Protocol: true
s3AliasHost: storage.googleapis.com/your-bucket-name
externalS3:
  host: storage.googleapis.com
  existingSecret: existing-secret-name
  protocol: https
  bucket: your-bucket-name
  region: region-name
  existingSecretAccessKeyIDKey: AWS_ACCESS_KEY_ID
  existingSecretKeySecretKey: AWS_SECRET_ACCESS_KEY

smtp:
  server: smtp-server
  from_address: your-smtp-email@com
  domain: your.domain
  delivery_method: smtp
  openssl_verify_mode: peer
  enable_starttls_auto: true
  auth_method: plain
  existingSecret: existing-secret-name

web:
  replicaCount: 1
  resources:
    limits:
      cpu: 1
      memory: 1280Mi
    requests:
      cpu: 250m
      memory: 768Mi

sidekiq:
  replicaCount: 1
  resources:
    limits:
      cpu: 1
      memory: 768Mi
    requests:
      cpu: 250m
      memory: 512Mi

streaming:
  replicaCount: 1
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 250m
      memory: 128Mi

initJob:
  createAdmin: false

externalDatabase:
  user: mastodon
  database: mastodon
  existingSecret: existing-postgres-secret-name
  existingSecretPasswordKey: password

redis:
  enabled: true
  architecture: standalone
  existingSecret: mastodon-bitnami-redis
  master:
    persistence:
      size: 2Gi
    resources:
      requests:
        cpu: 100m
        memory: 1Gi
      limits:
        cpu: 1
        memory: 2Gi
  auth:
    existingSecret: mastodon-bitnami-redis

postgresql:
  enabled: false

minio:
  enabled: false

elasticsearch:
  master:
    masterOnly: false
    replicaCount: 1
    resources:
      requests:
        cpu: 100m
        memory: 1Gi
      limits:
        cpu: 1
        memory: 2Gi
    heapSize: 250m
    persistence:
      size: 4Gi
  coordinating:
    replicaCount: 0
  data:
    replicaCount: 0
  ingest:
    replicaCount: 0

apache:
  enabled: true
  ingress:
    enabled: true
    hostname: your.domain
    path: /
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-issuer
    tls:
      - hosts:
          - your.domain
        secretName: your.domain-tls
    certManager: true

Keep in mind that the resource requests and limits are only examples – they suit my setup, but you may need to tune them.

Set up the HelmRepository:

Change your HelmRepository file:

kind: HelmRepository
metadata:
  name: mastodon
  namespace: default
spec:
  type: oci
  interval: 5m
  url: oci://registry-1.docker.io/bitnamicharts

Set up the HelmRelease:

Change your HelmRelease file:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: mastodon-bitnami
  namespace: default
spec:
  interval: 5m
  chart:
    spec:
      chart: mastodon
      version: 3.0.5
      sourceRef:
        kind: HelmRepository
        name: mastodon
      interval: 5m
  valuesFrom:
    - kind: ConfigMap
      name: mastodon-values
    - kind: ConfigMap
      name: pgsql-14-configuration
      valuesKey: DB_HOST
      targetPath: externalDatabase.host
    - kind: ConfigMap
      name: pgsql-14-configuration
      valuesKey: DB_PORT
      targetPath: externalDatabase.port

Notice that I don’t put the values straight into the HelmRelease file – I reference a ConfigMap generated with kustomize instead. The relevant Flux CD documentation is available here.

Run the migration

Create a new HelmRelease under a different name, e.g. mastodon-bitnami. This matters – simply updating the existing Mastodon HelmRelease won’t work, a separate one is required. The InitJob described earlier has post-install and pre-upgrade hooks. The pre-upgrade hook means the InitJob runs before any resources get updated. As a result, changes made to existing configmaps would be ignored. The InitJob would keep using stale vars and fail over and over.

Suspend the old Mastodon HelmRelease. Scale the old Mastodon deployments down to 0 replicas – the idea is to shut the server down while keeping the current configuration and Redis setup, just to be safe..

Add the new HelmRelease and watch the initJob. Once it succeeds, the Streaming, Web, and Sidekiq pods should come up together with Redis, Elastichsearch, and Apache. Check your ingress – by this point cert-manager should have issued the new certificate.

Monitoring Mastodon with Prometheus and Grafana

Mastodon exposes StatsD metrics. Before Prometheus can scrape them and Grafana can display them, they need to be converted. The Prometheus Community’s Statsd-exporter turns StatsD metrics into Prometheus metrics according to configured mapping rules.

Install the Statsd-exporter Helm Chart in your monitoring namespace.

Add the HelmRepository:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: prometheus-community
  namespace: flux-system
spec:
  interval: 5m0s
  url: https://prometheus-community.github.io/helm-charts

Add the HelmRelease:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: prometheus-statsd-exporter
spec:
  chart:
    spec:
      chart: prometheus-statsd-exporter
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
        namespace: flux-system
      version: 0.9.1
  interval: 5m0s
  releaseName: prometheus-statsd-exporter
  valuesFrom:
    - kind: ConfigMap
      name: prometheus-statsd-exporter-values
      valuesKey: prometheus-statsd-exporter-values.yaml

Reference the values from a ConfigMap generated with Kustomize – that way our HelmRelease stays neat and tidy.

Converting metrics from the Statsd format into one Prometheus can read requires mappings. I came across a really helpful article about monitoring Mastodon. Its author [publishes his mapping configuration](https://ipng.ch/s/articles/2022/11/27/mastodon-3.html#:~:text=The Statsd Exporter mapping configuration file: [/etc/prometheus/statsd-mapping.yaml]) along with a Grafana Dashboard that does the job perfectly.

Put the mappings into values.yaml:

serviceMonitor:
  enabled: true
  namespace: monitoring
  endpoint: "/metrics"
statsd:
  mappingConfig: |-
    # add here mappings

Remember to put the name of your Prometheus Statsd exporter service into the Mastodon Helm Chart values, under the STATS_ADDR env. Import your dashboard into Grafana, and you should be able to gather statistics and keep an eye on your instance.

Summary

At this point, your Mastodon instance should be successfully migrated with minimal downtime. This Helm Chart may still need a few pull requests and fixes, but it is comforting that the Bitnami Github repository is well-maintained and updated regularly. Have fun running your Mastodon server without hiccups!