Estimate project
← All posts

Your Whole System as Code – getting started with GitOps

These days we want to express everything as code – but can an entire system be described that way? It can! Find out what GitOps is and how it can simplify your work.

8 min read
DevOpsGitSystem as a Codegitops

These days we tend to want everything expressed as code, and with good reason – infrastructure as code brings many benefits: consistency, modularity, protection against human mistakes, version control and more. The same can be said about the other “things as code” – configuration as code, pipelines as code and so on.

Which naturally raises a question: can the entire system be kept as code?

The short answer: yes, it can! With Kubernetes and the GitOps approach, the desired state of a whole system can be described declaratively. In this brief article I’ll try to explain what GitOps really is and how it can simplify your work.

What is GitOps?

Usually, when a question like this comes up, the way to answer it is obvious: search the web and look at a handful of answers:

GitOps delivers infrastructure as code by treating Git repositories as the single source of truth.

Simple enough. Put the terraform code in a git repository and you’re done.

GitOps is an operational framework that brings DevOps best practices from application development – version control, collaboration, compliance and CI/CD tooling – to infrastructure automation.

Quite similar, except now CI/CD is required.

GitOps holds to the principle that Git is the sole source of truth.

This matters a lot. Anyone who has ever maintained documentation knows that the second law of thermodynamics is real and the entropy of the Universe keeps growing.

Developers already keep application source code in Git – GitOps carries that practice over to an application’s configuration, infrastructure and operational procedures.

Now it gets more interesting – keeping configuration and infrastructure (as code) in git is easy to picture, but… operational procedures?

So is there no single definition of GitOps at all? In fact, there is. A while ago several big players such as GitHub, Azure and Amazon (among others) set up a GitOps Working Group under the Cloud Native Computing Foundation. The result of that WG’s work was the announcement of the OpenGitops 1.0 principles.

Open GitOps 1.0

A system managed with GitOps must have a desired state that is:

  1. Declarative
    The desired state of a GitOps-managed system has to be expressed in a declarative way.
  2. Versioned and Immutable
    The desired state is kept in storage that enforces immutability and versioning and preserves the full version history.
  3. Pulled Automatically
    Software agents fetch the desired state declarations from the source on their own.
  4. Continuously Reconciled
    Software agents keep watching the actual state of the system and try to bring it to the desired state.

Clearly, OpenGitOps goes far beyond keeping infrastructure code in git.

Let’s take a closer look at each principle.

Declarative

A declarative description of the system decouples the desired state from how it is reached – the commands, API calls, scripts and so on. In essence, we state facts about the system rather than write procedures.

Versioned and Immutable

The desired state has to live in a system that enforces immutability. In other words, once a state has been created, it cannot be modified. To change the state of the system, you create a new version, and the previous one stays in the history. The “git” in GitOps obviously comes from keeping the system state in a git repository. Under this principle, however, any system meeting the requirement will do – an S3 bucket with versioning turned on, for instance.

Most of the time a git repository is the obvious choice – commits are immutable, and each new commit represents a new version.

Pulled automatically

This one is particular to Open GitOps. Most GitOps definitions only talk about storing the desired state. This principle demands “something” – software agents running on the managed system – that actively watches the storage and fetches the newest version. As a consequence, once such a system is in place, the commit is the last step the operator has to care about. Applying the state to the running system is no longer his job.

Continuously Reconciled

Just like with the previous principle, nobody – meaning no human – has to think about when and how the desired state reaches the running system. Software takes care of it automatically. It resembles how a Kubernetes cluster behaves – applying a pod manifest does not guarantee that the pod will run, since that depends on factors such as available resources, node selector, affinity and so on. Still, the system will try as hard as it can to satisfy what the operator asked for.

Do I want to use GitOps?

TL;DR – yes, you do ;-).
Let’s go through some benefits of this approach and see how it can make a developer’s work easier.

Git as a single source of the truth

Anyone with some IT experience knows that up-to-date documentation doesn’t exist. Well, it does, but a few New York seconds later it is already stale (the New York second is the shortest measurable unit of time, defined as the interval between the light turning green and the taxi behind you honking).

GitOps fixes this – the desired state kept in the git repository is not just documentation of the system, it is the state of the system itself! One could probably fill a book with the benefits of a single source of truth, but most of them are self-evident.

Git as a single tool to manage the system.

Because the desired state is pulled and applied to the running system automatically, all a developer has to do is commit and push changes to the git repository. No new tools to learn – and as a developer you already know git, right?

The history of the system

It is usually crucial to know not only the current state of the system but also what it looked like before – its history in general. This matters most when something breaks and the operator has to find out what changed. In short, the git repository gives you complete auditability of the system.

(Relatively) easy rollback

An important consequence of the above: what do you do when something really does go wrong? It’s code, after all. And how do developers bring code back to an earlier state? With git revert, of course.

Usually this works for the whole system as well! Unless, say, a database migration was involved – but for stateless services it works with no trouble at all.

PRs and code reviews

We get exactly the same tools as every other developer working with code. We can open PRs, request code reviews, run automated CI tests to catch obvious syntax errors and so on. This considerably improves the reliability and security of the managed system.

Disaster recovery

What happens if a genuine disaster hits the data center?

The data is, naturally, the most valuable part of the system, which is why a dependable backup policy matters so much – keeping backups in several geographically distant locations and so on.

But provided the data is backed up, rebuilding the entire system somewhere else takes minutes. Exactly the same services, in exactly the same versions.

Possible problems

GitOps obviously isn’t a cure for every problem in the world. Let’s look at some potential pitfalls of GitOps (and ways to deal with them).

Multiple environments

Having several environments for a system – dev, staging, uat, production and so on – is standard practice today.

How do you do that with GitOps? There is no single best practice. Some teams use several branches, others just several directories in git, and others simply several repositories. Every approach has advantages and drawbacks, so some experimentation is worthwhile. When combining GitOps with Kubernetes, it pays to use Customization – it helps keep the code DRY.

Secrets

GitOps calls for keeping everything in the git repository, yet – as most people know – storing secrets (credentials) in git encrypted is a very bad idea, even in a private repository.

There are a few ways around this. Some tools – such as Bitnami SealedScrets or KOPS – let you keep the secrets encrypted in git and decrypt them automatically on the cluster. Others simply keep secrets out of git and only reference secrets stored somewhere else – the External Secrets operator being the key example.

Easy for technical stuff, difficult for business

Everyone knows and uses git. Well, every technical person does. When someone from the business side asks the usual question “which version and which features are on production right now?”, the reply “you can look it up on GitHub” probably won’t satisfy them.

That pushes us to build extra tools that “translate” the git repository into something non-technical people can read easily.

Implementations of the OpenGitops 1.0

The two most important tools implementing the Open GitOps principles are probably FluxCD and ArgoCD (there may be others, but these two are likely the most popular).

Which one is better? Which should you pick? As with every such question, there is only one answer: it depends. Both come with excellent documentation and quick-start guides that make it easy to get to know them.

Plenty of articles and videos compare the two – I’d encourage you to check them out.

Personally I lean towards FluxCD, which by no means makes it better than ArgoCD. To me ArgoCD feels somewhat simpler, so its learning curve is gentler.

FluxCD, on the other hand, comes with multiple controllers and multiple CRDs, which – in my very humble opinion – map more closely to the objects it manages.

ArgoCD’s killer feature is its graphical user interface, which is probably why developers go for it.

Wrapping Up

If your project doesn’t use GitOps yet, I honestly think it should! It makes life much simpler, particularly in combination with Kubernetes. The bigger the project, the harder it gets to keep all its parts under control. You can start small, with just a part of the system, and see whether it suits you. I’m fairly confident you’ll love it 🙂.