Getting Developer Experience right – lessons from a service-based project
This series looks at several of our recent projects through the lens of developer experience, pointing out what went well, what didn’t, and where there is room for improvement.
As a software delivery partner for our clients, our team naturally get to work on lots of projects of different sizes and in various business domains. Every one of them differs in its organization, ways of working, technologies and tools, coding practices and the developer experience (DX) it provides. At times we join mature projects that already have great developer experience built in; at other times we start from scratch or join projects at an early stage, where building good developer experience together with the client is one of our goals. Last but not least, we are sometimes hired specifically to help evaluate and improve the DX side of existing projects, raising the level of our clients’ “platforms”.
In this new series, we want to describe several projects we have recently worked on from the point of view of their developer experience and so-called platform engineering – how it was approached, what was great, where it fell short, and what we believed could be improved and how.
What is developer experience?
Developer experience is a fairly new “umbrella term” covering everything related to how easy it is to work with and build software, as well as how productive and efficient one is while doing it. It spans everything from development tools and languages, through documentation, workflows and automation, to deployment, running and maintaining software in production.
It is sometimes discussed together with platform engineering, since the latter simply lays the groundwork for developer experience, concentrating on internal development platforms, required and custom tooling and automation, observability infrastructure, handling security and compliance at the infrastructure level, etc.
For the first project we want to present, let’s walk through a few areas of its Developer Experience and look at how they were tackled and built.
Infrastructure provisioning and change management
The project has separate production (oh, well), staging and acceptance environments. Staging serves for tests during development, feedback and integration, while acceptance is used to test the whole platform automatically, as far as possible.
Developers don’t need to deal with the low-level parts of the infrastructure, because a dedicated SRE team, supported by the Platform Team, manages and runs all of it. The project is built on a service (NOT micro-service) architecture, so we are not talking about hundreds or thousands of microservices, with new ones appearing daily and constantly needing resources and infrastructure to be provisioned. As a result, requests to set up and provision new infrastructure are rare, and they are handled through ordinary change-management-style tickets raised with the relevant team.
Custom, shared services, and code
Like in any project, there are shared libraries, models and abstractions that unify how tools are used, etc., typically in the shape of “commons” or “utils”. We may joke about them, but in reality offering a common, standardized way of using tools shared between services, and supporting patterns while making them easier to apply, greatly improves developer experience.
Developers don’t need to spend time setting up, configuring and writing their own abstractions over existing services and libraries (such as Kafka, config etc.); rather, they use the ones the platform team maintains and keeps extending with new solutions. Over time, this reduces the project’s cognitive load and makes services more uniform, so switching between developing different services is easier.
Something that could have been done better, however, was putting “shared” models into this common artifact/library. It speeds development up considerably at the beginning, but it can become a problem once a certain level of complexity is reached. Since most services share some domain models through this custom library (think: eg. Money etc.), introducing changes or new variants of them for just a few services can strongly affect services that have nothing to do with it. This can of course be dealt with, but every option has its pros and cons, and they need to be weighed carefully.
Delivery process
Service-based architectures, backed by dedicated teams, a strong engineering culture and great platform / infrastructure support, are what makes fast value delivery possible. And I have to say it is a real joy to work like this, without waiting for big-bang releases every quarter or so, and being able to ship changes even several times a day when needed.
Continuous integration pipelines that build service images, version them with git tags and notify the authors of changes once finished remove all the heavy lifting from developers. It isn’t “continuous delivery” yet – an engineer still has the final word, deciding when to tag a new deployment and press the “Deploy” button – but the experience is pretty smooth, thanks to the SRE and Platform teams, who provide tools that semi-automate this on the way to full continuous deployment.
Of course, deploying often doesn’t have to mean releasing often, particularly when functionality is unfinished. Feature flags (a.k.a. feature toggles) come to the rescue! They are used extensively to hide features still under development from end users, while letting them be deployed to production piece by piece.
Another excellent thing is the widespread use of progressive rollout – you first make your shiny new feature available to internal users, then to a portion of users outside the company, and continue rolling it out in batches while watching the adoption process and the service metrics.
Getting service development started
On the subject of metrics, every service comes with built-in reporting of a basic set of metrics from the JVM and the container itself (CPU, memory, GC, threads etc.). On top of that, the shared library mentioned earlier builds metrics and tracing into many of the abstractions it offers, so getting all the vital signals and traces of the application into monitoring dashboards is basically a no-brainer (gRPC endpoint stats, Kafka topic monitoring, Akka cluster state, database pools and calls etc.). With this setup, adding new service-specific metrics takes only a few lines of code.
So how does a new service become ready for development? As mentioned, because services are more coarse-grained, a new one isn’t needed very often, so there is no automated “provisioning” doing all the heavy lifting. Rather, since the tech stack is largely unified, there is simply a minimal blueprint service repository, with all of the above batteries included, which the developer copies and then builds upon.
This also means one has to fine-tune all the deployment-related artifacts (helm charts etc.), which isn’t the most enjoyable job, but it is mostly a one-off. It could certainly be fully automated, e.g. with a web form or a CLI tool, but when you weigh the investment and maintenance against the actual needs and how often it would be used, it isn’t the top priority on the roadmap.
Running and maintaining services
Once a service is live and handling users’ requests in production, we want to be told whenever something goes wrong – bugs that slipped through testing, edge cases nobody covered, configuration or integration problems etc.
To make this possible, every service and team has its own alerting configuration that sends notifications to the corresponding Slack channels. These include alerts raised by the service code itself – where, again, the shared library offers a toolset for raising alarms programmatically with ease – as well as alerts coming from the platform (memory limits exceeded, restart loops etc.). For troubleshooting, all the important pieces are integrated: logs, traces and metrics, which makes collecting data for analysis simpler.
Besides automated alerting, there are Slack channels to which important issues noticed by staff, or customer support requests, are routed for a fast response. A set of Slack workflows has been built around these requests, so that ownership and progress are clearly communicated.
Knowledge base
As far as running in production goes, there is not yet a dedicated, full-blown service catalog such as Spotify’s Backstage etc., although this area is being researched. In its place, the engineering wiki has an organized space that serves as a service catalog.
Every service has its own page with the key information: a brief description of what it does, which tools it uses, which team and manager own it, and how to handle some incidents that may occur but haven’t been properly fixed yet, a.k.a. runbooks. Every runbook includes a clear set of instructions (with the required commands, screenshots etc.) that any engineer who takes on the issue should be able to follow, and, in the worst case, escalation channels for when the procedure doesn’t work or fails.
For the big picture of the system, there is a place that holds most of the architecture and integrations as C4 architecture diagrams, the de-facto standard for describing such structures. It allows quick navigation to further technical details about the services and their surroundings.
Sadly, both of these (runbooks and diagrams) have to be updated manually after every change, which is why the research into automation tools mentioned earlier is under way. The market offers many tools and “platforms for building platforms”, and finding one that isn’t too complicated yet is a big enough improvement to justify the investment may take a while.
SRE and Platform teams
Last but not least, a few words about the people behind some of these improvements. A separate team may not be necessary at the very start, but as the project gains momentum and inevitably becomes more and more complex, they become indispensable. To quote the “Team Topologies” classification:
“Platform team: a group made up of other team types that delivers a compelling internal product, speeding up delivery by Stream-aligned teams”
Their role is to ensure that the teams delivering so-called “business value” can work at full speed. They look after the infrastructure, provisioning and maintenance, supply and build custom tools when necessary, track updates of core dependencies across services, and automate whatever is worth automating, so that tasks take less time and there is less room for human error.
Because of them, most tooling is standardized and known to engineers in every service team, rather than being a pile of home-grown team scripts or quickly hacked tools that nobody maintains in the long run. That said, it doesn’t mean ordinary developers can’t contribute to the tooling and the “platform” itself. In fact, a large part of the tooling grew out of issues that teams solved internally. Communication between the platform team and the other teams is essential here to avoid misunderstandings and building or standardizing the wrong things – in the end, all of it serves to improve developer experience.
Summary
Good developer experience isn’t built in a single shot; it is an ongoing process. It isn’t a matter of rolling out a tool that will fix every problem for us. It takes time, experience and knowledge, and usually dedicated, skilled people who know what to build and how, in order to empower stream-aligned teams and serve them well.
In the project described here, the Platform team emerged and keeps growing organically, driven by the actual needs of the other teams rather than by the idea that having a Platform Team is trendy now because all the big names have one. With this approach, only the tools and improvements that are truly worthwhile get delivered.
These improvements may not be immediate, but in the long run they will tackle the most troublesome aspects of day-to-day development, making feature delivery more enjoyable and straightforward.