Estimate project
← All posts

Getting Developer Experience right: common obstacles

A look at several of the most frequent obstacles that seriously undermine a good developer experience, what they lead to, and how they can be mitigated.

8 min read
Software DevelopmentDevelopers ExperienceDeveloper Productivity

The previous part of this series named developer experience (DX) as one of the central notions in a modern software development project. In the author’s words, it covers:

(…) everything to do with how easy it is to work with software and to build it, and how productive and efficient we are while doing so. It spans development tools and languages, documentation, workflows and automation, all the way to deployments and to running and maintaining the software in production (…)

This time, let’s look at lessons learned the hard way. Below I list several of the most frequent obstacles that seriously undermine a good developer experience, and we’ll go over what they lead to and how they might be mitigated.

Difficult onboarding

Bringing a new engineer or a whole team of engineers into an existing project always takes a while. Newcomers have to get to know the architecture of the services, the infrastructure, the coding conventions and how the teams are organized. Needing several days to get up to speed is normal. It turns into a problem, though, when it emerges that the existing environment isn’t prepared to take on new developers – for any number of reasons.

I came across a project where the purely technical onboarding process was documented after a fashion, but the instructions were badly out of date and needed many fixes. Setting up a working development environment required a great deal of help from the small SRE team, chiefly because the instructions either referred to tools no longer in use or leaned on quite a few “hacky” workarounds and solutions intended to hide part of the heavy infrastructure, which in practice proved awkward to maintain over time. One painful example: spinning up the (heavily overengineered) development environment took about 40 (forty!) minutes on average – for every developer, every morning.

I saw that as more and more people joined the project over time, you might have expected those arriving later to have an easier ride. And indeed, most onboarding issues were fixed thanks to what earlier joiners had discovered – yet fresh problems kept cropping up. For instance, the company ran out of licenses for an essential development tool, or the SREs turned into a bottleneck, since handling daily infrastructure maintenance on top of newcomers’ needs proved a bit too much for a fairly small team. In the most extreme case, an engineer still lacked a 100% fully working development environment after a month of trials & errors.

Newcomers took an unusually long time on average to become productive. The company plainly wasn’t prepared for fast growth; it seemed as if almost nothing had been learned since the first “batch” of new engineers came on board. Had more attention gone into the onboarding experience, the documentation and best practices, the process might have gone more smoothly. Far too much of the load also appears to have landed on the tiny SRE team (just two engineers). Keeping the platform and the development & production environments running while also dealing with the overly complex quirks of onboarding newcomers can easily clog the daily backlog; this is exactly where technical management should step up and try to ease the problem. There are many options, such as assigning some experienced developers to help or having them tackle the current problems in the ongoing onboarding process directly.

Some engineers grew frustrated quickly. Others felt bad about not being as productive as was expected of them. Meanwhile the company’s costs kept climbing while revenue stayed low.

Knowledge silos

“Tribal knowledge” refers to a situation in which a large share of technical know-how sits with a “tribe” – usually the team’s most experienced members – while remaining barely accessible to anyone outside it. It’s closely linked to another term you may have come across: “knowledge silos”.

Once more, it’s only natural that the “elders” know far more about the architecture, the code and the challenges than newcomers do; still, good communication and leadership are what make success possible. It is especially important to spread the availability of the most experienced leaders evenly across teams – whenever someone has questions or doubts, the people with the best knowledge should be reachable and willing to answer. Dropping a group of newcomers onto a problem without context is a recipe for disaster, or at the very least for weak results. The same applies to established patterns (if there are any) – open & clear communication between teams (made up of members with differing levels of experience) is the only way to ensure that approaches to solving software problems don’t drift far apart over time.

The formation of knowledge silos should be actively countered – they tend to emerge naturally over time because of the boundaries between groups focused on their day-to-day work. Teams can begin with knowledge-sharing activities, such as internal presentations (e. g. “how we solved X, Y or Z”) and Q&A/AMA sessions with tech leads, but also by giving engineers new challenges or having them explore new technological ground from time to time.

Teams evolve. People come and go, and their knowledge & experience go with them. Organizations should ensure that as much of their know-how as possible stays in-house and is available to others in a variety of forms – documentation, recordings, a clear architecture and self-explanatory, maintainable code.

Lack of common standards

The previous paragraph brings us to the problem of missing common standards. This mainly concerns
the agreed tech stack, templates/scaffoldings for new services & delivery processes. Technologies are constantly changing & improving, but it is at least sensible to maintain a recommended set of libraries and patterns for any new service in the organization.

Picking yet another variant of, e.g., an HTTP framework for one more node.js-based microservice without a clear reason hardly sounds like a considered decision; likewise, starting anything (as long as it resembles existing parts of the infrastructure) without a well-known template – at least at company level – leaves room for improvement.

And there’s more: unified delivery strategies, centralized documentation, shared reporting & metrics tools, mature standards for runbooks – every one of these feeds into a feeling of good developer experience. They simply make team collaboration easier, more predictable and more maintainable over time.

In my view, keeping established technical standards and rules for an organization’s development amounts to having a clear strategy. Sure, the general direction has to and will shift over time – it’s never carved in stone. But it is one of the main ways of avoiding unexpected costs – whether in time, frustration or money.

Security chaos

Let’s not forget security! Chaos at the level of platform standards often goes hand in hand with a careless approach to safety. Without strict rules on sharing secrets, people quite often end up swapping private keys, passwords and other sensitive data through ordinary communication channels; they simply stop thinking about it, assuming nobody really cares.

I once encountered a developer who wanted to debug the production environment “quickly” by sprinkling in a handful of console.log(...) statements. Guess what – those logs would have printed the values of live authentication tokens… he came that close to a serious leak! On top of that, even the most seasoned engineers hardly bothered with centralized password managers or AT LEAST self-expiring one-time messages – secrets were passed around casually on Slack, and not just in private messages, but often in public threads!

I’m quite convinced the platform should strictly enforce security standards. This can take many forms – from vaults & secure channels for sharing secrets, through company-wide password managers and automated code audits, down to regular penetration tests. Believe me – security is an area where it’s almost always better to be “safe than sorry”. Having to rotate every private key because someone carelessly leaked a secret in a commit certainly doesn’t make anyone happy about developer experience!

A culture focused on security is one of many signs of a mature approach to platform engineering. Remove it, and the organization’s overall developer experience will suffer considerably.

Summary - frustration over poor DX

Frustration with poor developer experience builds up in teams over time. At worst, people may become reluctant to even attempt improvements for fear of making things worse than they were. That is obviously unhealthy, and it quite often leads talented engineers to start leaving the team, doubting that any positive change will ever come.

Once again, this shows why DX is a key driver of success. Ideally, it gets attention from day one; but even when a team grew up in a startup-like culture and had barely any time beyond rapidly prototyping a new, revolutionary product, investing in good developer experience and essential platform engineering eventually looks unavoidable. Simply too much is on the line.

Teams that operate blindly, lacking the backing of well-established standards across every area of everyday software development and maintenance, are heading straight for a massive wall. I’ve watched companies & projects fail this way – and those were rather bitter lessons. Still, it’s vital to share such experiences, pinpoint root causes and do everything possible to prevent failures in the future.