What I wish I had known when starting out with Event Sourcing - part 1
This series walks you through what I have learned so far – the discoveries, realizations and surprises I came across during years of working with Event Sourcing.
During the past several years, I’ve been lucky enough to learn about and work on many services built with event sourcing. Admittedly, at times our relationship leaned more towards the “hate” than the “love” side of the love-hate pair, and at times event sourcing was used too much, or at least where it wasn’t necessary.
Looking back on all those years, I consider it a genuinely powerful, practical and clearly undervalued tool in an engineer’s toolbox. Even so, it is just one more approach, intended to help with some fairly narrow parts of our engineering work – not a silver bullet that claims to solve every problem and challenge our projects run into.
In this new series, I want to walk you through the lessons I’ve picked up so far, along with the discoveries, realizations and surprises I encountered during those years of using event sourcing in the systems I built. This article covers several of the technique’s core strengths and how I came to value them. Later on, I’ll write more about the challenges and a few gotchas I ran into. So, let’s begin!
Time-traveling superpowers
We’ll start with what is likely the most obvious and best-known property of event sourcing, and usually the main argument of anyone trying to “sell” it.
The true value of time traveling only becomes clear when you need to troubleshoot and dig into a bug that has somehow slipped into production. All of a sudden, your service goes haywire for some users: either they can’t reach a particular feature, or it shows garbage data.
You put on your detective hat and start by going through all the metrics and system health checks - everything looks fine. Next, a quick glance at the latest logs, but once more there’s nothing suspicious apart from messages saying that something is wrong with this feature for a particular user.
In “traditional” systems, where persistence is based on state snapshots, your next step is probably something like SELECT * FROM accounts WHERE user_id = 123 to see what’s in there, hoping for an answer along the lines of “Oh, the state is incorrect.” Perhaps you see the problem right away and know how to fix it with UPDATE accounts SET … WHERE user_id = 123, but what if you don’t? How do you find out why, how and when the entity ended up in this wrong state?
All you’ve got is the current state snapshot and perhaps, if you’re lucky (and a bit paranoid about logging, like me), a pile of old logs. Sometimes that isn’t enough, though; you’re working blind, and this is where the investigation becomes difficult. That doesn’t mean it can’t be done - we’ve all been in that spot, and there’s usually a way to fix it; connecting all the dots may just be somewhat harder.
If, however, your state persistence relies on event sourcing, you suddenly get far more context and data to work with. You have every fact that happened to the entity over time – the facts that caused all of its state changes – in chronological order and with every detail included.
You can then begin traveling back in time: replay the events one at a time up to a chosen moment (e.g., 2 weeks ago) and look at what the state was at that point. Does it look right? Then replay a few more until you arrive at the wrong state. Is this one already wrong? Step back and find the event (or events) responsible for the corrupted state.
Combined with a few events before and after, the sequence lets you reconstruct the whole story of what happened and why. Congratulations! You now have a broader view of what most probably caused the problem. Fixing the spot in the code where your logic goes wrong (an edge-case that isn’t supported, perhaps?) should now be fairly straightforward.
Sounds like a Hollywood plot, right? It isn’t all sunshine, though, and to complete the picture there are a few things I haven’t mentioned yet. How, exactly, do you time-travel on production data? How do you look at intermediate states when all you’ve got is a sequence of events, sometimes stored in binary rather than any human-readable form (more on that later in the series)? And lastly, how do you fix the bug for users who’ve already been affected - do you rewrite the past? Let’s take these questions one at a time.
Invest in and build your project tooling
Event sourcing has no notion of a “persistent” or “materialized” state, so how do you examine the state after each event in a sequence while troubleshooting?
Essentially, you repeat the same routine your entity goes through when it replays its event stream to handle a command. The difference is that the entity replays every event from start to finish to arrive at the latest state for command handling, whereas you want to stop sooner – before or after a specific event, before or just after a given timestamp, etc.
A devtool like this is typically the first thing I write when working on event-sourced services, and also one of the first I grab when troubleshooting such problems. Usually it’s a command-line tool or some easy-to-launch program that sits alongside my production code, so I can replay events exactly like my production entities do, only restricting which events get fed in. I run it on production data, either by loading a dump of the given entity’s event stream into a local database or, where possible, by connecting to a read-only production database. It accepts the faulty entity id plus either an event sequence number or a timestamp up to which the stream should be replayed, and it relies on the same code as production entities to fetch, decode and apply events. When it hits the limit, it just prints the entity’s current state to the console. Sometimes I add an extra feature: once a certain event is reached, each following event in the stream is applied only “on key press,” with the state printed after every one. It’s like building your own small debugger for your entities.
Even when your events are stored in a binary format, using the real event handler machinery means you don’t have to handle that explicitly or decode them by hand once more. With a tool like this, time travel becomes easy: you can see exactly what state the entity was in, e.g. 2 weeks ago, and how each event changed it. It only reads a set of records from the database, and applying events triggers no side effects, so you can run as many attempts as it takes to find the faulty event.
Over time, the project’s dev toolbox typically keeps growing, and specific, ad-hoc views/projections get built to answer frequent questions quickly. Additional tools around entities appear that make day-to-day work easier, troubleshooting in particular, but the one for time-traveling and debugging event streams is usually the first I create.
Correcting the past
With the problem identified, it’s time to fix it. You know the routine: change the code, write and run a set of tests, wait for the CI pipeline to complete, and ship it. That’s only half the job, though. What you have now is correct behaviour for commands arriving in the system from this point on and for the new events produced in response to them.
What about the entities the bug has already affected, whose streams contain these faulty or just plain wrong events? You can no longer simply run an UPDATE statement on some database rows, since no actual state exists. Keep in mind that the past can’t be changed – events are immutable facts that you shouldn’t modify.
But why is that, really? At first glance, it seems like a reasonable approach: replace the faulty event in the event store with a correct one, and you’re done. For one thing, if you do this, you must ensure the new version of the event can really be applied correctly to the corresponding state, i.e. that the event makes logical sense in the sequence. That’s the easier bit, though, and it can usually be checked. The less obvious and typically invisible gotcha is how “local” such a manual intervention is.
Apart from the write side, event-sourced systems also include so-called read sides – pieces of code that consume the event streams your entities write and act on them. What they do can range from building views/projections of the data (e.g., reports, graphs, etc.), through building dedicated read models in external, specialized databases needed for more or less critical business decisions, to ordinary event handlers that explicitly call other services or publish messages (or “public events”) to queues that drive other parts of the system.
Every downstream component depends on your event streams being logically consistent and on the guarantee that a saved event stays there forever, unchanged. This means every report and read model has already consumed that event and applied it on its side (likely producing wrong numbers, which might be how you discovered the original problem in the first place). Every event handler has consumed the event too, calling other systems or publishing messages based on its data. After handling an event, they move on and never look back, so they won’t even notice that you manually altered one of the earlier events in the stream.
Although you “fixed” the event itself, you didn’t fix everything that the wrong event affected. Of course, you can try to modify or rebuild your read models by hand and re-trigger calls to other services with correct payloads, but are you certain you know about and control every downstream consumer of your events? And what about data consistency should an audit happen?
One more big selling point of event sourcing and its event log is easy auditability, which comes practically for free. What if you work in a heavily regulated environment, or are simply audited on a regular basis? Inconsistencies introduced by hand like this will surface during an audit sooner or later, and you’ll face some rather unpleasant consequences. Permitting changes to past events destroys any trust an auditor might place in the event store. Immutability is a strict requirement for any log-style data like this, particularly when it has to meet regulations.
Don’t misunderstand me – overwriting events isn’t completely off-limits. There are services and situations where doing so is perfectly fine: only a handful of downstream consumers that you fully control, which are easy to correct, just like the read models; the system isn’t “regulated,” etc. I’ve done it myself a few times – operating on history is quite stressful, but in each of those cases it went well.
That said, the proper and most elegant way to deal with such problems is to make the correction explicit. In practice, this means sending a healing/compensating command to the affected entity, which then stores a compensating event that fixes whatever was wrong with the earlier state.
The event payload can contain anything you need – corrective values, a reference to the event/action being corrected – that’s your call. Sure, you could call it overengineering, since it generally takes time and requires new code (sometimes code you’ll run only once), but, well, it’s all about tradeoffs. Let’s look at what you gain in return.
Publishing this new kind of healing/compensating event, first of all, makes the correction explicit: an event is durably recorded in the event stream – so no more risk of inconsistencies when audited, etc.
As a result, you in a way “force” every read model and event handler to handle this new event explicitly. They might decide to ignore it, of course, but at least they have to acknowledge receiving it. They can react by, e.g., correcting the figures in their reports or sending corrected messages downstream – that’s their decision.
You’ve made your corrective decision explicit by recording a fact about the entity, and everyone interested in changes to that entity’s state had to pick the appropriate way (in their own context) to respond to it.
Alternatively, your entity logic may be entirely correct and the bug you’re facing may exist only on the read side, e.g. some reports produce incorrect data, some views don’t show what they ought to, etc. Fixing these is usually fairly easy and safe (setting aside the inherent difficulty of working out the change itself).
Keep in mind that your event streams are the single source of truth, so you can rebuild the state at any point in time simply by replaying events and examining the resulting state from whatever perspective you like. That’s precisely how read models work: they just replay event streams and interpret the incoming events in their own way.
All you have to do is fix the broken projection (read-side) and let it rebuild from the very start (or from some later point preceding the problem). It may take a while, of course, depending on how many events must be processed, and you may have to work out a graceful switchover where necessary, but in essence that’s all there is to it.
Since your entities hold their complete history of changes, you can replay it whenever you like, from the beginning or from any chosen point in time.
Architecting for tomorrow’s questions
Event streams as the source of truth make practically “free replays” possible. A practitioner put this idea into a single sentence that I first came across when a well-known event sourcing advocate quoted it, and to me it captures the biggest strength of event sourcing. Several times, after a service had been running in production for a while, a request came in from the so-called “product team” asking whether we could say this or that based on the activity and actions in the service up to that point. Usually they wanted some specific view of the data, e.g., how many, how much, how often, what’s the sum, total, average, trend, etc.
At first sight, this all seems like a job for the standard business metrics you should build into your system no matter how it’s designed. And that’s absolutely true – some of these things can be read from metrics. The catch is that you can’t foresee every metric you’ll need in the future.
You can only write code for what you know right now. There’s no guarantee that each business-relevant change has a matching metric, because metrics are just optional extra steps on the side that the process doesn’t need in order to work. With event sourcing, by contrast, supporting the business flow as it stands requires you to store all the information needed to move from one state to the next, in the form of an event stream.
As a result, you potentially record all the relevant information about the changes, so you can later look at that data from many angles, depending on what the business needs. Even if certain reports were never planned, as long as you have the data, answering the new questions is just a matter of building a new read model or projection that focuses on the relevant aspects of it.
This has happened to me many times already: sometimes the business sent ad-hoc reporting requests that we could fulfil using the event data, and other times new read side event handlers were created to send part of our historical data to external systems for a new business feature, etc.
In general, the longer a service runs and the more data it accumulates, the greater the chance that this data will help with new business decisions nobody even considered when the service was first built.
Summary
That’s all for now. In this first, rather long part, I concentrated on some less obvious properties of event-sourced systems related to how state is represented in them. I also talked about the ability to travel in time – how to do it and what it means for engineers, when troubleshooting and fixing bugs, and for product people, who can grow the business by gaining deeper insight into the data.
In the next part, I’ll concentrate on practices and a few gotchas of modeling event sourced systems – essentially, how to improve your odds of not shooting yourself in the foot while building one. In part 3, I’ll look at the event persistence options available, with consistency as one of the key factors.
Stay tuned!