Estimate project
← All posts

What I wish I had known when starting with Event Sourcing – part 3, storage

Let's look at the options we have for persisting events, treating consistency as one of the main criteria.

12 min read
Event SourcingArchitectureSoftware DevelopmentSystem Architecture

Earlier, we went through data consistency considerations in event-sourced systems. We covered read-side data consistency only briefly, since that is where consistency implications are most often noticed and thought about. There is, however, another crucial data-related aspect of event-sourced systems: the write side, where events get persisted as the outcome of handling commands. In this part, let’s take a short look at the options we have for persisting events, treating consistency as one of the main criteria.

What Makes an Event Store?

What does a database, or any other persistence mechanism, need to offer to be a realistic candidate for storing events in an event-sourced system? Clearly, the storage has to be durable. It also has to provide an adequate transaction isolation level, so that only fully committed data can be read. Because we deal with append-only logs, it helps if the system offers decent insert performance (updates and deletes matter little here).

On top of that, it has to:

  • offer atomic and sequential writes (when a series of events is written at once),
  • support ordered sequential reads, at least within a given aggregate (loading the full sequence of events belonging to that aggregate),
  • support globally ordered sequential reads, so that all events can be read in the order in which they were stored.

These requirements narrow our choices considerably, to database solutions, since we need at least a large part of the ACID guarantees, along with other features that databases offer (filtering, indexing, etc.)

The internet hosts a continuing and rather heated debate on whether Kafka is suitable as event sourcing storage (back in the day, we even had two or three fairly long discussions about it with colleagues from our company). Kafka is excellent when it comes to events, but in my view it isn’t especially helpful for the kind of event “processing” required here – more on that below.

What is an Event?

Seen from the storage side, an event in an event-sourced system can be regarded as a structured piece of data with at least these attributes:

  • Aggregate ID: The aggregate to which the event belongs.
  • Sequence Number: Its position within the event stream of a given aggregate.
  • Timestamp: The moment the event was recorded.
  • Payload: The event’s actual “content”.

Other useful, though not strictly necessary, attributes include:

  • Global Sequence ID: Handy for system-wide projections.
  • Tags: Likewise handy for system-wide projections, groupings, etc.

There can be additional fields specific to a storage or solution, but fundamentally this is everything we need in order to write an event to an event store and read it back later, when replaying a given aggregate’s event stream to handle a command. So how do popular databases measure up against these requirements?

Relational Databases

Not surprisingly, relational databases are quite a solid option here. They generally satisfy the ACID requirements we have, and with the sharded/multi-node solutions that have recently gained popularity for larger-scale problems, scalability and consistency can sometimes be tuned.

Events live in a single table, which may reference the aggregate ID from a separate table, because putting the aggregate ID into every event might not be desirable. When the ID is long, it is better to point to another table via a foreign key to save space. Some RDBMS let you partition event tables by time or shard them (e.g., by ranges of aggregate IDs) to improve long-term performance, particularly for reads. Sharding by time is especially handy when event stream snapshots are used – only a handful of the latest events are then needed for replay.

Sequence numbers within an aggregate serve two purposes: ordering events and optimistic concurrency control that prevents concurrent writes (when the application does not enforce a single-writer principle, as Akka/Pekko Persistence does). Sequence numbers within aggregate streams are probably best managed explicitly by the application (because of optimistic concurrency control), whereas global sequence numbers can typically be generated automatically from sequences. This comes with downsides, though: in Postgres, for instance, sequence numbers are incremented in separate transactions, which can lead to gaps or out-of-order values in the global sequence (e.g., events received a sequence number, but the transaction storing them was rolled back or failed for some reason). That complicates things on the read side.

As for the read side, one weakness of RDBMSs is that they typically lack dedicated mechanisms for reading event streams for read models/subscriptions/projections. That is to be expected, since they weren’t built with this use case in mind. The closest-to-native options are the various flavors of CDC (Change Data Capture), based either on triggers or on op-logs (in Postgres, the Write-Ahead-Log combined with the logical decoding feature). They demand a fair amount of read-side code to build the infrastructure for reading, streaming and then reliably processing the events.

Alternatively, you can fall back on good old polling of the events table while keeping track of the event’s sequence number. Depending on whether the read model concerns a single aggregate or the whole system (e.g., by tags), you track either the aggregate’s sequence number or the global one. Here be dragons, though, because the gaps and mixed ordering described above can cause trouble. To make sure no event is missed on the read side, you need some control-and-wait mechanism – which is, for instance, what Akka/Pekko Persistence Query does under the hood.

One more approach is to rely on Postgres’ internal transaction IDs (xid) for global ordering. Their semantics suit our use case “better”. If you want the details, I recommend a blog post dedicated to precisely this subject: Ordering in Postgres Outbox.

To sum up RDBMS: they do quite well on the write side and are still the most popular choice, even in bigger systems. In exchange for their ease of use and familiar concepts, you have to put in some extra work on the read side, but once that is in place, it generally runs without a hitch.

NoSQL Databases

This is the next family of storage engines that can serve as event stores. Probably the best known are Cassandra (or ScyllaDB) and AWS’s DynamoDB, both column-oriented databases, and MongoDB, a document database. I used Cassandra as event sourcing storage only briefly (we eventually moved to Postgres), so what follows comes more from my research into the subject than from hands-on experience.

Document-oriented, schemaless models can fit events better, because they offer more flexibility, depending on how you design the event payload (either as an application-managed blob of serialized data, like in RDBMS, or as a first-class structure inside a database document). In document databases (e.g., MongoDB), events can be kept either in a single document (one document per aggregate) or in separate documents. Each option has advantages and disadvantages; for instance, a single document may have a size limit, which can become a problem if an aggregate grows a lot. On the other hand, operations on a single document are ACID-compliant by nature. MongoDB does support multi-document transactions, but they can cost you in performance – there’s no free lunch.

MongoDB uses sharding, which naturally lets events be spread across shards, giving more headroom for handling large volumes of events.

As with RDBMS, the application itself has to maintain sequence numbers for a single aggregate’s events, but a major weakness of MongoDB seems to be that it has no built-in way of sequencing events globally without extra logic and handling on your side. Since MongoDB uses a leader-follower architecture and every write passes through the leader, a dedicated collection acting as a sequence generator should be feasible (though this probably requires multi-document transactions and thus possible performance costs). Unofficially, MongoDB keeps documents in a stable insertion order, but as far as I know, this isn’t documented, and relying on it is discouraged.

In column-oriented databases such as Cassandra, events can be stored as columns in a row whose partition key is the aggregate ID and whose clustering key is the sequence number in that aggregate’s stream, which makes it possible to order events within one aggregate. Sadly, because Cassandra is distributed and leaderless, it offers no built-in mechanism for global sequencing/ordering of all events, so the application has to take care of it. Why is global ordering useful? When you want to follow events across many entities (e.g., to react only to events of a certain type), you need a way to track and checkpoint the events you have already seen and processed, so you can resume from that point.

As Cassandra is distributed, conflicting writes may happen, and they are resolved with a “last-one-wins” strategy based on timestamps. This can occasionally cause events to be lost. One remedy is to set consistency levels appropriately, e.g. QUORUM or ALL when strict consistency is required, but that of course affects latency.

Then again, being leaderless and distributed, Cassandra is a great fit for write-heavy, read-heavy (provided reads are key-based) and possibly geographically distributed applications that can give up some data consistency in exchange for availability.

For MongoDB as well as Cassandra, I don’t know of any native, built-in mechanism for subscribing to event streams, so once again this has to be done explicitly with polling or CDC mechanisms (e.g. Debezium with Cassandra CDC or with a MongoBD replicaSet enabled).

One more point about consistency: Greg Young, the man who “invented” Event Sourcing, gave a great talk dealing with exactly this subject – the consistency of the event store on the write side, particularly in geo-distributed systems. I strongly recommend watching it.

Dedicated Event Stores

The market offers several products built mainly for event sourcing use cases. They are databases designed to store events as streams and to let the read side query those events. Likely the best known is EventStoreDB, created by Greg Young himself. Another one I know of is Axon Server, which has an event store among its core components. Axon Server belongs to the Axon stack, an event sourcing solution for the Java ecosystem.

In general, these solutions deal with some (though not all) of the weaknesses of the approaches described earlier.

For me, the most valuable feature of EventStoreDB is its built-in support for subscriptions and various read-side queries, together with global ordering of events. Each stream naturally has its IDs sequenced, with a guarantee that they increase strictly monotonically (with no gaps). On top of that, every event is also sequenced globally, so you can read the so-called $all stream in the order the events were stored. An event’s global position in the stream is a composite of a “prepare position” and a “commit position,” which guarantees correct event ordering.

EventStoreDB also offers a neat concept of “projections,” which essentially lets you create a new, intermediate stream made of events matching certain criteria (e.g., event type) and subscribe to it with a catch-up subscription. This lets you organize and extend your read side cleanly, shaping projections and subscriptions however you like.

However, EventStoreDB uses a leader-follower architecture, so it doesn’t scale as well as, say, the leaderless Cassandra – every write must still pass through the leader, although replicating to followers and read-only nodes does improve availability and fault tolerance.

Overall, dedicated event stores are products that are hard to compare one-to-one with RDBMS or NoSQL databases. Because of their specific purpose, they use novel approaches that the other solutions naturally lack, such as built-in subscriptions, categories, etc. Whether one of them solves your event sourcing problem depends heavily on your use case, so, as always, use your judgment and research thoroughly before committing fully.

And What About Kafka?

Kafka handles events very well, but where it really excels is high-throughput, fault-tolerant event streaming. As an event store (in the sense event-sourced applications require), it has some limitations that make it a poorer fit for the job. Let’s quickly go through a few of them.

To begin with, Kafka guarantees event order only within a single partition of a topic. So, to guarantee event order within one stream, you would need either a partition per stream or a single-partition topic per aggregate. Neither approach scales well, though.

A proper event store should make it possible to replay a given aggregate’s events quickly. Kafka can be set up to retain events for a long time through its retention parameters, but there is no efficient way of picking out a given aggregate’s events from the whole stream, unless you split up the aggregates as described above, which isn’t practical.

On top of Kafka, there is no way to implement concurrency control when saving events, which can put data consistency at risk. Some conflicts can be detected and dealt with in code, but doing that for every possible case isn’t practical. One way to soften this is to apply the single-writer principle and handle concurrency control entirely at the application level.

These are only some of the downsides of using Kafka as a real event store. Some of these issues can certainly be worked around with extra tools and infrastructure, such as ksqlDB, Kafka Streams, topics and partitioning strategies, but my impression is that the more cases you try to cover, the setup gets exponentially more complicated.

Don’t get me wrong – I like Kafka and use it extensively for data streaming and as a message broker. That is where it undeniably shines and where it remains a dominant market player. It simply isn’t a great choice for an event store.

Summary

We have gone through several of the available options for storing events in an event-sourced system, from the most widespread RDBMSs, through highly scalable and distributed NoSQLs, to dedicated event store databases. Each option has its strong and weak points across the different aspects of event sourcing requirements, yet relational databases are still the most common choice thanks to their popularity and flexibility. Deep down, I wish dedicated event stores were more popular and held a bigger share of the market, since they can provide excellent solutions built specifically for event-sourced applications.