Estimate project
← All posts

Caching in a Microservice Architecture

Caching on its own is a very wide notion. It may refer to a reverse proxy caching HTTP responses at the front of an application, or to caching database calls deep in the backend. Let’s explore the subject in more detail.

11 min read
MicroservicesArchitectureCacheRedishazelcast

It seems no text on caching is complete without the well-known quote about the two hard problems in Computer Science, so I will open with it too:

Computer science has just two hard problems: invalidating caches and naming things.
– Phil Karlton

If getting caching right is so painful, why bother at all? Perhaps we should simply steer clear of it whatever the cost? Sadly, it’s not that simple, especially when we are targeting a microservices architecture. Creating a fast distributed system without any caches can be really hard.

Caching on its own is a very wide notion. It may refer to a reverse proxy caching HTTP responses at the front of an application, or to caching database calls deep in the backend. Let’s explore the subject in more detail.

Why caching is important

A microservice is a self-contained piece of a software system that wraps a portion of its logic. Typically it is also in charge of some subset of the system’s data (for example, making sure invalid updates don’t corrupt it). This duty of a service is frequently referred to as data ownership. Even so, different services in the system have to share the data they own. Most often they make it available through public, synchronous interfaces such as a REST or gRPC API.

A synchronous request always implies a call over the network. Even on a fast internal network, sending data takes far longer than sharing it within a single local process of a monolithic application. On top of that, network roundtrips create temporal coupling: the service being queried and the one querying must both stay available for the entire duration of the call. This can hurt the overall availability of the system.

This style of communication does have its advantages, too. Above all, we always receive a fairly fresh copy of the data from the queried service, which is essential when working on stale data is not an option.

Keep in mind that even synchronous communication offers no guarantees about data consistency. The data a service owns may change immediately after it has sent its response. If we need strong, atomic guarantees, the changes have to happen within a single service.

The main weakness of synchronous communication, though, is that it’s hard to scale while keeping performance acceptable. Fortunately, the right caching lets us do better.

Caching strategies

It happens quite often that one service keeps asking another for the same pieces of information. To skip unnecessary network calls, we can just keep that data in an in-memory, in-process cache. Once the first call has fetched the data, subsequent requests can just read the cached entity. Because the data passes through the cache on its way after a read, this strategy is known as read-through.

image1

Caching drastically shortens the time needed for later data accesses, but the very first read still needs a slow call over the network. We also have to think about how to tell whether the cached data is still current. We’ll come back to this later.

The cache may also be refreshed when we send data to another service or persist it in the datastore. Intuitively this is very sensible: something that was just created or modified is quite likely to be read again soon. When the cache content gets updated after the request (or alongside it), we call the approach write-through. The write-back strategy updates the cache first and only afterwards sends the request that updates the entity in the other service.

Either approach works only when every modification passes through one service instance. If writes happen on different replicas, the caches will drift out of sync very quickly. The idea that only a single node is in charge of updating given groups (shards) of entities is known as the Single Writer Principle (SWP). A well-known example of a framework that implements SWP is Akka Cluster.

Cache invalidation

Keeping data in an in-process cache raises the memory requirements of the service. A cache with unlimited size from which nothing is ever removed will soon cause out-of-memory errors. For this reason caches typically have a bounded size and apply some eviction strategy when they overflow:

  • Least-recently-used (LRU) ranks among the most popular strategies. Whenever the cache has to discard an item, it picks the one that has gone unaccessed for the longest time. It rests on a practical observation: an item that was used recently is quite likely to be requested once more. Computer science calls this phenomenon temporal locality of reference.
  • The weakness of LRU is that it can discard even often-used items when a lot of new items get cached in one go. Even if nobody ever touches those new entries again, they will still manage to push other hot entries out of the cache. LFU (least frequently used) solves this by evicting only the items that were used least often.

Plenty of other approaches exist (such as most recently used - MRU or first-in-first-out - FIFO), and they can be a good fit for particular cases.

We should always verify with metrics that the caching strategy we picked is the right one.

Among the best measures of how efficient a cache is, is the ratio of hits (data successfully served from the cache) to misses.

A large number of misses can signal that the eviction strategy doesn’t fit.

Cold and warm cache

The in-memory cache of a freshly started process is empty at first (cold). It only fills up once the application has done some reads and writes. This is what we call the cache warming up. As a result, a service can be somewhat slow right after startup and reach its top performance only after a while. An elegant fix is pre-warming: populating the cache asynchronously right after the service starts. The problem is that we can’t always tell beforehand which data is worth caching.

A different issue appears when the service is scaled out to multiple instances. Each running application process keeps its own separate in-process cache, so inevitably some entities end up cached in one instance but not in another. Every time a request hits the wrong instance, it results in a cache miss. Sticky sessions or sharding with consistent hashing can help avoid this. A standalone cache is an even better option.

Standalone cache

A standalone cache is a dedicated infrastructure component that the application’s instances share. It addresses some of the issues mentioned in the previous section. Because the cache lives outside the application, its contents survive restarts. And since every instance of the application shares all of the cached data, reaching the correct node stops being a concern.

Redis and Memcached are popular choices for a standalone cache. More specialized options exist as well, such as in-memory-data-grids (for example Hazelcast or Apache Ignite), which can additionally run distributed computations over the stored data.

At times the volume of data we’d like to cache far exceeds what one machine can hold. With sharding, terabytes of data can be kept in the memory of many nodes. For the application, this kind of sharding is invisible - it’s the cache that routes requests to the instance that holds the given entry.

When it runs as a single instance, a standalone cache may turn into a single point of failure. For the system to be highly available, the cache component has to either run as a distributed cluster or have failover nodes. Redis, for example, supports both clustering (Redis Cluster) and promoting replicas when the main node fails (Redis Sentinel).

image3

A standalone cache does not have to be deployed as an independent component. In containerized systems (such as ones managed by Kubernetes) it can also run as a sidecar. A sidecar container is tied to a particular instance and shares its lifecycle (just like an in-process cache). At the same time, it provides all the features that cache systems offer. Put simply, it sits halfway between a fully fledged distributed cache server and an in-memory cache.

The downside of a cache running as a separate component is that it has to be reached over the network, so it can never match the speed of the in-memory option. Interestingly, in-memory caches can be distributed too. Hazelcast IMDG is a great example. It can act as a local cache within individual JVM processes while also replicating data to other instances. As a result, every read is served from the in-memory cache, and only writes carry the extra cost of the network roundtrips required for replication.

Cache expiration and invalidation

Sooner or later, the data in the cache will diverge from its source. Sometimes this merely confuses users, for instance when they made an update yet still see the old version served from the cache. In more serious cases it can cause bugs and corrupt data.

The simplest way of removing stale items from the cache relies on time. All we need to do is set a TTL (time to live) for items put into the cache. Once the TTL runs out, the item gets removed. When the item is requested again, it is fetched from the source once more and cached anew. This whole process is known as cache eviction.

This method has its shortcomings. When the queried data changes in an unpredictable way (some entities are more dynamic, others static), picking a suitable TTL can be hard. With a TTL that is too high we’ll frequently serve stale data, and with one that is too short we’ll evict items from the cache sooner than needed. Furthermore, once an item has left the cache, the next request for it must make a possibly slow network roundtrip to retrieve it again.

To prevent this, a scheduled job could refresh cached entries at regular intervals. It’s an excellent way to avoid latency spikes caused by evicting a hot cache item. This technique is known as refresh-ahead. Its downside is that when data rarely changes, we end up making lots of unnecessary requests.

We can also make the cache-refreshing logic somewhat smarter. To begin with, the queried service should send back a hash code computed from the data it returns. The caller can keep that data in the cache together with the hash and attach the hash to every subsequent request for the same item. The queried service can then use it to check whether the data has changed. If nothing changed, the service can just send back an empty response. If the entry was modified, it should respond with the data. We still have to make a network call this way, but it will typically be very light. The HTTP standard offers a dedicated mechanism for this kind of “optimistic” caching (the ETag header), though it’s simple to implement over other communication channels too, such as gRPC.

On the other hand, if we update the in-memory cache on writes, reads are guaranteed to return the latest version of the item, but only on the instance that performs the save (single writer). Reader nodes still have to invalidate their contents in some other way. With a distributed cache this is not an issue either: every instance sees the latest updates. We must remember that persisting something in the database (or sending a request) and updating the cache do not form one atomic operation. If the service crashes halfway through an update, the cache and the original data may end up out of sync.

Eventual consistency

Caching inevitably brings the system into a state in which some of its parts see stale data. Those parts no longer keep strong consistency guarantees and become eventually consistent. That is the cost of better performance and resilience.

A good rule of thumb in a microservices architecture is to accept eventual consistency wherever it is practical. Strong, synchronous coupling should be reserved for places where it is truly needed.

Items can also be invalidated in response to events. This is a rather elegant approach, particularly in an event-driven architecture. Once the original item changes, the data owner can publish a message that notifies all caching services, which then eventually refresh their caches.

Caching responses from other services exactly “as is” is sometimes not the best choice. Microservices can instead build complex data models derived from external data sources. They can collect bits of information from different parts of the system and store a denormalized version of it in a local datastore. In an event-driven approach, asynchronous messages can once more be used to signal that the original data has changed. This can be seen as yet another form of caching, except that the data is kept persistently in the querying service’s database.

Wrapping up

This article is intended as an introductory overview of various techniques related to caching. I have only just scratched the surface of this complex subject and leave the deeper exploration to you. The key takeaway is that caching can massively boost the performance of our system.

It’s a powerful tool, yet when misused it can lead to errors that are difficult to track down.

I hope this article was helpful to you. Take care!