Estimate project
← All posts

Evolving Protobuf schemas the right way with ScalaPB and fs2grpc

When APIs are defined with Interface Description Languages, schemas can evolve in a sturdier and safer way than with JSON-based messages. Whether adding/removing message fields is completely safe depends on several factors. Let’s go through different communication modes and think about which one to choose.

9 min read
ScalaProtobufgRPC

The schema-driven paradigm has gained wide popularity across many communication models. When APIs are defined with Interface Description Languages, schemas can evolve in a sturdier and safer way than with JSON-based messages. Depending on a number of factors, adding/removing message fields may be entirely safe or may break binary compatibility once deployed. gRPC services, event sourcing, and Akka Cluster actor communication may each call for a different schema evolution policy – and the choice may also vary with the particular wire protocol. Let’s go through different communication modes and think about when to choose backward, forward, or full compatibility.

Technology

The focus of this article is Google Protocol Buffers, AKA Protobuf – the default underlying protocol of gRPC services and a strong option for messaging-based communication. The gRPC communication style is It takes its own approach to permitting schema changes, required fields, and default values. If you build backends in Scala, ScalaPB lets you generate case classes from the .proto contract while the project compiles. These generated types can be made sturdier still with extra annotations, which we’ll look at below.

Communication style

gRPC

This form of synchronous service-to-service communication keeps gaining popularity. Classic REST contracts are awkward to evolve – and that’s before we even mention the better performance and streaming support. A further problem is expressing service use cases through HTTP verbs such as POST/PUT/DELETE/etc. together with resource URLs, which frequently fit real use cases poorly. gRPC solves these problems. In the gRPC world, we usually speak of clients and servers. Typically, the server service owns its contract in the form of protobuf files, from which case classes can be generated with ScalaPB**.** Next, fs2grpc can produce a thin client layer built on Cats Effect, and the CI can package all of this code into a versioned .jar artifact that the server publishes. Client services then depend on these generated artifacts.

Messaging

Asynchronous, message-based communication brings plenty of benefits, such as decoupling, delivery guarantees, event sourcing, command sourcing, etc. You can build it on Kafka, Pulsar, or other infrastructure solutions. With Kafka, this pattern is frequently complemented by an external Schema Registry that helps manage schema compatibility between producers and consumers. Let’s concentrate on the following two integration patterns:

  • 1 Producer - multiple consumers
    A service publishes its events, and consumers may join or leave to read and process them. Typically, schema changes are driven by the producer.
  • 1 Consumer - multiple producers
    A service reads commands from a topic to which potential client services can publish. Here it is the consumer that updates the schema, which makes it resemble the gRPC server.
  • Communication between actors of the same type
    This scenario is specific to models in which actors of the same type message one another, e.g. inside Akka Cluster. We won’t go into it in detail. Suffice it to say that it requires full compatibility, so that during a rolling update actors can exchange messages in both the new and the old versions.

Compatibility modes cheat sheet

Now let’s translate the Protobuf usages mentioned above into lists of permitted operations, such as adding/removing fields, changing optionality, etc. To start, a short reminder of the compatibility modes:

  • backward – the best-known mode. Data consumers must be able to consume older versions of data objects.
  • forward – applies when the consumer holds an older schema version yet has to understand data serialized with a newer one.
  • full – the strictest constraint, demanding both backward and forward compatibility.

gRPC

For requests, keeping backward compatibility is recommended. The server updates its schema first, without having to worry about requests from clients still on older schemas, and the clients then update their schemas step by step. For responses, the mode you want is forward, so that clients can still handle new messages coming from the server. Note that the no_box wrapper is explained in detail further on, in the Optionality section.

mode communication scenario method
BACKWARD gRPC requests add an optional field to the request add it to proto
BACKWARD gRPC requests make a required request field optional remove no_box (specific to ScalaPB)
BACKWARD gRPC requests make an optional request field required breaking
BACKWARD gRPC requests drop an optional field from the request remove it from proto
BACKWARD gRPC requests drop a required field from the request remove it from proto
BACKWARD gRPC requests add a required field to the request breaking
BACKWARD gRPC requests add a field to a oneof in the request add it to proto
BACKWARD gRPC requests move a field into a new oneof in the request move it in proto
BACKWARD gRPC requests drop a field from a oneof in the request breaking**
FORWARD gRPC responses add an optional field to the response add it to proto
FORWARD gRPC responses add a required field to the response add it to proto
FORWARD gRPC responses drop an optional field from the response remove it from proto
FORWARD gRPC responses make a required response field optional breaking
FORWARD gRPC responses drop a required field from the response breaking*
FORWARD gRPC responses add a field to a oneof in the response breaking**
FORWARD gRPC responses move a field into a new oneof in the response only for an optional field, move it in proto
FORWARD gRPC responses drop a field from a oneof in the response remove it from proto

Bear in mind that in Protobuf, removing fields is fine as long as the index number is never reused. The reserved keyword lets you make it explicit that an index must not be used again.

  • doable under certain restrictions – see the Optionality section below.
    ** A special remark about oneofs: if a oneof variant is removed in requests, the server decodes that value as UNKNOWN, which in theory cannot be told apart from an unset value.
    This is why I classify it as backward incompatible. In the same way, adding a field to a oneof counts as a forward incompatible change. If you are certain that all your clients use ScalaPB and that you handle UNKNOWN values properly, you can make such changes – but very carefully.

Messaging: events

Here, a publisher owns the contract and consumers read the events it publishes. This style is frequently realized through a .jar artifact with the .proto file plus generated ScalaPB classes and fs2grpc services, which the server maintains and publishes. Usually, the publisher is the first to adapt its code to the new contract, and consumers come after. Still, let’s go through every sub-case:

  1. If the producer needs to update events without breaking consumers, which can keep using the old schema until they update, forward compatibility suffices.
  2. Or you may prefer to update consumers first. If you choose that order, backward compatibility is what you must maintain. This approach is described in the Avro and Schema Registry documentation – take a look to go deeper.
  3. Full compatibility is the strictest option. It is advisable when we need to guarantee that consumers, even once adapted to the newest schema, can still parse older events and specially handle fields whose values are present or missing. Pick it when you expect historical events to be replayed, so that consumers handle every older version correctly – in event sourcing, for instance.
mode communication scenario method
FORWARD/FULL messaging: events add an optional field add it to proto
FORWARD messaging: events add a required field add it to proto
FULL messaging: events add a required field breaking
FORWARD/FULL messaging: events drop an optional field remove it from proto
FORWARD ONLY messaging: events make an optional field required add no_box
FULL messaging: events make an optional field required breaking
FORWARD/FULL messaging: events make a required field optional breaking*
FORWARD/FULL messaging: events drop a required field breaking
FORWARD/FULL messaging: events add a field to a oneof breaking
FORWARD/FULL messaging: events move a field into a new oneof move it in proto (optional only)
FORWARD ONLY messaging: events drop a field from a oneof remove it from proto
FULL messaging: events drop a field from a oneof breaking

Messaging: commands

This mode closely resembles the handling of GRPC requests. The contract is owned by a command handler, so it has to stay backward compatible.

mode communication scenario method
BACKWARD messaging: commands add an optional field add it to proto
BACKWARD messaging: commands add a required field breaking
BACKWARD messaging: commands drop a required field remove it from proto
BACKWARD messaging: commands drop an optional field remove it from proto
BACKWARD messaging: commands make a required field optional remove no_box
BACKWARD messaging: commands make an optional field required breaking
BACKWARD messaging: commands add a field to a oneof add it to proto
BACKWARD messaging: commands drop a field from a oneof breaking
BACKWARD messaging: commands move a field into a new oneof move it in proto

Optionality and no_box

Starting with version 3, the Protobuf protocol no longer permits required fields, and optional is the default. The reasoning is that letting people add/remove required fields led to too many issues with unexpectedly broken wire compatibility (see the original comment on GitHub).

Yet, as the tables above demonstrate, in certain scenarios required fields are reasonable. With BACKWARD compatibility, for example, it is completely acceptable to start with required fields in the initial schema and remove them later on. This holds for gRPC requests and for asynchronous commands. To stop ScalaPB from wrapping non-primitive types in Option, add a no_box annotation, for example:

google.protobuf.Timestamp createdAt = 1 [(scalapb.field).no_box = true];

Dropping required fields breaks FORWARD compatibility, but in particular situations this operation can be carried out in steps:

  • Make the required field optional by removing the no_box wrapper
  • Update every consumer so it handles the None case
  • Begin producing the None case

Watch out! This is a risky operation. It does not entirely comply with the compatibility rules in our cheat sheet, so I still label it “breaking”.

Default values

ScalaPB does not wrap primitive protobuf types such as string, double, and others in Option. Default values are set instead, for example the empty string “”. This can be quite dangerous, since in most cases such default values are in fact invalid – so if you leave out a field while constructing a case class, you’re in trouble. To improve type safety, you can take one of two approaches:

  • Put no_default_values_in_constructor in your .proto file, which just turns default values off. Downside: the field becomes strictly required, and there’s no simple way to turn it into an optional field later.

    option (scalapb.options) = {
    no_default_values_in_constructor: true
    };
  • Rely on wrapper types
    They can add some noise to the code, since reaching the actual value takes extra .value calls, but together with no_box this approach gives you better control over how optionality evolves.

Bonus: buf

Maintaining protobuf files, distributing the contract, enforcing compatibility rules, and keeping a coherent set of API design rules all demand substantial investment. buf is an interesting toolset designed to automate these processes. It comes with a freshly built high-performance Protobuf compiler, a linter that enforces sound API design choices and structure, a breaking change detector that enforces compatibility at the source code or wire level, and agenerator that runs your protoc plugins according to a configurable template. It also includes The Buf Schema Registry (BSR) – a hosted SaaS platform acting as your organization’s source of truth for its Protobuf APIs. Think about bringing parts of buf into your project pipeline to get more out of working with Protobuf.

Conclusions

Which compatibility requirements your schemas need depends on many factors. ScalaPB extensions such as required fields or disabled default values can raise type safety on the application side without sacrificing Protobuf wire compatibility, provided that the evolution rules are well matched to the use case. I hope this guide helps you tune those rules for your particular application – or even fully automate how they are enforced.