Evolving Protobuf schemas the right way with ScalaPB and fs2grpc
When APIs are defined with Interface Description Languages, schemas can evolve in a sturdier and safer way than with JSON-based messages. Whether adding/removing message fields is completely safe depends on several factors. Let’s go through different communication modes and think about which one to choose.
The schema-driven paradigm has gained wide popularity across many communication models. When APIs are defined with Interface Description Languages, schemas can evolve in a sturdier and safer way than with JSON-based messages. Depending on a number of factors, adding/removing message fields may be entirely safe or may break binary compatibility once deployed. gRPC services, event sourcing, and Akka Cluster actor communication may each call for a different schema evolution policy – and the choice may also vary with the particular wire protocol. Let’s go through different communication modes and think about when to choose backward, forward, or full compatibility.
Technology
The focus of this article is Google Protocol Buffers, AKA Protobuf – the default underlying protocol of gRPC services and a strong option for messaging-based communication. The gRPC communication style is It takes its own approach to permitting schema changes, required fields, and default values. If you build backends in Scala, ScalaPB lets you generate case classes from the .proto contract while the project compiles. These generated types can be made sturdier still with extra annotations, which we’ll look at below.
Communication style
gRPC
This form of synchronous service-to-service communication keeps gaining popularity. Classic REST contracts are awkward to evolve – and that’s before we even mention the better performance and streaming support. A further problem is expressing service use cases through HTTP verbs such as POST/PUT/DELETE/etc. together with resource URLs, which frequently fit real use cases poorly. gRPC solves these problems. In the gRPC world, we usually speak of clients and servers. Typically, the server service owns its contract in the form of protobuf files, from which case classes can be generated with ScalaPB**.** Next, fs2grpc can produce a thin client layer built on Cats Effect, and the CI can package all of this code into a versioned .jar artifact that the server publishes. Client services then depend on these generated artifacts.
Messaging
Asynchronous, message-based communication brings plenty of benefits, such as decoupling, delivery guarantees, event sourcing, command sourcing, etc. You can build it on Kafka, Pulsar, or other infrastructure solutions. With Kafka, this pattern is frequently complemented by an external Schema Registry that helps manage schema compatibility between producers and consumers. Let’s concentrate on the following two integration patterns:
- 1 Producer - multiple consumers
A service publishes its events, and consumers may join or leave to read and process them. Typically, schema changes are driven by the producer. - 1 Consumer - multiple producers
A service reads commands from a topic to which potential client services can publish. Here it is the consumer that updates the schema, which makes it resemble the gRPC server. - Communication between actors of the same type
This scenario is specific to models in which actors of the same type message one another, e.g. inside Akka Cluster. We won’t go into it in detail. Suffice it to say that it requires full compatibility, so that during a rolling update actors can exchange messages in both the new and the old versions.
Compatibility modes cheat sheet
Now let’s translate the Protobuf usages mentioned above into lists of permitted operations, such as adding/removing fields, changing optionality, etc. To start, a short reminder of the compatibility modes:
- backward – the best-known mode. Data consumers must be able to consume older versions of data objects.
- forward – applies when the consumer holds an older schema version yet has to understand data serialized with a newer one.
- full – the strictest constraint, demanding both backward and forward compatibility.
gRPC
For requests, keeping backward compatibility is recommended. The server updates its schema first, without having to worry about requests from clients still on older schemas, and the clients then update their schemas step by step. For responses, the mode you want is forward, so that clients can still handle new messages coming from the server. Note that the no_box wrapper is explained in detail further on, in the Optionality section.
| mode | communication | scenario | method |
| BACKWARD | gRPC requests | add an optional field to the request | add it to proto |
| BACKWARD | gRPC requests | make a required request field optional | remove no_box (specific to ScalaPB) |
| BACKWARD | gRPC requests | make an optional request field required | breaking |
| BACKWARD | gRPC requests | drop an optional field from the request | remove it from proto |
| BACKWARD | gRPC requests | drop a required field from the request | remove it from proto |
| BACKWARD | gRPC requests | add a required field to the request | breaking |
| BACKWARD | gRPC requests | add a field to a oneof in the request | add it to proto |
| BACKWARD | gRPC requests | move a field into a new oneof in the request | move it in proto |
| BACKWARD | gRPC requests | drop a field from a oneof in the request | breaking** |
| FORWARD | gRPC responses | add an optional field to the response | add it to proto |
| FORWARD | gRPC responses | add a required field to the response | add it to proto |
| FORWARD | gRPC responses | drop an optional field from the response | remove it from proto |
| FORWARD | gRPC responses | make a required response field optional | breaking |
| FORWARD | gRPC responses | drop a required field from the response | breaking* |
| FORWARD | gRPC responses | add a field to a oneof in the response | breaking** |
| FORWARD | gRPC responses | move a field into a new oneof in the response | only for an optional field, move it in proto |
| FORWARD | gRPC responses | drop a field from a oneof in the response | remove it from proto |
Bear in mind that in Protobuf, removing fields is fine as long as the index number is never reused. The reserved keyword lets you make it explicit that an index must not be used again.
- doable under certain restrictions – see the Optionality section below.
** A special remark about oneofs: if a oneof variant is removed in requests, the server decodes that value asUNKNOWN, which in theory cannot be told apart from an unset value.
This is why I classify it as backward incompatible. In the same way, adding a field to a oneof counts as a forward incompatible change. If you are certain that all your clients use ScalaPB and that you handle UNKNOWN values properly, you can make such changes – but very carefully.
Messaging: events
Here, a publisher owns the contract and consumers read the events it publishes. This style is frequently realized through a .jar artifact with the .proto file plus generated ScalaPB classes and fs2grpc services, which the server maintains and publishes. Usually, the publisher is the first to adapt its code to the new contract, and consumers come after. Still, let’s go through every sub-case:
- If the producer needs to update events without breaking consumers, which can keep using the old schema until they update, forward compatibility suffices.
- Or you may prefer to update consumers first. If you choose that order, backward compatibility is what you must maintain. This approach is described in the Avro and Schema Registry documentation – take a look to go deeper.
- Full compatibility is the strictest option. It is advisable when we need to guarantee that consumers, even once adapted to the newest schema, can still parse older events and specially handle fields whose values are present or missing. Pick it when you expect historical events to be replayed, so that consumers handle every older version correctly – in event sourcing, for instance.
| mode | communication | scenario | method |
| FORWARD/FULL | messaging: events | add an optional field | add it to proto |
| FORWARD | messaging: events | add a required field | add it to proto |
| FULL | messaging: events | add a required field | breaking |
| FORWARD/FULL | messaging: events | drop an optional field | remove it from proto |
| FORWARD ONLY | messaging: events | make an optional field required | add no_box |
| FULL | messaging: events | make an optional field required | breaking |
| FORWARD/FULL | messaging: events | make a required field optional | breaking* |
| FORWARD/FULL | messaging: events | drop a required field | breaking |
| FORWARD/FULL | messaging: events | add a field to a oneof | breaking |
| FORWARD/FULL | messaging: events | move a field into a new oneof | move it in proto (optional only) |
| FORWARD ONLY | messaging: events | drop a field from a oneof | remove it from proto |
| FULL | messaging: events | drop a field from a oneof | breaking |
Messaging: commands
This mode closely resembles the handling of GRPC requests. The contract is owned by a command handler, so it has to stay backward compatible.
| mode | communication | scenario | method |
| BACKWARD | messaging: commands | add an optional field | add it to proto |
| BACKWARD | messaging: commands | add a required field | breaking |
| BACKWARD | messaging: commands | drop a required field | remove it from proto |
| BACKWARD | messaging: commands | drop an optional field | remove it from proto |
| BACKWARD | messaging: commands | make a required field optional | remove no_box |
| BACKWARD | messaging: commands | make an optional field required | breaking |
| BACKWARD | messaging: commands | add a field to a oneof | add it to proto |
| BACKWARD | messaging: commands | drop a field from a oneof | breaking |
| BACKWARD | messaging: commands | move a field into a new oneof | move it in proto |
Optionality and no_box
Starting with version 3, the Protobuf protocol no longer permits required fields, and optional is the default. The reasoning is that letting people add/remove required fields led to too many issues with unexpectedly broken wire compatibility (see the original comment on GitHub).
Yet, as the tables above demonstrate, in certain scenarios required fields are reasonable. With BACKWARD compatibility, for example, it is completely acceptable to start with required fields in the initial schema and remove them later on. This holds for gRPC requests and for asynchronous commands. To stop ScalaPB from wrapping non-primitive types in Option, add a no_box annotation, for example:
google.protobuf.Timestamp createdAt = 1 [(scalapb.field).no_box = true];
Dropping required fields breaks FORWARD compatibility, but in particular situations this operation can be carried out in steps:
- Make the required field optional by removing the
no_boxwrapper - Update every consumer so it handles the None case
- Begin producing the
Nonecase
Watch out! This is a risky operation. It does not entirely comply with the compatibility rules in our cheat sheet, so I still label it “breaking”.
Default values
ScalaPB does not wrap primitive protobuf types such as string, double, and others in Option. Default values are set instead, for example the empty string “”. This can be quite dangerous, since in most cases such default values are in fact invalid – so if you leave out a field while constructing a case class, you’re in trouble. To improve type safety, you can take one of two approaches:
-
Put
no_default_values_in_constructorin your .proto file, which just turns default values off. Downside: the field becomes strictly required, and there’s no simple way to turn it into an optional field later.option (scalapb.options) = { no_default_values_in_constructor: true }; -
Rely on wrapper types
They can add some noise to the code, since reaching the actual value takes extra.valuecalls, but together withno_boxthis approach gives you better control over how optionality evolves.
Bonus: buf
Maintaining protobuf files, distributing the contract, enforcing compatibility rules, and keeping a coherent set of API design rules all demand substantial investment. buf is an interesting toolset designed to automate these processes. It comes with a freshly built high-performance Protobuf compiler, a linter that enforces sound API design choices and structure, a breaking change detector that enforces compatibility at the source code or wire level, and agenerator that runs your protoc plugins according to a configurable template. It also includes The Buf Schema Registry (BSR) – a hosted SaaS platform acting as your organization’s source of truth for its Protobuf APIs. Think about bringing parts of buf into your project pipeline to get more out of working with Protobuf.
Conclusions
Which compatibility requirements your schemas need depends on many factors. ScalaPB extensions such as required fields or disabled default values can raise type safety on the application side without sacrificing Protobuf wire compatibility, provided that the evolution rules are well matched to the use case. I hope this guide helps you tune those rules for your particular application – or even fully automate how they are enforced.