Estimate project
← All posts

First steps with ZIO, part 1

Get to know ZIO – a powerful Scala library that makes building asynchronous and concurrent applications easy.

15 min read
ScalaApache KafkaZioDependencies

Hello! This post opens a series dedicated to ZIO – a powerful Scala library that makes developing asynchronous and concurrent applications easy. Over the course of the series, I’ll show you step by step how to use the various building blocks of the ZIO framework to create a fully working application, so that you come away with a solid grasp of what ZIO can do. Throughout the series we will be working with ZIO version 2.

Using this application as an example, I’ll demonstrate how to write ZIO services, build efficient streaming workflows, run an HTTP server with web sockets, tackle concurrency problems with Software Transactional Memory (STM) and suitable data structures, configure a ZIO app and, naturally, test it.

What will we build?

Context

You’ve most likely heard of Kafka, haven’t you? If not, I suggest reading a little bit of basic information before jumping into this series. In short, it is a streaming platform that lets you publish (write), store and subscribe to (read) events.

In application architectures that use Kafka, it is usually an internal component living inside some VPC and hidden from the outside world. That is, of course, the setup we want: it blocks unauthorized access to Kafka Brokers and protects our data from malicious or simply accidental operations. Still, it is easy to picture a situation in which we’d like clients to be able to read some of the data.

Suppose we’re building a platform to which clients (meaning external systems) upload data. Inside the platform we run various asynchronous computations on that data, producing notifications that are kept on Kafka topics internal to the platform. Let’s say we have to share these notifications with our clients, because they rely on that information to begin or continue their own workflow. We have to pass on the notifications stored internally on topics sealed off within our infrastructure, and we want to do it in real time – every new notification should reach the subscribed clients right away.

This calls for a middleware component, and that’s exactly what we’re going to build. The aim is to stream events from internal topics and pass them on to external clients that have subscribed. Keep in mind that, since the app serves an educational purpose, it contains plenty of simplifications – apart from that, though, the code is production grade.

Overview

Before diving into an overview of the system, let’s pin down the requirements. We’ll assume that what we basically want is to let the client “tail” selected Kafka topics. Put differently, a client who unsubscribes and then subscribes again 10 minutes later won’t get the messages published while they were away. This simplifies the implementation and keeps the code reasonably short and straightforward. We’ll call our service k-tail. Now let’s sketch our target in a simple diagram.

k tail overview

The application will run two core processes: the first consumes messages from Kafka, and the second forwards them to subscribed clients. Let’s walk through the components involved, beginning on the right.

First of all, we need a consumer instance that keeps reading new messages from a set of preconfigured topics. These messages are then held temporarily in a buffer. Thanks to the buffer, consuming and forwarding messages can be handled independently, which gives us flexibility in terms of speed and error handling. Lastly, we need a component whose job is to forward messages to clients. We’ll call it the broadcast.

Taken together, these components make up a web socket server that lets clients connect and get new data in real time.

Plan

The Getting started with ZIO series consists of three parts, in which we explore ZIO concepts while implementing our application.

Part one (the post you’re reading) concentrates on implementing the buffer and consumer components; the main aim is to become comfortable with ZIO types and learn how to create services and handle dependencies.

Part two covers the implementation of the broadcast and server components. We’ll address some concurrency challenges with ZIO building blocks and learn how to run a web socket server.

Part three is all about testing. We’ll use the ZIO testing library together with test containers.

Effects

If you’ve written Scala code before, you have probably encountered the term effects. They are usually described as a way of handling parallel computations, concurrency or I/O in our applications. An effect normally comes with an output type parameter, for example Future[String] or IO[Int], it may fail, and it may depend on resources or services in order to run.

Often, figuring out these properties means digging into and analyzing the implementation details of methods or values of such types. With well-written ZIO code, however, the type signature alone can sometimes tell us a lot more of the essential information about a given effect.

In the ZIO framework, an effect is defined as ZIO[-R, +E, +A] with 3 type parameters. This may look confusing at first, but within a second or two it turns out to be quite simple, since it stands for ZIO[Require, Error, Output]. The type describes a computation that needs things of type R, may fail with an error of type E, or may yield a successful output of type A.

Conversely, an effect without any requirements has the type ZIO[Any, _, _] and can run in “any” circumstances. Likewise, an effect that is guaranteed never to fail has the type ZIO[_, Nothing, _].

ZIO evaluates effects lazily, so creating a value of type ZIO doesn’t run the underlying code right away. To run the effects and execute the program, we have to bootstrap our components and set up the runtime environment. ZIO provides strong compile-time guarantees that help us structure and organize dependencies correctly. We’ll look at examples of such features later on.

The library offers many type aliases for the ZIO type. For instance, an effect that has no requirements and cannot fail has the alias UIO[+A], which takes only the type parameter for the successful output. Wherever we can, we’ll use type aliases in our application code so that you get accustomed to them.

Services

With the ZIO type introduced, let’s move on to our first service, a Buffer. There are a few key steps and good practices worth following when writing services, and I’ll go through them below. To make the reasoning clearer, I’ll also add the full types as comments in the code snippets throughout the posts.

Buffer definition

Step one is to create a service definition. A good practice – or even a “law” – is to always use traits for this and never express the service dependencies as environment requirements in the definition itself. What does this mean? If our Buffer implementation depends on another service, that dependency should not appear as a requirement in the service interface. Let’s write the service definition for our Buffer.

trait Buffer {
  def offer(message: Message): UIO[Unit] // ZIO[Any, Nothing, Unit] 
  def poll(): UIO[Option[Message]] // ZIO[Any, Nothing, Option[Message]]
}

Neither offer nor poll needs anything, and both are guaranteed not to fail, so we make no use of the R (no dependencies in the definition) and E type parameters and rely on type aliases to keep the notation short.

It’s customary to give the service a companion object with accessor methods. We use variants of ZIO.service[T] to retrieve instances of a particular service from the environment. This means buffer clients would have to summon the service with ZIO.service[Buffer] and then call, for example, offer on that instance. We can make this easier by defining accessor methods like this.

object Buffer {
  def offer(message: Message): URIO[Buffer, Unit] =
    ZIO.serviceWithZIO[Buffer](_.offer(message))

  def poll(): URIO[Buffer, Option[Message]] =
    ZIO.serviceWithZIO[Buffer](_.poll())
}

Clients can now call it with a shorter syntax, simply Buffer.offer(...), and the type of the resulting effect will carry Buffer as its environment requirement.

Buffer implementation

Next comes the implementation. Let’s examine the Buffer implementation and talk about the second good practice: service dependencies should always be passed in through the class constructor.

final case class BufferImpl(queue: Queue[Message]) extends Buffer {
  override def offer(message: Message): UIO[Unit] =
    queue.offer(message).unit *> 
       ZIO.logInfo(s"buffer message ${message.id}")

  override def poll(): UIO[Option[Message]] = queue.poll
}

This particular buffer implementation has a single dependency, declared in the class constructor. There’s little to explain – it should look familiar, as it follows the classic OOP style.

That one dependency is Queue from the ZIO library – an asynchronous in-memory data structure whose operations for offering and polling elements are non-blocking. We use the unbounded variant of the Queue, which you’ll see when the Queue gets instantiated. The queue serves as a buffer so that messages can be processed in a controlled way. It preserves message order (FIFO), which usually matters when dealing with Kafka.

Once the message is placed on the queue, we chain the execution of another effect, using the handy ZIO.logInfo for logging. To join these two effects we use the *> operator, a variant of flatMap that discards the result of the effect it’s applied to.

Buffer constructor

With the service definition and implementation ready, we can go on to the constructor – the final step in creating ZIO services. Hold on, though: Buffer is just a case class, so it already has a class constructor. True, but because we want to take advantage of the ZIO environment’s features, we’ll create so-called layers for our services. This is how it’s done.

object BufferImpl {
  // ZLayer[Any, Nothing, Buffer]
  val live: ULayer[Buffer] = 
     ZLayer(Queue.unbounded[Message].map(BufferImpl(_)))
}

The layer is typically defined in the companion object of the implementation class, which lifts it into ZLayer. Once more, a type alias expresses a layer that has no extra requirements (BufferImpl relies on no other service) and is guaranteed not to fail when it’s created. The ZLayer.apply() method accepts a ZIO effect as its argument, which makes it an effectful constructor. Thanks to this, we can cover cases where the service needs resources – such as a database connection – that may take a while to obtain.

Layers let us handle such initialization processes efficiently, something that’s hard to achieve with plain class constructors. Even the Queue.unbounded constructor is an effect, because it creates a queue inside the ZIO runtime and returns UIO[Queue[Message]]. Mapping over it gives us a BufferImpl instance.

Consumer

In the same way, the first step is to write a service definition for our Kafka consumer component.

trait KTailConsumer {
  // ZStream[Any, Throwable, Message]
  val consume: Stream[Throwable, Message]
}

This is where ZStream, another core concept of the library, comes in.

You can picture it as a pipeline that pulls data from a source, transforms it and emits some output. Streams can be combined, filtered, mapped or reduced to accomplish all kinds of tasks. They are highly declarative, so as soon as you recognize that an operation is “streaming” by nature, modeling it with ZStream semantics becomes easy.

Consuming messages from Kafka is a great example of such a use case, because it describes a process that receives input elements, processes them, emits some results and keeps doing so forever.

Just like ZIO and ZLayer, ZStream has three type parameters, and they mean exactly the same thing. Our consumer definition states that it must implement a stream that may fail with an exception and emits values of type Message. The consume value is only a description of a streaming pipeline; creating it doesn’t start any processing yet.

For the implementation we use zio-kafka – a library from the ZIO framework that eases the integration between ZIO and Kafka by offering ready-made stream definitions and common operators. It’s worth noting that ZIO provides a wide range of libraries for typical application operations, such as access to various databases, metrics and JSON support. You can find the complete list of official ecosystem extensions and utilities here.

final case class KTailConsumerImpl(topics: Set[String], consumer: Consumer)
    extends KTailConsumer {
  override val consume: Stream[Throwable, Message] =
    consumer
      .plainStream[Any, Array[Byte], Array[Byte]](
        Topics(topics),
        Serde.byteArray,
        Serde.byteArray
      )
      .map(record =>
        Message(
          record.offset.topicPartition.topic(),
          record.partition,
          record.offset.offset,
          record.key,
          record.value
        )
      )
}

This time the layer type expresses a specific dependency on the KTailConfig service, which is our application’s configuration service. We’ll build it in the following parts of the series, where we’ll also see how ZIO makes it easy to map application configuration files.

We depend on the Consumer instance from the zio-kafka library, which provides a method named plainStream. It returns a ZStream description of an endless stream of Kafka messages from a given topic, with the key and value serialized by the specified serializers.

To process the messages, we transform them with the map operation, turning Kafka records into domain Message objects. All that’s left is to define a layer for our consumer.

object KTailConsumerImpl {
  // ZLayer[KTailConfig, Throwable, KTailConsumer]
  val live: RLayer[KTailConfig, KTailConsumer] =
    ZLayer.scoped {
      for {
        config <- ZIO.service[KTailConfig]
        topics = config.topics.toSet
        consumer <- Consumer.make(
          ConsumerSettings(config.bootstrapServers)
            .withGroupId(config.groupId)
        )
      } yield KTailConsumerImpl(topics, consumer)
    }
}

To reach a particular service through the ZIO environment, we call one of the ZIO.service[T] methods. From then on, those services consistently show up as requirements in the type parameters (R) of our effects and layers. Here we retrieve the config service and use its values to build the actual consumer instance.

Scope

Did you spot that the snippet above uses another constructor, ZLayer.scoped? Whenever a Scope type appears in an effect’s requirements, it means the effect is accessing a resource that must be cleaned up properly afterwards. Put another way, Scope represents a resource’s lifecycle.

Here, the KTailConsumerImpl layer builds a Consumer instance via Consumer.make, which attaches a Scope responsible for shutting it down. If we dig into the zio-kafka code, we’ll find a spot containing the following (simplified) code.

ZIO.acquireRelease { 
    new KafkaConsumer[Array[Byte], Array[Byte]](...) 
} { consumer => consumer.close() }

The ZIO.acquireRelease method lets us create scoped resources and asks us to supply a release callback in which the cleanup happens. The Scope parameter is then propagated as a requirement.

In the layer we built for the k-tail consumer, the for comprehension expression is of type ZIO[Scope & KTailConfig, Throwable, KTailConsumerImpl]. Yet, as you may have noticed, Scope doesn’t appear among the layer’s requirements. This is thanks to ZLayer.scoped.

It eliminates the Scope requirement and ties the lifetime of the acquired resource to the lifetime of the layer itself. Consequently, the resource is released as soon as any workflow depending on this layer finishes running. In our example, that happens when the application shuts down.

Other ZIO types offer scoped constructors as well. Consider ZStream.scoped, and picture a pipeline that reads records from a database and runs some computations on them. We’d need a scoped database connection pool, but once we build a stream with it, the pool’s lifecycle is tied to the stream itself. That makes sense, since the connection pool should stay alive for as long as we’re processing records from the database.

From the user’s point of view, this mechanism is clear and simple. The ZIO runtime does all the hard work, guaranteeing that the resource gets closed gracefully whether our program succeeds, fails or is interrupted.

Program

Now let’s put together our first ZIO program, which consumes and buffers messages.

object Main extends ZIOAppDefault {

  private val program: RIO[Buffer & KTailConsumer, Unit] = for {
    consume <- KTailConsumer.consume
    _       <- consume.mapZIO(Buffer.offer).run(ZSink.drain)
  } yield ()

  override def run = program
}

The entry point of our application extends ZIOAppDefault, which supplies a default runtime configuration. We have to override the run method and plug in our program logic. The program’s first line defines a consumer stream. Next, with mapZIO, we place every message from the stream into a buffer. To turn the stream description into a runnable effect, we must call one of the run variants on the ZStream type.

We also have to specify a sink – the final destination for the stream’s outputs. Since here we don’t need to do anything with the results, we can use ZSink.drain, which discards the outputs and drains the stream until it finishes. It won’t compile yet, though. Notice the Buffer & KTailConsumer requirement in the type of our program effect. Compiling the code at this point produces an error like this:

Your effect requires services that are not in the environment.
Please provide layers for the following 2 types:

    1. ktail.Buffer
    2. ktail.KTailConsumer

Wouldn’t life be much easier if every compile-time error were this helpful?

Let’s follow the compiler’s advice and supply the dependencies to our program.

override def run = program
  .provide(
    KTailConfig.live,
    BufferImpl.live,
    KTailConsumerImpl.live
  )

Quite intuitively, we call the provide method of the ZIO type to pass in all the required services as layers. Our application is now ready to run.

Summary

This first post in the series introduced several fundamental concepts of the ZIO library. We looked at what the type parameters in Z types mean, how to write services, and how to compose our programs and supply their dependencies.

Any questions are welcome. Thanks!