Estimate project
← All posts

Creating a highly available, multi-regional scheduler on AWS

A walkthrough of building a multi-regional, highly available scheduler with AWS services that guarantees an at-least-once delivery policy for critical workflows.

17 min read
ArchitectureSoftware ArchitectureHigh AvailabilityAWSSystem ArchitectureSystem Design

Today’s distributed systems frequently span several regions to provide high availability and low latency. Such systems often need a way to schedule HTTP requests for execution at a later time, which powers workflows such as notifications, deferred API calls, or time-based triggers. When deployed across multiple regions, a scheduler like this has to deal with challenges such as consistency and resilience during failover. This article explains how to create a multi-regional, highly available scheduler with AWS services that guarantees an at-least-once delivery policy for critical workflows.

The tool also has to meet several further requirements:

  • It must be able to send thousands of requests per second with an accuracy of seconds (in the happy path scenario; delivering them somewhat later during service outages is acceptable).
  • At-least-once delivery is mandatory. On top of that, we should strive for best-effort exactly-once delivery.
  • Each request must be sent from the region in which it was created (to save costs and simplify data consistency) – our target endpoints can be regional, so the scheduler has to know which region a request should be sent from.
  • It must support region failover – we have to be able to drain the traffic leaving the scheduler in one region and move it entirely to another region, so that all schedules are delivered from there. (e.g., although particular schedules were created in us-east-1, during the region failover they should be delivered in us-east-2). A region failover doesn’t mean shutting down every service instance in that region.

Why not rely on the Event Bridge Scheduler?

It was certainly our first idea, since it looked like a ready-made solution that would fit our use case, but a closer look revealed these problems:

  • It doesn’t support HTTP endpoints as scheduler targets. A Lambda could serve as an intermediate layer, which isn’t a huge problem, but it does add complexity to the architecture
  • More importantly, EventBridge scheduler schedules aren’t replicated or synchronized across regions, so region failover isn’t supported out of the box. Schedules created in a particular region would also have to be delivered from that region.

Overall, after comparing the EventBridge Scheduler approach with our requirements, we concluded that to make this architecture meet them, we would have to surround this AWS service with many additional puzzle pieces. That could make the whole setup very complex and possibly flaky. Given all that, it would hardly qualify as an out-of-the-box solution. A custom tool, in contrast, would offer more flexibility and let us extend the set of supported features as our needs evolve. That’s why we chose to build our own scheduler and deploy it in a multi-AZ and multi-regional setup. The following sections present the high-level architecture of this idea.

Persistency and a simple API

To begin with, we want to store schedules in a database and keep all the information there, which makes management and auditing easier.

We picked DynamoDB, because a NoSQL database would offer good performance and scalability, and with a well-planned data structure we should be able to get by without complex queries.

On top of that, we opted for the DDB global table feature to take advantage of active-active replication between our regions in support of region failover (more on that later).

To prevent hot partitions, scheduleId serves as the partitionKey. Its value can be generated whenever a schedule is created.

With a database holding all the schedules in place, we can expose an API for performing CRUD operations on them.

Cache

In principle, we could finish the article right here by building a Sender component that queries the DynamoDB table every second (which would need a dedicated GSI (Global Secondary Index) value for fetching all schedules due in a particular second) and just sends them to their target endpoints.

Simple as it sounds, it’s far from the best idea, because querying a DynamoDB table every second would put a heavy load on it and raise the total cost of the architecture. A much smarter approach is to split the schedules into buckets, e.g. 10-minute chunks, load them into a cache ahead of time and, when the moment arrives, look them up in the cache and send whatever is due.

A Sender reading schedules straight from the database compared with a Sender relying on a cache layer.

This approach also brings all the more obvious advantages of caching, such as lower latency, reduced database call costs, better scalability etc.

Note that these 10-minute buckets have to be regional, meaning each one holds schedules from a single region only.

Two buckets spanning the same time window in two regions hold different regional schedules.

Which service can act as the cache layer?

We quickly ruled out an in-memory cache, because our service has to run in a multi-region setup and, more importantly from the caching point of view, a multi-instance one (within a single region), so the cache has to be shared between the regional instances to keep them in sync. (i.e., we don’t want every instance to send all the schedules, which could happen if each instance had its own cache).

ElastiCache initially looked like our best option, but in the end we went with an SQS queue. This (perhaps unexpected) choice comes down to the following reasons:

  • Messages in an SQS queue are persistent (the same could be achieved with ElastiCache with Redis, but it would need extra puzzle pieces). So once a schedule lands on the CacheQueue, we can rely on its at-least-once delivery policy and be confident that the Sender will eventually process the schedule.
  • Using an SQS queue as the cache allows us to run more than one Sender without having to synchronize them.
  • If the Sender fails unexpectedly for any reason, we can rely on the queue’s retry policy.
  • SQS offers a feature called message timers, which we chose to use so that schedules can be executed at a precise moment even though they were cached in advance.

Sender

Nothing too sophisticated here: the Sender component just polls the CacheQueue for messages and sends the schedules as soon as they show up. Several Senders can be run to boost performance (for simplicity, we’ll use a single one). Once delivered successfully, the schedules are marked as sent (their GSI gets removed and their status is updated) and saved in the database.

Reader

At a high level, this component’s task is to “wake up” at a given time, read from the database all schedules that make up a particular bucket, and load them onto the CacheQueue (after first computing the delay).

In practice, the component looks like this:

The Reader component, made up of the ReaderTaskProducer, a FIFO queue, and the ReaderTaskConsumer

ReaderTaskProducer

This component has a single job: to wake up every ten minutes (slightly before a given bucket begins – so for the 12.10 AM - 12.20 AM bucket it might wake up at, say, 12.07 AM), create a ReaderTask for that bucket and place it on a FIFO queue. A ReaderTask is a simple command telling the Reader that a particular regional bucket should be loaded from the database and put into the cache.

This is a good moment to stress that our service can run multiple instances at the same time within one region (or even one AZ). We have to ensure that no bucket or schedule ends up in the cache more than once.

The sole reason for splitting the Reader into two components (ReaderTaskProducer + ReaderTaskConsumer) is to place an SQS FIFO queue between them. To get exactly one ReaderTask per bucket on the queue, the SQS FIFO deduplication mechanism must be used (the deduplicationId is a bucket id that is identical across all service instances).

ReaderTaskConsumer

This part of the Reader polls the FIFO queue, and each ReaderTask message that appears there gets processed by just one ReaderTaskConsumer. (More precisely – ReaderTaskConsumers belonging to other service instances won’t process that record. This is achieved with the VisibilityTimeout SQS feature)1.

You don’t need to read the footnotes to grasp this architecture at a high level. Some concepts, however, are explained there in more detail, so they certainly help in seeing the complete picture.

As for the processing itself – the component queries the database via the GSI to fetch every schedule in a given bucket. It then computes the delay for each one before placing it on the CacheQueue.

GSI shape

nextAttemptTimestamp: 2025-10-01T17:16:04Z 
gsi: us-east-1#2025-10-01T17:10:00Z

NextAttemptTimestamp is “rounded” down to the closest 10 minutes to produce the GSI, which represents the 10-minute buckets.

Retries

What happens when the delivery of a schedule fails for some reason? In line with best practices, a retry policy kicks in.

To handle retries, the Sender component increments the schedule’s attempt counter, computes the next delivery time (exponential backoff with jitter works well here) and, on that basis, updates the record’s GSI value (when necessary) and saves it in the database. As a result, the schedule moves to another bucket and gets processed in one of the upcoming bucket iterations2.
If the maximum number of attempts has been used up, though, the schedule is marked as skipped and an alert is raised (this article won’t go into how the alerting mechanism was implemented).

Flow Summary

  1. ReaderTaskProducers create ReaderTasks and place them on the SQS queue. Only a single message actually lands there.
  2. A single ReaderTaskConsumer handles the ReaderTask.
    a. The bucket is fetched from the DynamoDb table.
    b. The delay for each schedule is computed.
    c. Every schedule is placed on the CacheQueue using the message timers feature.
  3. Once the delay has passed, the Sender reads the schedule and sends it to its target endpoint.
    a. If it succeeds, the schedule is marked as delivered.
    b. If it fails, the retry policy is applied.

Region Failover

As noted at the start of this article, a key challenge of this solution was supporting the ability to “switch off” any region at any moment and, from then on, deliver all of its schedules from another region (target endpoints are regional).

It’s worth stressing that in this architecture, all the work is done by background tasks. So routing all CRUD API calls to the service in another region isn’t enough, because schedules created earlier would still be sent from the possibly faulty region.

Therefore, to drain traffic from a region completely, every scheduler instance in the failovered region has to pass the switch-off command to all of its components so that they stop working.

The same goes for the takeover region (the region that sends the failovered region’s schedules during the outage) – the components of its service instances must be aware of the mode change.

We chose to keep the current region failover status in an SSM Parameter Store and poll it for value changes to detect a region failover3.

To see how this requirement is supported, let’s begin with the changes to the architecture:

FailoverPoller

This component polls the SSM parameter value and, as soon as it notices a change, forwards that information to the Orchestrator.

ReaderTaskProducer

As explained before, the buckets fetched from the database are regional. A ReaderTask is a command telling the ReaderTaskConsumer which bucket to process. During a region failover, the ReaderTaskProducer has to stop, so that the FIFO queue receives no more ReaderTasks for this region. As a result, the schedules won’t be put into the cache and therefore won’t be sent from this region.

In the takeover region, an extra ReaderTaskProducer instance can be started alongside the default one to generate the commands for the failovered region. In turn, the ReaderTaskConsumer puts the schedules into the cache for the Sender to handle, enabling the complete region switch4.

Sender

The Sender has to be given a set of regions whose schedules it should process (send). By default, the component handles schedules from the region it runs in, but once a failover occurs, it should stop sending right away. In addition, it should purge every remaining message from the CacheQueue (the schedules will be handled in the other region), so that they aren’t sent a second time after the failover ends and the Sender goes back to consuming from the queue in its default mode.

The Sender in the recovery region, in turn, just needs to be instructed to send schedules from its own region + from the failovered region5.

Orchestrator

To let the ReaderTaskProducers and Sender operate in a particular mode that can be changed at runtime, an Orchestrator component needs to be added.

The Orchestrator owns the instances of the Sender and ReaderTaskProducers components. Thanks to that, when failover information reaches the Orchestrator, it can shut down the running components and, if needed, start new ones in another mode.

Region failover example

Suppose the us-east-1 region has to be switched off due to an outage (or just for testing, which, incidentally, is also a good practice).

  1. The regional SSM parameters get updated. us-east-1 is evacuated, and us-east-2 becomes the recovery region.
  2. The FailoverPoller notices the change in the parameter value and forwards it to the Orchestrator.
  3. us-east-1
    3.1 The Orchestrator stops the ReaderTaskProducer that generated the us-east-1 ReaderTasks. It also instructs the Sender to purge messages for us-east-1 schedules in order to drain the queue.
    3.2 The cache stops receiving new schedules.
    3.3 Every schedule already in the cache is purged as soon as it becomes visible to the Sender, so nothing is sent from this region.
  4. us-east-2
    4.1 The Orchestrator launches a new ReaderTaskProducer instance responsible for the us-east-1 buckets. It also instructs the Sender to send schedules from both the us-east-1 and us-east-2 regions6.
    4.2 ReaderTaskProducers generate ReaderTasks for the us-east-1 (beginning, e.g., a few days back) and us-east-2 regions.
    4.3. ReaderTaskConsumers fetch buckets from the us-east-1 and us-east-2 regions and load all those schedules into the cache.
    4.4. The Sender sends every schedule from both regions.

Region recovery example (failover is switched off)

  1. The regional SSM parameters get updated. Everything returns to normal.
  2. The FailoverPoller notices the change in the parameter value and forwards it to the Orchestrator.
  3. us-east-1
    2.1 The Orchestrator launches the default ReaderTaskProducer and instructs the Sender to begin sending schedules from us-east-16
    2.2 ReaderTasks for us-east-1 are generated (beginning, e.g., a few days back), and the schedules land in the cache.
    2.3 The schedules get sent.
  4. us-east-2
    3.1 The Orchestrator stops the ReaderTaskProducer instance that handled the us-east-1 buckets. It also instructs the Sender to stop handling us-east-1 schedules and to send only those from us-east-2.
    3.2 Only us-east-2 ReaderTasks are generated, and those schedules are loaded into the cache.
    3.3. The Sender handles the us-east-2 schedules. From then on, any us-east-1 schedules still in the cache are purged from the queue.

The solution outlined above has been built and successfully rolled out to production, proving that it can handle the required workloads reliably and efficiently. It delivers the expected scheduling precision, high availability, and fault tolerance, satisfying the needs of the entire system.

Alternative Approaches and Considerations

Several decisions we faced while designing this architecture illustrated perfectly the rule that holds for designing any system – trade-offs are everywhere…

Speaking of trade-offs, before implementing this architecture we reviewed the idea and tried to come up with improvements. For one, we felt it would be good to decouple the service further by relying on existing AWS services rather than implementing every component within a single ECS task. Also, at first glance the components based on polling looked less than ideal.

That led us to an alternative – the Reader component could also have been built with the EventBridge scheduler + a lambda target (in place of ReaderTaskProducer + SQS FIFO + ReaderTaskConsumer), but to support region failover, the lambda would need to run in a particular mode depending on the SSM parameter value (possibly calling the SSM API to read the value on every run). The same is true of the Sender component, which could likewise be swapped for a lambda. That’s why we concluded that an Orchestrator owning the sub-components and able to switch their modes easily is the option we want to go with.

That said, the ReaderTaskConsumer could be replaced with a lambda almost seamlessly, and we’re considering making that change in the future.

Either way, we’d be glad to hear your ideas for improving this architecture or for tackling the whole problem in an entirely different way!

Notes


  1. In the happy path scenario, only one ReaderTaskConsumer processes the message. If something goes wrong and the VisibilityTimeout expires, another instance will pick it up, so messages may in fact land on the CacheQueue more than once. The service should still do its best to send everything exactly once, but the at-least-once delivery principle takes precedence. ↩
  2. In practice, the logic has to differ slightly for schedules that are going to be retried very soon. If the next attempt falls within 15 minutes from now, or exactly at that point (15 minutes being the maximum delay SQS supports), they are placed in the cache directly and saved to DynamoDB with a skip flag=true. That way a schedule should always land in the cache if it still belongs to the same bucket (e.g., the next attempt is due in 5 seconds). At the same time, if it moves to the next bucket (e.g., the next attempt is 12 minutes away), it shouldn’t be duplicated by the ReaderTaskConsumer handling that next bucket, because the consumer filters such a schedule out based on the skip flag. ↩
  3. SSM parameter store parameters are regional and aren’t replicated out of the box. Therefore, in practice, every supported region needs its own dedicated parameter, used by the service instances running in that region. The regional parameters must be kept in sync with one another.
    Furthermore, one of the regions must be explicitly designated as the takeover region for the failover. This can be done in the parameter itself or, alternatively, somewhere in the application configuration.
    Updating the parameter values can also be automated using an event-based approach, which also keeps them automatically in sync across regions. That part of the architecture, however, is outside the scope of this article and isn’t covered here. We only look at what happens after the value changes and the scheduler learns about the failover. ↩
  4. This extra ReaderTaskProducer instance begins by going through all past buckets, from e.g. a few days ago up to now, to create ReaderTasks for each of them and feed the cache. This reduces the risk of losing any schedule, since the takeover region should handle them (if, e.g., an SQS outage triggered the failover in the first place, schedules due between the outage and its detection might not be sent before the region switch is complete). As noted before, a schedule’s GSI is removed once it has been sent successfully, so the takeover region won’t fetch such records from the database. ↩
  5. Once a failover happens and a schedule is “moved” to a takeover region, it is treated as though it had belonged to the takeover region all along, i.e. if the retry policy applies, subsequent delivery attempts take place in that region (unless it also becomes unavailable). ↩
  6. A delay can/should be introduced here before starting the components, to be extra certain that nothing is sent twice. This matters especially for reducing the risk of the same schedule being sent from several regions at almost the same moment, as that can cause data inconsistencies. As stated in the requirements section, delivering schedules later than expected is acceptable when needed, so delaying a few schedules because of region failover/recovery is fine for us. ↩ ↩