How Should You Approach ECS Autoscaling?
Get a grip on ECS autoscaling: find out how to optimize CPU usage, cope with traffic spikes and tune your clusters for top cloud performance.
Whenever cloud services come up in conversation, the word “scalable” is almost guaranteed to appear. And the claim that “cloud services are scalable” is certainly true. They simplify scaling and frequently handle it out of the box. Sometimes, though, we need to take the reins and specify precisely how scaling should behave. That was the situation our team faced when scaling the microservices running in our ECS cluster. Let me share how it went.
Product overview
To keep it short – we are a small team of three developers who split backend, frontend and ops duties between us while building a data platform. Its traffic isn’t constantly high. Rather, we regularly get traffic spikes when clients run large workloads, such as uploading lots of files. From the business point of view, our goal is to process these workloads as fast as we can so users can begin working with the freshly uploaded data.
The system is asynchronous and message-driven, with responsibilities distributed among many microservices. MSK (AWS-managed Kafka) serves as our durable message log, RDS as our primary relational database, and OpenSearch (AWS-managed Elasticsearch) as the main engine behind data reads in our web UIs.
Initial setup and problems
Initially, every one of our services ran in one ECS cluster with a shared autoscaling policy. Scaling depended on memory usage – once memory usage exceeded 80%, the service scaled out, and once it fell under 20%, it scaled back in. This default was also used in other clusters hosting our client’s other products.
As more customers came on board and traffic spikes became more frequent, it turned out our setup wasn’t good enough. The metrics revealed that memory usage wasn’t actually an issue for our microservices; what we were hitting was CPU contention. None of the services had hard memory or CPU limits, so one container task could consume every resource on an EC2 instance – particularly at traffic peaks. As a result, other services occasionally didn’t respond to automated AWS health checks and ended up in restart-and-retry loops.
On occasion, when we knew a traffic peak was on its way, we tried to get ahead of it by manually adding EC2 instances and tasks to the cluster. It was a quick fix, but it also meant additional work and extra stress. It helped sometimes, yet it didn’t address the underlying problems. CPU contention remained, we didn’t know how many extra tasks of each kind to spin up, and we had no time to watch the results and adjust on the fly.
Determine scaling factors
Eventually, we found the time to deal with our autoscaling problems. Step one is figuring out the right scaling factors for your deployment, and the best way to do that is to look at your metrics and understand your code. As I said before, we already knew CPU was a bottleneck, so it became our first scaling factor. The second, typical for message-driven systems, was processing lag on the event stream – for us, the consumer group lags on a few core Kafka topics.
We also noticed that some services mostly handle HTTP requests that boil down to simple database reads and writes. They need fewer resources than the services doing CPU-intensive work, such as image processing, which depend more on consumer group lag.
We chose to separate these two groups into two ECS clusters: one for the CPU-intensive services and one for the lightweight ones. That lets us use different autoscaling strategies and monitor each cluster on its own.
Determine resource limits
CPU contention can be solved by putting hard resource limits on ECS tasks. That, however, brings some important trade-offs and consequences. To begin with, resources go to waste when a task is idle. Moreover, it’s very hard to pick good values for hard limits – you have to know, or even measure, how much CPU and memory a given service requires under various workloads.
We went with soft limits for our ECS container tasks instead. What’s the difference? With soft limits, tasks that have the same resource ratio, for example 0.5 vCPU, can share those resources among themselves when needed. Say two tasks each have a soft limit of 0.5 vCPU, and one sits idle while the other does CPU-intensive work: the busy task can temporarily borrow the idle task’s unused 0.5 vCPU. Once the idle task wakes up, it just gets the CPU it needs – without contention or competition. This leads to better resource utilization overall and doesn’t demand the precision or rigid configuration that hard limits do.
Regular cluster
As I mentioned, we divided our services between two ECS clusters. The “regular” cluster hosts services that primarily handle HTTP traffic and scale according to CPU usage. In practice, this cluster seldom spins up EC2 instances beyond its initial minimum. Under heavy workloads, a few core HTTP services supporting the most critical business processes, such as those dealing with file uploads and parsing, scale out by a couple of instances, and that’s typically all.
Most of these services run relational database operations against our RDS instance, which the microservices share, with every service owning its own RDS schema. During scale-out, we occasionally reached the database connection limit, because every task keeps its own connection pool. We fixed this by adding an RDS Proxy to our environment, which manages and distributes connections more efficiently. The downside is a slight increase in latency. To compensate, a few core services that are critical for workload performance still connect to RDS directly, while the less important or less frequently used services route their database traffic through the proxy. Since then, we haven’t had any connection problems. At some point in the future, we might scale our RDS instance vertically, or analyze the load and possibly make use of RDS read replicas.
Compute cluster
The second cluster runs our CPU-intensive services and scales mainly on consumer group lag. By default, every service begins with one running task. This is effectively a standby mode – we can process some Kafka messages without scaling out when the workload is small. This cluster scales more aggressively than the regular one.
In other words, when lag stays above a certain threshold for a set period, services scale out in larger steps (which vary per service), going from 1 task instance to 5, then to 10 and so forth. Because we don’t apply any fine-grained partitioning strategy to our Kafka topics, messages can be expected to spread fairly evenly, so partition assignment during scaling is not a concern. The maximum number of task instances is therefore capped at the number of partitions per topic, which for us is 30. Scaling down is simple – once consumer group lag falls to 0, we go back to a single task instance.
Fine tuning
To settle on the right scaling thresholds and step sizes, we tested many different configurations to see how the platform reacted. There’s honestly no shortcut – it requires time and hands-on testing with real workloads. We ran load tests in our staging environment and later in production too.
At this point, good monitoring is essential. The key metrics we use to gauge scaling effectiveness are CPU and memory utilization – both cluster-wide and per service – the number of EC2 and task instances, response times and consumer groups lag. It’s also worth keeping an eye on your single point of failure,, such as databases, since they may face a heavier load than your team is used to seeing when the platform scales out. In our case, we found and optimized a number of SQL calls in services that were driving up CPU usage on the shared RDS instance.
Ultimately, we arrived at a setup that served us well. Under heavy upload workloads, with large batches of files sent to the platform, our compute cluster managed to keep up with all the asynchronous processing and delivered all the results almost as soon as the uploads completed. Fine-tuning relies heavily on the nature of the workflow you want to scale. Each project is different, so understanding your system thoroughly from the start is key.
Summary
Including fine-tuning and load testing, the entire process took more than a month. We learned a great deal about our services and about ECS autoscaling itself. From the business side, we improved the platform’s overall performance, which directly affects how our clients use the product. As developers, we also got peace of mind: traffic peaks are less stressful, and we no longer have to jump in and try to fix things by hand.
In my view, the two most crucial steps in ECS autoscaling work are finding the right scaling factors, which means understanding your workload, and fine-tuning the scaling parameters. Every product is different and some things don’t generalize well – there’s no one-size-fits-all answer. Don’t rush, trust your metrics and load test results, and you will eventually land in the right place.
Thanks for reading. I hope this short article proves useful to you.