Oh no, a timeout...
Is your application running into timeouts before it can respond? Here is what you can do about it.
Has your application ever had trouble with responses that time out?
Timeouts can have many different causes. Server misconfiguration, bloated request or response payloads, or a slow, congested network can all lead to them. Your system may also be under heavy traffic and simply unable to handle requests in time.
Requests that start long-running processes are another frequent source of trouble. When handling the action takes longer than the configured timeout threshold, the client gets an error with a 408 status code. Depending on how your app is configured, the process is then either aborted or eventually completed. In the second case, although the action did finish, the client has already received an error response and won’t learn that it succeeded.
At times, fixing a timeout may come down to optimizing the backend task – for example, tuning SQL queries or switching to more efficient algorithms. Performance tuning isn’t always easy, though, and some processes just need a certain amount of time to finish.
So what are our options?
Fine-tuning the configuration
When you can estimate how long the task will take, the simplest fix may be to raise the timeout threshold for that specific request type. Usually, changing the value in one place isn’t enough: the client, the server and any proxies (e.g. load balancers) all need updating.
The trouble is that we can’t always clearly pin down the correct timeout value. Depending on the input data, the backend task may need more or less time. Pick a value that’s too low, and some requests may still time out. Pick one that’s too high, and you face plenty of downsides. Your application will have to deal with more concurrent HTTP connections, which can mean higher resource usage and server load. It may also leave you more vulnerable to DDOS attacks such as Slowloris, which rely on the attacker being able to keep many parallel connections open.
If tuning timeouts for synchronous requests proves problematic, there is always the option of processing tasks asynchronously.
The asynchronous handling of long-lasting tasks
If waiting for processing to finish leads to timeouts, we can just stop waiting.
Our application could instead tell the client that the request has been registered successfully and that processing will begin shortly.
To start, the application should store the intention to run the task and then reply with a status code designed specifically for this situation: 202 Accepted. A background worker can then regularly pick up new tasks from the store and start processing them. Once a job succeeds (or fails), it should update the stored record with the processing result or the cause of the failure. The key thing is to ensure the same job is never started twice concurrently. This can be done by electing a single leader responsible for processing or by using some sort of locking mechanism.
The response to the original request should contain a Location header telling the client where to find the results once they’re ready. Typically, it points to a different endpoint of the application. While the task is still running, that endpoint should keep responding with 202 Accepted, and the response body may optionally include progress information. Once the task is done, the following response should deliver the computed results with a 200 Ok or 201 Created status. If something went wrong, the endpoint might return 404 or even 410 Gone.
In this flow, the client is responsible for polling the resource endpoint until the task completes. The Retry-After header lets us suggest how long it should wait before trying again. We can also make the solution more responsive by pushing updates from the server – for example, by exposing a Server-Sent Events endpoint that tells the client when the job has finished.
In terms of software architecture, this async workflow implements the inbox pattern using HTTP endpoints.
An example
Here’s an example. Our application exposes 2 endpoints: POST /item, which starts creating an item, and GET /item/{id}, which returns the job’s status and results.
For a more robust API, we could make the POST endpoint immutable. When the client sends another request with identical input data, we can just return the location of the item that’s already being created (or has even been created already) – in our case, simply a URL with the same item id. While the task is in progress, any attempt to fetch the results returns 202 Accepted.
Once the item has been created, the worker process updates the record in the items table, and the endpoint begins returning the results with the 201 Created status code.
Summary
With this article, I wanted to offer some insight into building asynchronous processing on top of HTTP endpoints. I hope it proves useful!