Estimate project
← All posts

Running your ML PoC on GCP Compute Engine

This article explains how to deploy your ML model fast, with restricted user access and the ability to scale to zero. The tutorial uses Compute Engine on the Google Cloud Platform.

7 min read
Machine LearningCompute Engine

Plenty of excellent tools exist for orchestrating ML projects, such as KubeFlow, Airflow combined with MLFlow, or offerings from cloud providers (for example Google Vertex AI, SageMaker, and Azure AI), yet all of them are designed for large-scale systems. What should you do if you need a low-cost way to serve your ML demo with little, irregular traffic? This article explains how to deploy your ML model fast, with restricted user access and the ability to scale to zero. The tutorial uses Compute Engine on the Google Cloud Platform.

Why Compute Engine?

GCP offers several ways to host applications: from the simplest, Cloud Run, through Vertex AI, a solution built specifically for hosting ML models, up to Compute Engine, which gives you self-managed infrastructure. So why could Compute Engine be the right pick?

Cloud Run

Cloud Run is a serverless Platform as a Service (PaaS). It can scale to zero and requires hardly any management. It’s ideal for cost-effective applications, since it runs only while an HTTP request is actively being handled. Cloud Run, however, has no GPU support, which rules it out for most ML projects.

Vertex AI

Vertex AI is a solution built specifically for managing ML models. It makes versioning and serving your ML models simple. Vertex AI Endpoints, however, cannot scale down to zero. A feature request for this has been open since 2021, but so far the feature has not been delivered.

Compute Engine

Compute Engine is an Infrastructure as a Service (IaaS). It offers far more customization than Cloud Run, but it also needs more configuration. With the Managed Instance Group (MIG) service, we can configure auto-scaling for our VMs. In theory, the auto-scaling feature should allow scaling down to zero, but I wasn’t able to make it work. Still, we can control MIG scaling programmatically and get scale-to-zero behaviour for our GPU VM.

Alternatives

Another option could be a HuggingFace space. It supports scaling to zero and offers GPUs. HuggingFace spaces, however, require your source code to be public, which can rule them out for many projects.

Solution architecture

The solution is based on an inexpensive frontend instance that is always on and a costly GPU instance that the CPU instance launches whenever traffic shows up.

Figure 1. Solution architecture.

This gives us high availability, the ability to scale to zero, and GPU acceleration. Every instance is managed by the MIG and sits behind the load balancer. For security, Identity Aware Proxy (IAP) acts as a single control point for managing who can access the web applications.

Implementation details

Artifact registry

The frontend app (which runs only on the CPU VM) and the backend part (which runs on the VM with a GPU) are both docker images. On top of that, the frontend docker image needs to include the scaling functions covered in Scaling to and from zero. This is why the extra Service Account Roles described in Service Account are required.

Once the docker image is built, it needs to be tagged and pushed to the artifacts registry. Detailed step-by-step instructions are available here.

Service Account

The frontend app needs a custom service account so that it can scale GPU instances to and from zero. You need to create a custom role with some or all of the permissions below:

  • Compute.autoscalers.create
  • Compute.autoscalers.delete
  • Compute.autoscalers.get
  • Compute.autoscalers.list
  • compute.autoscalers.update

For the backend part, you should create a custom role with the permission

artifactregistry.repositories.downloadArtifacts.

Instance template

Every Instance group needs an instance template from which the VM is created. Since the ML PoC is made up of two instance groups (CPU frontend and GPU backend), two templates are required.

Frontend instance template
Configure the values below:

  • Machine configuration: E2, for instance.

  • Container: Container image -> europe-west4-docker.pkg.dev/<team>/<repo>/<frontendimage_name>:<tag>

  • Boot disk:

  • Identity and API access: a Service account with the Editor role and a role with

  • Firewall: Allow HTTP traffic

Backend instance template
Configure the values below:

  • Machine configuration: pick the GPU and an instance type, N1 for instance

  • Boot disk:

  • Firewall: Allow HTTP traffic

  • Advanced options: Management -> Automation -> startup-script

# install nvidia drivers
sudo /opt/deeplearning/install-driver.sh
# setup access to the google cloud docker registry
VERSION=2.1.2
OS=linux
ARCH=amd64
curl -fsSL "https://github.com/GoogleCloudPlatform/
docker-credential-gcr/releases/download/v${VERSION}/
docker-credential-gcr_${OS}_${ARCH}-${VERSION}.tar.gz" \
| tar xz docker-credential-gcr \
&& chmod +x docker-credential-gcr && sudo mv docker-credential-gcr /usr/bin/
docker-credential-gcr configure-docker --registries=europe-west4-docker.pkg.dev
docker pull europe-west4-docker.pkg.dev/<team>/<repo>/<image>:<tag>
docker container prune command

docker run -d --gpus=all -v <src>:<dest> -p 11434:11434 --name 
<imagename> europe-west4-docker.pkg.dev//<team>/<repo>/<backend_image>:<tag>

The startup script installs the Nvidia drivers, sets up the artifact registry so docker can access it, and then pulls the docker image and runs it on port 11434. Configuring the artifact registry for docker credentials is required because the operating system isn’t the Container Optimised OS; with that OS, you could skip this step. The official guide to authenticating Docker with Artifact Registry is available here.

Instance Managed Group

Create an instance group for every instance template.

Frontend instance group
Add a new health check that uses the right port.

Set up port mapping for incoming requests.

Backend instance group
Disable autoscaling. Scaling of the GPU VM will be handled programmatically.

Add a health check for the GPU VM.

Set up port mapping for the backed service.

External Load Balancer

The external load balancer spreads incoming load from the Internet to the frontend instance. The load balancer’s frontend service should be given a static IP address and a valid SSL certificate for the HTTPS protocol. Google offers a tutorial explaining how to create the certificate for development use. The backend part should point to the frontend instance group.

Internal Load Balancer

The internal load balancer takes care of traffic coming from the frontend application. HTTP serves as the endpoint protocol. The frontend part should get a static internal IP address, and the backend part should point to the backend instance group. Write down the frontend IP address and port, since the frontend instance group will use this address to talk to the backend instance group.

For the frontend part, you need to create the Proxy-only subnet.

Raising the default load balancer timeout is important, particularly for ML models that run for a long time. The default of 30 seconds frequently isn’t enough for model inference.

Firewall rules

For the entire setup to work, add these firewall rules to your Virtual Private Cloud (VPC):

Direction:
ingress

IP addresses:
130.211.0.0/22
35.191.0.0/16

Ports:
<frontend web-app port>
<backend port>

Direction:
ingress

IP addresses:
<internal load balancer subnet IP address>

Ports:
<internal load balancer frontend port>

Identity Aware Proxy

IAP guards access to applications hosted on Google Cloud. It lets you whitelist the email addresses that are allowed to use your web application. To set up IAP, we need to create the External Load Balancer Backend Service and set up the VPC firewall rules. Google offers an official step-by-step guide to configuring it.

Scaling to and from zero

Implementing VM scaling programmatically is well documented elsewhere, so I’ll skip the details.

A scale-up should be requested whenever traffic shows up on the CPU instance. This can be done in the middleware, provided the request is not a health check.

@app.middleware("http")
async def spawn_gpu(request: Request, call_next):
    response = await call_next(request)
    if "/health" not in request.url.path:
        # request GPU
        scale_up()
        global LAST_ACTIVITY_TIME
        LAST_ACTIVITY_TIME = time.time()

Scaling down can be triggered by inactivity. When the server has had no traffic for longer than GPU_TURN_OFF_TIME, the GPU instance can be scaled down.

class GpuActivityMonitor(Thread):
    def __init__(self):
        Thread.__init__(self)
        self.daemon = True
        self.start()

    def run(self):
        while main_thread().is_alive():
            if time.time() - LAST_ACTIVITY_TIME > GPU_TURN_OFF_TIME:
                log.info('scaling down GPU due to inactivity')
                try:
                    scale_down()
                except Exception as e:
                    log.error(f'Error scaling down GPU: {e}')
            time.sleep(10)

Both scaling up and scaling down should be implemented as thread operations, and before scaling in either direction we should check whether other threads are running. With a docker image of roughly 0.5 GB, scaling the GPU instance up from zero takes around 3.5 minutes.

On a Google VM, we can rely on default credentials, because the VM is already authenticated. If the instance runs outside Google Cloud, though, we need to authenticate first using a service account key or workload identity federation.

Summary

This article covered deploying and hosting a low-cost version of your ML PoC. For large production systems, I suggest ML orchestration tools such as KubeFlow, AirFlow + MLFlow, ZenML, or cloud provider offerings such as Vertex AI, SageMaker, and Azure AI. Feel free to contact me with any questions – I’ll gladly help.