Applying CT/CD principles in machine learning projects – how?
Taking a machine learning project from its Proof of Concept phase to Continuous Training & Deployment.
The problem
Handling invoices and bank transfers in big companies is a complicated process. With numerous contractors and customers, errors in transfer titles happen easily, and sometimes a single transfer covers several invoices or just a portion of one. Simple rules fail in such cases, so bank transfers are matched to invoices manually, under human supervision.
This approach has a number of downsides, the biggest being that the process doesn’t scale. Suppose the company currently has one employee whose task is pairing bank transfers with invoices. If the company expands, taking on 2x more subcontractors and gaining 2x more clients, then – provided the process and work efficiency stay the same – it will need 4x more staff. Automating the process solves this, and artificial intelligence can help with that.
Within our in-house enterprise financial management system, we carried out a Proof of Concept project for an AI system meant to automate this manual process.
Where the project stood
The project we took over had finished its Proof of Concept stage.
Source: DALL-E
That is, code already existed for analysing data, training, evaluating, and deploying the machine learning model. These features lived in separate scripts that had to be run by hand. On top of that, the training data was a static snapshot of the system’s state, kept in the repository alongside the code as the PoC’s training data. Whenever new training data arrived (which happens monthly), updating the model would involve plenty of manual effort: fetching the training data and converting it into a particular format, running training and model evaluation, picking the best model and deploying it to production by hand. And the same again every month… For a single project it’s still manageable, though frustrating. With more projects, keeping things this way becomes impossible.
What the data looks like
The training data is made up of invoice and bank transfer metadata, together with information about which invoice each one is linked to. This structured data is tagged by hand in the internal system by the administration and finance department, which means the training examples are labelled by humans and of high quality. Both invoices and transfers enter the system monthly. The data lands in BigQuery in real time, and BigQuery will act as the source of training data for the training process.
It’s a textbook case of a problem where periodically arriving training data can be used to make the machine learning model more effective while keeping prediction quality high and the model up to date.
Planning
So that we don’t have to retrain the model by hand, we can automate the process with MLOps techniques – in particular, Continuous Training and Continuous Deployment (CT/CD) pipelines. Here, we’ll think about how to split up the AI model development process and how to automate it.
Splitting the process
The first step that has to run on a schedule is gathering training data from the data sources. In the system we use, bank transfer and invoice metadata reside in BigQuery. So we can write a function that pulls the data from BigQuery, transforms it, splits it into training and test sets, and saves them in the appropriate cloud bucket.
Next comes training and evaluating the machine learning model. In the previous step, we decided the training data would live in the cloud. That lets us assume the machine learning pipeline will fetch data from the specified cloud bucket, then train, evaluate, and store the fine-tuned machine learning model. Automatic deployment will require keeping metadata about trained models in a model registry. This allows models to be compared against the one currently deployed. If the freshly trained model does better, it should take the place of the current one. That way, we’re always using the best model available at any point in time.
The training and evaluation process described above decides whether the model in use should be updated. The last step is deploying the model. How this works depends on the project’s infrastructure.
To sum up, automating the process requires 3 processes to run periodically:
- gathering and preparing training samples
- training and evaluating the machine learning model
- conditional deployment
Automation
Building a machine learning pipeline that carries out these 3 steps only solves part of the problem. Running the code on a local machine saves some time, but it isn’t automation. To stop relying on remembering to trigger the process, we can use cron (an option for masochists) or turn to cloud solutions that scale well.
Source
Earlier parts of the project, such as BigQuery, run on Google Cloud, so looking at this cloud provider was the obvious next move.
What do we need to run the pipeline in Google’s cloud? Cloud Storage for keeping training data and machine learning model artifacts. Google Cloud Builder for building Docker containers in the cloud and pushing them to the right container registry. For ML model training, VertexAI lets you pick suitable hardware, including the GPU required for training more sophisticated AI models. For scheduled BigQuery data fetching, training, and deployment - Google Cloud Run functions.
In short, the chosen technology stack is made up of Google services: Google Cloud Storage, BigQuery, VertexAI, Cloud Run, and Google Cloud Builder.
Implementation
Since the heart of the project – processing training data and training, evaluating, and deploying the machine learning model – was already in place, the main job was to adjust it to run in a cloud environment.
To make the machine learning pipeline production-ready, we’ll use the ZenML framework. It offers an abstraction layer over the services and components of various cloud providers, so migrating the project between clouds doesn’t call for code changes. I recommend having a look at this project!
Pipelines running in the cloud require a ZenML Server deployment. A handy option is deploying it on HuggingFace Spaces. The pipelines need a ZenML stack made up of these components: image builder, orchestrator, and artifact-store. Containers will be run by GCP’s default orchestrator, the VertexAI Orchestrator. Running scheduled jobs requires correctly setting up service accounts and the orchestrator itself in ZenML, following the instructions in the documentation. Deployment relies on the BentoML component. Local Docker serves as the image builder.
Building the training dataset
At this stage, the pipeline is granted read/write permissions for Google Cloud Storage and BigQuery. It authenticates with BigQuery and runs a query that returns data as a pandas DataFrame, transforms the data, divides it into a training set and a test set, and saves it in .csv format to the right GCS bucket. The pipeline runs on a monthly schedule. The test dataset is generated only during the pipeline’s first run. Once the test set exists, later runs store new data samples as training data.
Training, evaluation, and model registry
The model-training pipeline pulls data from the specified Google Cloud Storage bucket, trains the model on the training data, evaluates it on the test data, and compares it against the models in the model registry. As the model registry we use zenml’s built-in Model Control Plane component. If the new model is better, it gets registered in model-registry, an archive containing the code and artifacts required to build the container is created and stored in GCS, and finally a Google Cloud Builder job is triggered – this is the deployment step covered in the following section. This pipeline likewise runs every month. It has to run after the pipeline that collects the training data samples has finished.
Deployment
The container is built in Google Cloud Builder. It lets you create and start a job via a POST request whose payload describes the container build steps, the archive with code and artifacts, and the target container registry where the resulting container will be stored.
Below is a sample payload sent to Google Cloud Builder. The source parameter points to the archive the Docker container will be built from. That archive has to include the Dockerfile and every file required during the build. The steps parameter lists the steps of the process. Here, it’s only the container build command. The final parameter, images defines the name under which the container will be stored in the specified container registry.
{
"source": {
"storageSource": {
"bucket": "finanse-ml-dev",
"object": "models/8f19cd96-ca7b-40bd-980e-f2ce19908a6b/model.tar.gz"
}
},
"steps": [
{
"name": "gcr.io/cloud-builders/docker",
"args": [
"build",
"-t",
"gcr.io/finanse-project/model",
"."
]
}],
"images": [
"gcr.io/finanse-project/model"
]
}
In our setup, the model runs as a Pod on a GKE cluster that was already configured. Kubernetes relies on the image automation feature of flux.cd. As a result, deploying simply means pushing the built Docker container to the right registry. Kubernetes takes care of updating the model internally.
Because of infrastructure requirements, this step builds a Docker container exposing a REST API used for inference on the ML model. We use the BentoML framework for this, which makes it easy to build an REST API for ML model inference.
Results
The end result of the whole CT/CD pipeline is periodic retraining of the machine learning model on the most recent training data, followed by conditional deployment of the model to production. All of this happens on a recurring, automated basis without any need for human oversight.
What’s next
The next stages of the project will concentrate on monitoring both the model and the process. What does that mean? One simple enhancement will be adding Slack alerts that notify the team on a dedicated channel about the training process, model results, task status, and conditional deployment. As for model monitoring, I mean adding a recurring task that uses evidently to detect date and model drift based on how the training data differs from the newest data entering the system.
Issues we ran into
Large artifacts
ZenML applies a cache mechanism to each step of a running pipeline. Every step declares types of input and output data, which are kept as artifacts in a stack component - the artifact-store. The goal is to cut pipeline execution time and resource usage when a step’s input parameters haven’t changed. With large artifacts (e.g., whole datasets), serializing and uploading them takes unacceptably long.
We fixed this by optimizing the dataset passed from step to step. Bear in mind that the dataset is fairly small and made of tabular data. In projects working with images or video, this could prove challenging.
Summary
This post walked through turning a machine learning project from the Proof of Concept stage into an automated CT/CD process and outlined the next stages of its development. Using MLOps techniques in a real project let us verify how useful the tools are and try out proposed approaches to problems around operationalizing AI models.
From the project’s business perspective, because new training data shows up in the system periodically, it paid off to apply a CT/CD architecture on top of machine learning pipelines, which enables the model’s quality to improve cycle after cycle. The planned addition of periodic data and model-drift detection will make it possible to react more quickly when the characteristics of incoming data change.
I’ll cover the next stages of the project in upcoming articles, so stay tuned.