Estimate project
← All posts

MLOps 101 - Getting Started with MLOps

In this blog post, you will find technical details and tools that help you put Machine Learning Operations (MLOps) into practice.

8 min read
Machine LearningMLOpsGCPAWSNeptunewandb

In this blog post, you will find technical details and tools that help you put Machine Learning Operations (MLOps) into practice. I will start by describing the principles of MLOps and how to put them to work, then walk through the levels of MLOps maturity in projects, and wrap up with an example of Revolut’s approach.

MLOps principles

Let me begin by explaining the core pillars of MLOps, which lead you toward a solid and mature ML system.

1. Put versioning and reproducibility first

At the beginning of a machine learning project, versioning and reproducibility usually do not get much attention. Once the project matures, however, it becomes crucial to obtain reproducible and deterministic results from the same setup and hyperparameters.

Experiment code can change frequently, which is why it is so important to track those changes with version control tools such as Git.

Besides the software code base, we can also version:

  • Configuration files the code relies on. To keep configuration files under version control, take a look at Git.
  • Data, since it can evolve over time as new examples get added, others are removed, and labels change. For dataset versioning, tools such as DVC or Neptune are worth considering.
  • Jupyter Notebooks. Every ML Engineer depends on them, and they can be versioned with Git, although ReviewNB is also an option, since it produces clearer summaries of the differences between versions.
  • Infrastructure. To make the solution more portable and describe infrastructure as a code (IaaC), look into tools such as Terraform or AWS CDK.

Changes in a Jupyter Notebook shown with ReviewNB, Source: reviewnb.com

2. Do not forget about monitoring

Monitoring tends to be treated as the final step of a machine learning project. It should, however, be considered from the very beginning, even before the model reaches production. Do not think of monitoring purely as a deployment concern, because training experiments are worth tracking too. For each experiment, you can track:

  • how weights are distributed over the course of training,
  • hardware usage (CPU, GPU, RAM),
  • how good the predictions on the test set are,
  • the history of metrics such as loss, F1, accuracy, and precision.

For monitoring during training, Weights & Biases, Neptune or MLFlow are worth a look.

Inference monitoring can be split into these groups:

  1. Service level monitoring - keeping an eye on deployed services, e.g., how long user requests take to process, how resources are being used, etc.
  2. Model level monitoring - looking at the distributions of inputs and outputs. This type of monitoring can sound an alarm when the data distribution shifts significantly and the model may start performing worse. Such an alarm can kick off model retraining.

For inference, the choice of monitoring tools depends heavily on your setup. When models run on the AWS Lambda service, AWS Cloudwatch offers rich logging capabilities. In GCP, the equivalent products are Google Cloud Functions for deployment and Stackdriver for logging and monitoring. If, instead, Kubernetes serves as the deployment environment, you should look at:

  • Prometheus to track service liveness, resource utility, and core service parameters.
  • Loki to produce and aggregate logs.
  • Grafana to visualise everything on a dashboard.

A table listing tools for monitoring ML Systems

3. Test everything you can!

Testing of machine learning systems is still far less formalised than software testing. So what is there to test? Here is a list:

  1. The amount and quality of input data.
  2. The schema and distribution of input data (value ranges, expected values, etc.).
  3. The pipeline itself along with the data it produces at every step.
  4. Compliance (e.g., GDPR, EU AI Act) of the ML System’s components (data, pipelines).

Tests make your system sturdier and more resistant to unexpected changes on both the data side and the infrastructure side. Through data validation, the distributions and statistics of training and production data can be compared.

When testing pipelines and transformations, you can use regular software testing practices. As for infrastructure, code written with IaaC can be covered by unit tests. Compliance tests depend on the specific problem and need to be implemented very carefully.

4. Automate!
This step ties in with every principle mentioned above. As machine learning systems mature, models get iterated on and retrained more frequently. Eventually, retraining this often is only feasible when every step is automated. MLOps teams strive for end-to-end model deployment that requires no human intervention.

Since the field is still evolving, you have plenty of freedom to pick what fits your use case best. Treat the guidelines below as a starting point for your own exploration (State of MLOps).

Levels of MLOps

Google says MLOps can be implemented in three ways: as a manual process, semi-automated, or fully orchestrated.

MLOps level 0 (manual process)

Companies just beginning their MLOps journey usually sit at this level, where the ML workflow is manual and decisions are made by Data Scientists. Models are retrained rarely. This level has the following traits:

  1. Everything is done by hand, with Data Scientists writing the scripts and making the micro-decisions.
  2. Training and operations are separate, and data scientists hand over models as an artifact. Every step must be run manually and needs a good deal of human oversight.
  3. No CI/CD. It is not needed, because models are not iterated on often.
  4. Deployment is treated as a basic prediction service (e.g., a REST service on Lambda AWS). Sophisticated scaling algorithms are not considered, since a static deployment is enough.
  5. No algorithm for monitoring performance. Predictions are neither tracked nor logged.

ML systems at this level tend to break often when data keeps changing. Adopting CI /CD and tracking/monitoring algorithms is a good way to address this. An ML pipeline with monitoring and CI/CD enables fast testing, stable builds, and safe rollout of new implementations. This is the final level at which models are trained locally. At every level beyond it (1 and 2), models are trained in the cloud as a job, for instance with Airflow or a similar tool.

MLOps level 1 (semi-automated)
Level 1 centres on continuous training through automated ML pipelines. This setup fits solutions that work in an environment that keeps changing.

This level has the following traits:

  1. Experiments are orchestrated and run automatically.
  2. Production models are trained continuously, on fresh data, whenever a trigger fires.
  3. Development, experimental, preproduction and production environments do not diverge, because they share the same pipeline.
  4. The code of components and pipelines is split into modules so it can be reused and shared between ML pipelines (via containers).
  5. Models are delivered continuously, because training, validation and deployment are all automated.

At MLOps level 1, the deployment code and the training code are both published to the cloud.

Beyond the components already present at level 0, these extra ones must be implemented:

  • Data and model validation: Since the goal is a fully automated pipeline, a data validation component is needed to check the fresh data coming in. Validating data is not easy, although for classification problems it is fairly simple. For tasks like text generation, it is very hard (because labeiing is unclear). After a new model has been trained, it must be validated so that a defective model does not get shipped.
  • Feature store: a component that standardises and stores features and manages access to them for training and serving.
  • Metadata management: details about every run of the ML pipeline are recorded, both to debug errors and anomalies and to support reproducibility, comparisons between runs, and data lineage.
  • ML pipeline triggers: Automated retraining can happen in various scenarios:
    a) on demand,
    b) on a schedule,
    c) when model performance degrades,
    d) when the data distribution changes,
    e) when new data becomes available.

Level 1 MLOps works well if you want your training automated. It only accommodates changing data, though, and any change to the training scheme means redeploying the entire pipeline.

MLOps level 2
This level is characteristic of strongly tech-driven companies that frequently experiment with various implementations of pipeline components. They also train new models often (daily or hourly), update them in an instant, and redeploy them to server clusters all at once. For these companies, this would be impossible without an end-to-end MLOps cycle.

This level has the following traits:

  • Iterative, automated development and experimentation: Thanks to orchestrated ML pipelines, new ideas are easy to test and adopt.
  • CI of the ML pipeline: Building source code and testing components are both straightforward.
  • CD of the ML pipeline: Components and artifacts from the CI stage are easy to deploy.
  • Automated triggering: In production, the pipeline runs automatically on a predefined schedule or is triggered in response to some event.
  • Monitoring: Statistics on model performance are gathered. Using them, a dedicated mechanism can launch retraining once data drift is detected. Logs can also serve for debugging.

Analysing both data and models remains a manual process, even though tooling is available.

Revolut: MLOps for fraud detection
To better illustrate how tech companies put MLOps to use, let me present the case of Revolut, a financial technology company based in the UK. Its flagship product is an app providing a range of banking services and handling millions of transactions every day.

To prevent losses caused by fraudulent card transactions, the company uses a machine learning fraud prevention algorithm named Sherlock.

Source: Building a state-of-the-art card fraud detection system in 9 months_

Revolut mostly runs its services on Google Cloud, which is where all the components are deployed. Data and feature management relies on Apache Beam via DataFlow. Models are developed with Python and Catboost and deployed as a Flask app on App Engine. Couchbase serves as in-memory storage for customer and user profiles.

Google Cloud Composer (running Apache Airflow) takes care of production orchestration.

Revolut monitors its system with two tools combined. The first is Google Cloud Stackdriver, used for real-time analysis of latency, the number of transactions handled per second, and more. The second is Kibana, which tracks merchants, the number of alerts and frauds, and model performance - true positive and false positive rates. Monitoring signals reach the fraud detection team by email and SMS.

Cloud ML Engine makes it possible to track model lineage. Lastly, human feedback is collected inside the app, where every user can indicate whether a transaction was malicious. Because this is a classification task, labels for the new dataset version can be gathered directly.

Source: Building a state-of-the-art card fraud detection system in 9 months_

Useful resources

If you want to dig deeper, I can suggest these resources: