3 ways to integrate machine learning into your systems

Integrating outputs from machine learning models in batch mode is one of the most widely used patterns in the industry to leverage machine learning in software solutions.

There are many ways in which custom solutions are engineered for batch prediction using machine learning. In most cases there are 4 main components:

1. The data source from which the features that will be used are read.
2. The compute service responsible for doing transformations on the data such that it can be fed to a trained model. The compute is also responsible to run the model or models such that predictions from the models could be obtained. It might also transform the outputs into a format that makes it acceptable for a downstream system to use the same.
3. The data sink where the output from the models along with other factual data are stored which can be used by a downstream system.
4. A workflow management platform that can schedule the different compute steps and also ensure that all dependencies are met for each step before processing.

The following sections will discuss 3 ways in which the 4 components are realized in practice and also their pros and cons.

Hadoop, Spark and Airflow

Hadoop, spark, and airflow (image by author)

Apache Hadoop is used as the data lake to store the data in partitioned Parquet files. Apache Spark is used to do the transformations and based on the data volume, separately trained SparkML or Python-based models/frameworks are used to do batch inference and write the outputs to the data lake also as partitioned Parquet files. The whole process is orchestrated by Airflow which also provides operational monitoring of the pipelines.

Pros

1. Availability of big data
2. Model management with MLFlow integration
3. Use existing setup for Cloudera clusters (for example)

Cons

1. Python environment management is difficult when there are many dependencies
2. Model update will need re-deployment of the whole pipeline

Public cloud — Azure

Azure Blob, Azure Machine learning, and Data factory (image by author)

Data stored as partitioned parquet files can be accessed by Azure Machine learning pipelines via datastores and datasets. Azure machine learning provides capabilities for environment and model management and these building blocks can be used to create machine learning pipelines. The pipelines can then be published with parameters to get a unique URL to reference the machine learning pipeline. The published pipeline endpoint can then be connected to Azure data factory via a linked service to schedule and for operational monitoring.

Pros

1. Less moving parts/points of failures as all are managed services
2. Easy Python environment management
3. Standardized MLOps available from Azure

Cons

1. Vendor lock-in with Azure SDK usage
2. Model update will need re-deployment of the whole pipeline

PaaS service as compute, model available as API

PCF, Spring Batch, Model in K8s, and MySQL

Pivotal Cloud Foundry (PCF) which is a popular PaaS service is used to run Java Spring Batch applications to do the data processing. PCF has a scheduler for scheduling and monitoring. Maven libraries can be used to connect to Hadoop and MySQL storage. The models are deployed in enterprise Kubernetes (e.g. Openshift) as REST services. The Spring batch application creates a batch request and retrieves a batch response from the model endpoint. The outputs are then written to MySQL tables from where downstream applications can consume the output.

Pros

1. Separation of responsibilities. Data transformation can be done in JVM languages and model inference script and environment are still based on Python.
2. Complete decoupling allows for easier blue-green deployment without any operational downtime
3. Easy deployment of model inference with MLFlow integration

Cons

1. Complex and many points of failure
2. API needs to be able to handle spikes in loads
3. Costlier as 2 different compute services need to be available

While the above list is not exhaustive, in most enterprise settings batch machine learning in software systems will be used in a manner that will resemble one of the above patterns.