Distributed Machine Learning at Scale

01 The Challenge

The client’s internal data-science group served other business units with forecasts — sales and other metrics correlated against economic trend signals such as consumer sentiment. The ranking solution at the heart of this work ran data-science match models against an enormous set of features — sometimes up to 200,000 — testing which features produced the best-correlated trend scores.

The hard problem was distribution. A single ranking run could require hundreds of millions to billions of calculations. Run sequentially it was hopeless; the work had to be spread across a compute cluster and massively parallelized to be usable at all. Making that large-scale ML compute tractable is what Leopard Data was hired to research and build.

02 Parallelizing the Ranking Engine

Leopard Data evaluated the full range of parallel-processing options — a custom scheduler, Azure Durable Functions, Kubernetes with KEDA, Ray (already in use at the client), and Anyscale — and the team chose KEDA on Kubernetes. We built it up through staged proofs of concept: first proving KEDA against RabbitMQ on minikube, then scaling that PoC into a full AWS EKS cluster and consolidating onto a single cloud.

The ranking solution was structured as three cooperating components — a main driver command-line app that reads a job’s config, writes its working set to the database, and dispatches work; a target runner; and a model runner — with KEDA scaling the runners against queue depth. We benchmarked the KEDA design head-to-head against the data-science team’s Anyscale prototype, and debugged the concurrency and database-relationship failures that surface when this much work runs in parallel inside Kubernetes.

03 Deep Learning in the Prediction Engine

The engagement expanded into the prediction engine, where we designed and built an LSTM deep-learning module in PyTorch — learning the existing codebase, mastering the PyTorch and Ray interfaces, and integrating it end-to-end through design, development, unit testing, and full integration testing. Prediction runs parallelized across Kubernetes using Ray.

We worked hand-in-hand with the data scientists on the model and built its unit-testing framework alongside them, and consulted on a DeepAR implementation in the same engine. We also added per-user job visibility — attributing each training and prediction run via Kong middleware and exposing new data-access endpoints to retrieve jobs by user.

05 Tech Stack