Optimizing Machine Learning

Accelerate popular Machine Learning algorithms through system awareness, and hardware/software differentiation

Develop novel Machine Learning algorithms with best-in-class accuracy for business-focused applications

AI in Business – Challenges

Snap Machine Learning (Snap ML in short) is a library for training and scoring traditional machine learning models. Such traditional models power most of today's machine learning applications in business and are very popular among practitioners as well (see the 2019 Kaggle survey for details). Snap ML has been designed to address some of the biggest challenges that companies and practitioners face when applying machine learning to real use cases. These challenges are illustrated in the graphic below.

 

Speed

Train and re-train on new data online

Large parameter, model searches

Make fast decisions

Efficiency

Use resources judiciously

Less resources means less $

On-prem and in the cloud

Accuracy

Make accurate decisions or predictions

Cost savings (e.g. card fraud), higher revenue (e.g. portfolio allocation)

Data Size

Learn from all available data

More data, better models, higher accuracy

Handle big data efficiently


Snap ML is

Fast

Multi-threaded CPU solvers as well as GPU and multi-GPU solvers that offer significant acceleration over established libraries.

Scalable

Distributed solvers (for generalized linear models currently) that scale gracefully to train TB-scale datasets in mere seconds.

Accurate

A novel gradient boosting machine that achieves state-of-the-art generalization accuracy over a majority of datasets.

Resource-efficient

Ability to complete large training jobs in less resources, with high resource utilization.
 

Consumable

Familiar Python scikit-learn APIs for single-server solvers and Apache Spark API for distributed solvers.

Supported Machine Learning Models

Generalized Linear Models

State‐of‐the‐art solvers on multi‐core, multi‐socket CPUs. Twice‐parallel, asynchronous stochastic coordinate descent (TPA‐SCD) for training linear models on GPUs.

card_1

Tree-based Models

Memory‐efficient breadth‐first search algorithm for training of decision trees, random forests and gradient boosting machines.

card_2

Gradient Boosting

Heterogeneous boosting machine that employs multiple classes of base learners, versus only decision trees.
 

card_3

Seamless acceleration of scikit-learn applications

Snap ML offers very powerful, multi‐threaded CPU solvers, as well as efficient GPU solvers. Here is a comparison of runtime between training several popular ML models in scikit‐learn and in Snap ML (both in CPU and GPU). Acceleration of up to 100x can often be obtained, depending on model and dataset.
 
card_1
su_2
card_3
At the same time, the above acceleration is almost seamless to the user. For data scientists using Python, only minimal changes are needed to their existing code to take advantage of Snap ML. Here is an example of using a Random Forest model in both scikit‐learn as well as Snap ML.
 
random forest
 

Gradient Boosting Machine

Gradient Boosting models comprise an ensemble of decision trees, similar to a random forest (RF). Although Deep neural networks achieve state-of-the-art accuracy on image, audio and NLP tasks, on structured datasets Gradient Boosting usually out-performs all other models in terms of accuracy. Some of the most popular Boosting libraries are XGBoost, LightGBM and CatBoost. Snap ML introduces SnapBoost, which targets high generalization accuracy through a stochastic combination of base learners, including decision trees and Kernel ridge regression models. Here are some benchmarks of SnapBoost against LightGBM and XGBoost, comparing accuracy across a collection of 48 datasets. SnapBoost learns a better model in about 2-out-3 of the datasets tested.

Accuracy

OpenML (www.openml.org) is a platform for collaborative data science. Snap ML’s Gradient Boosting was benchmarked against XGBoost and LightGBM using 48 binary classification datasets from OpenML. Hyper-parameter tuning and generalization estimation was performed using 3x3 nested cross-validation. Snap ML provides best-in-class accuracy for a majority of datasets.

Snap ML Evolution

Evolution of Snap ML
Evolution of Snap ML

Where to get / How to try Snap ML


 
• pip install snapml

• Support for: Linux/x86, Linux/Power, Linux/Z, MacOS, Windows

• GPU support available for Linux

• Documentation: https://snapml.readthedocs.io/

• Example Jupyter notebooks: https://github.com/IBM/snapml-examples

Videos

(PyData Global, 2020)

(NeurIPS, 2018)

Publications

Lazuka, M., Parnell, T., Anghel, A. and Pozidis, H.,
“Search-based Methods for Multi-Cloud Configuration,”
In IEEE 15th International Conference on Cloud Computing (CLOUD) (pp. 438-448). IEEE, 2022.

G. Damaskinos, C. Mendler-Dünner, R. Guerraoui, N. Papandreou, T. Parnell,
“Differentially Private Stochastic Coordinate Descent,”
AAAI, 2021.

T. Parnell, A. Anghel, M. Lazuka, N. Ioannou, S. Kurella, P. Agarwal, N. Papandreou, H. Pozidis,
“SnapBoost; A Heterogeneous Boosting Machine,”
NeurIPS, 2020.

D. Sarigiannis, T. Parnell, H. Pozidis,
Weighted Sampling for Combined Model Selection and Hyperparameter Tuning,”
AAAI, 2020.

A. Anghel, N. Ioannou, T. Parnell, N. Papandreou, C. Duenner, H. Pozidis,
Breadth‐first, Depth‐next Training of Random Forests,”
NeurIPS Workshops, Systems for ML, 2019.

N. Ioannou, C. Mendler-Dünner, T. Parnell, ,
SySCD: A System-Aware Parallel Coordinate Descent Algorithm,”
NeurIPS, 2019 (Spotlight).

K. Atasu, T. Mittelholzer,
Linear-Complexity Data-Parallel Earth Mover's Distance Approximations,”
ICML, 2019.

C. Dünner, T. Parnell, D. Sarigiannis, N. Iouannou, A. Anghel and H. Pozidis,
SnapML: A Hierarchical Framework For Machine Learning,”
NeurIPS, 2018.

C. Dünner, M. Gargiani, A. Lucchi, A. Bian, T. Hofmann and M. Jaggi,
A Distributed Second-Order Algorithm You Can Trust,”
ICML, 2018.

T. Parnell, C. Dünner, K. Atasu, M. Sifalakis and H. Pozidis,
Tera-Scale Coordinate Descent on GPUs,”
FGCS, 2018.

A. Anghel, N. Papandreou, T. Parnell, A. De Palma, H. Pozidis,
Benchmarking and Optimization of Gradient Boosted Decision Tree Algorithms,”
NeurIPS Workshops, Systems for ML, 2018.

C. Dünner, T. Parnell and M. Jaggi,
Efficient Use of Limited-Memory Accelerators for Linear Learning on Heterogeneous Systems,”
NIPS, 2017.

C. Dünner, T. Parnell, K. Atasu, M. Sifalakis and H. Pozidis,
Understanding and Optimizing the Performance of Distributed Machine Learning Applications on Apache Spark,”
IEEE Big Data, 2017.

K. Atasu, T. Parnell, C. Dünner, M. Vlachos, H. Pozidis,
High-Performance Recommender System Training using Co-Clustering on CPU/GPU Clusters,”
ICPP, 2017.

T. Parnell, C. Dünner, K. Atasu, M. Sifalakis and H. Pozidis,
Large-Scale Stochastic Learning using GPUs,”
IPDPSW – ParLearning, 2017.

K. Atasu, T. Parnell, C. Dünner, M. Sifalakis, H. Pozidis, V. Vasileiadis, M. Vlachos, C. Berrospi and A. Labbi,
Linear-Complexity Word Mover’s Distance with GPU Acceleration,”
IEEE BigData, 2017.

C. Dünner, S. Forte, M. Takac and M. Jaggi,
Primal-Dual Rates and Certificates,”
ICML, 2016.