• What is Machine Learning OperationsWhat is Machine Learning Operations
  What is Machine Learning Operations
  
  A set of culture and practice that aims at unifying ML system development and operations, automate and monitoring all steps including integration, testing, rele...
• Why MLOpsWhy MLOps
  Why MLOps
  
  
  Drivers for MLOps
  Data Scientist spend too much time on deployment >50% of their time
  Deployment Gap
  
  
  Evolution of MLOps
  
  
  
• End-to-end Machine Learning LifecycleEnd-to-end Machine Learning Lifecycle
  End-to-end lifecycle
  
  
  
  
  [[MLOps - Data Engineering]]
  [[MLOps - Model Engineering]]
  [[MLOps - Model Deployment]]
  
  

MLOps Principles

Automation

Further reading

Continuous X

• Continuous Integration (CI)
• Continuous Delivery (CD)
• Continuous Training (CT)
• Continuous Monitoring (CM)

Versioning

Model Management Framework

• Data versioning
  • https://emilygorcenski.com/post/data-versioning
• Model versioning

Experiment tracking

• DVC, wandb, mlflow

Testing

Further reading

• The ML Test Score: A Rubric for ML production
• Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology
• How to Test Machine Learning Code and Systems
• Effective testing for machine learning systems
• Testing Machine Learning Code and SystemsTesting Machine Learning Code and Systems
  Resources
  
  
  How to Test Machine Learning Code and Systems
  Testing ML - Repostory
  Effective testing for machine learning systems
  Beyond Accuracy: Behavioral Testing of NLP Models with CheckL...

Monitoring

• The ML Test Score: A Rubric for ML production

Data Science Project Template

• Cookiecutter Data Science Project Template
• The Data Science Lifecycle Process Template
• PyScaffold

ML-based software delivery metrics

Further reading

Feature Store

• Fact Store at Scale for Netflix Recommendations
• MLOps.toys - Feature Store

Machine Learning Platform

• Uber - Michelangelo
  • Evolving Michelangelo Model Representation for Flexibility at Scale
• Lyfe - Flyte: A Cloud Native Machine Learning and Data Processing Platform
• Spotify - The Winding Road to Better Machine Learning Infrastructure Through Tensorflow Extended and Kubeflow
• Grab - Catwalk: Serving Machine Learning Models at Scale
• MLOps.toys - Training Orchestration
• Shopify - The Magic of Merlin

• https://ml-ops.org/content/mlops-principles
• https://medium.com/usf-msds/choosing-the-right-metric-for-machine-learning-models-part-1-a99d7d7414e4
• https://medium.com/usf-msds/choosing-the-right-metric-for-evaluating-machine-learning-models-part-2-86d5649a5428
• https://stanford-cs329s.github.io/syllabus.html
• https://madewithml.com/#mlops
• https://fullstackdeeplearning.com/
• https://github.com/graviraja/MLOps-Basics
• https://neptune.ai/blog/packaging-ml-models
• https://madewithml.com/courses/mlops/cicd/
• https://eugeneyan.com/writing/challenges-after-deploying-machine-learning/
• https://eugeneyan.com/writing/practical-guide-to-maintaining-machine-learning/
• Challenges in Deploying Machine Learning: a Survey of case studies
• Netflix Model Lifecycle Management at Netflix