How to Structure Machine Learning Projects
This article aims to help Machine Learning practitioners who are starting out to:
i) Organize machine learning tasks so as to achieve a productive workflow.
ii) Learn how to break down the project into well-defined phases.
iii) Manage and monitor machine learning experiments.
This guide assumes that you have basic machine learning knowledge. Hopefully, by the end of this article, you will learn how to approach that machine learning hackathon, and how to tackle that data science side-project that you have been planning to start on.
1. PROJECT SETUP AND STRUCTURE
This is the first phase of a Machine Learning project realization, in this stage of the project, you should be able to define the task and scope requirements needed and further plan the development. In determining the tasks, you will end up with clear strategic goals to achieve.
The goals may include but are not limited to:
- An Evaluation Metric;
- Satisfying and Optimising Metrics;
- Training, Development and test distributions
A satisfying metric is a metric that has to be met while an optimizing metric is one that should be as good as possible.
Discussions on project feasibility should also take place during this stage as well as on model tradeoffs. By doing this, identification of project risks and dependencies is carried out in the early stages of the project. For your project folder, you should aim at achieving a structure similar to this:
2. DATA COLLECTION AND LABELLING.
Data is the pillar of any Machine Learning project. According to Andrew Ng, the founder of deeplearning.ai, the development and test set should come from the same distribution. The two sets have a significant impact on the performance of the model. You should strive to improve your data quality, as this will have an effect on your model performance. Carry out Exploratory Data Analysis(EDA) to understand your data before going into the modeling stage; this will mostly involve using the tools such a Jupyter Notebook, R, SPSS, Tableau, or Pandas to gain insights of the data.
3. MODEL EXPLORATION AND BUILDING.
Read widely on conventional approaches associated with solving a similar problem. By doing this, you will be able to arrive at strategies that will be best suited to solving your problem. Start by using a simple model on the data before going for a SoTA(State of the art) model relevant to your problem domain. It is imperative to fit the training set well to achieve low avoidable bias that will, therefore, form a performance baseline.
Remember that the machine learning development cycle is a highly iterative process; hence it is advisable to build your first system quickly and then iterate as you experiment with other ideas.
4. TESTING AND ERROR ANALYSIS.
If you are tackling a classification or regression problem, make sure you use an evaluation metric that is relevant to the type of problem you are solving.
A few examples of metrics for regression models are R2, Mean Absolute Error(MAE), Mean Squared Error(MSE), In contrast, metrics for classification models include F1 score, AUC, categorical cross-entropy and log loss/binary cross-entropy.
To minimize bias, you need to train a bigger model and train longer or better optimization algorithms while to achieve low variation, use more data, and carry out regularisation on the model.
Error analysis will also assist you in prioritizing different strategies to use to get improvement on your evaluation metrics.
The following are tools that will enable you to achieve a productive workflow:
This is a cloud platform that assists in building, training and deploying deep learning models. It is being used by institutions like Stanford University, deeplearning.ai, and Kiwi. FloydHub comes with pre-installed/built-in dependencies, which makes it easy to maintain consistency in the environments. Some of the pre-installed packages include: Numpy, scipy, SpaCy, OpenCV, OpenAI Gym, etc. FloydHub also provides workspaces based on the given environments.
This is an open-source platform that helps in the management of the Machine Learning lifecycle. The platform has four components, namely: MLflow Tracking, MLflow Projects, MLflow Models and Model Registry. The components offered on the platform come in handy during experimentation, reproducibility, deployment, and a central model registry. MLflow has built-in integrations for a good number of machine learning libraries and environments. The platform is being used and contributed to by organizations like Microsoft, Wix, The University of Washington, among others.
To get started with MLflow, you can have a look at the official documentation.
This is a company that provides tools for data professionals to turn their deep learning research projects into software that can be deployed. They provide tools for experiment tracking, model optimization, and dataset versioning. Some of the tools in Weights and Biases include a dashboard, artifacts, sweeps, and reports.
This is an experimental management tool that can be used as a service or deployed on any cloud or hardware. The tool helps in keeping track of experiments, recording data exploration, versioning Jupyter notebooks and organizing teamwork.
This is an open-source framework that assists in deploying machine learning experiments in the form of web apps. It allows data science engineers to build interactive web applications from their data and models.
This is an open-source provider of NLP technologies.
Check out the following resources for more reads and extensive guides:
1. A Layman's Guide to Data Science. Part 3: Data Science Workflow
2. How to manage Machine Learning and Data Science projects 3.Structuring Machine Learning Projects 4.Structure and automated workflow for a machine learning project — part 1 5.Structure and automated workflow for a machine learning project — part 2
Feel free to leave your question and feedback on the comments section.