Machine learning is a key driver of technological advancement today. Establishing a systematic machine learning strategy is essential for efficiently advancing projects and achieving desired outcomes. This requires careful consideration of several critical steps, including goal setting, model selection, data processing, and results evaluation.

In this blog, we will explore these steps in detail. We will particularly focus on effective strategies and methods for setting machine learning goals, evaluating model performance, and optimizing models. By the end of this blog, you should have a deeper understanding of the machine learning project lifecycle and be able to apply these methods to enhance your project’s performance.

This blog also covers content from the third course in the deep learning specialization by Andrew Ng. Given the brevity of this course, we will summarize it in a single post. Let’s dive in!

1. Orthogonalization: Streamlining Goals and Means

Orthogonalization is a crucial concept in machine learning strategies. It involves optimizing different aspects of a machine learning system by ensuring each aspect focuses on a specific goal independently.

Consider the example of adjusting the picture quality on an old TV. Older TVs typically had multiple control knobs, each dedicated to a single attribute such as height, width, or brightness. This setup exemplifies orthogonalization, simplifying adjustments by avoiding interference between controls. In contrast, if a single knob adjusted multiple attributes simultaneously, it would complicate the process, violating the principle of orthogonalization.

In machine learning systems, we often need to tweak several aspects to enhance performance, including:

• Performance on the training set
• Performance on the development set
• Performance on the test set
• Performance in real-world applications

These four goals are like the four knobs on an old TV, with each knob focused on a specific goal. For instance, if the model underperforms on the training set, we might increase the network size or adjust the optimization algorithm. If it underperforms on the development set, we could adjust regularization parameters or gather more training data. According to orthogonalization principles, each adjustment should target a single goal to avoid complexity, making the tuning process more systematic and clear.

The main advantage of orthogonalization is that it breaks down a complex problem into several manageable sub-problems. By focusing on one goal at a time, we can make more precise adjustments. This method also facilitates efficient team collaboration, as different sub-tasks can be assigned to different team members with minimal interference.

However, some methods, like early stopping, may affect multiple goals. In such cases, it’s crucial to balance the priorities of different sub-tasks and determine the primary optimization goal.

2. Single Evaluation Metric: Quantifying Goals

In machine learning, dealing with multiple performance metrics like accuracy and recall can be challenging. To evaluate models more efficiently, we often use a single evaluation metric.

For example, in cat image classification, we are concerned with both precision and recall:

• Precision measures the proportion of images classified as cats that are actually cats.
• Recall measures the proportion of actual cat images correctly classified as cats.

To balance these metrics, a common evaluation metric is the F1 Score, calculated as the harmonic mean of precision and recall. If a classifier serves users in various regions, such as China, the United States, and India, we can calculate the F1 Score for each region and then average them. This single numerical metric simplifies model selection. Whenever we try a new approach, we can quickly gauge its effectiveness using this metric, enhancing efficiency.

Setting a single evaluation metric is like placing a "target" for the machine learning team to aim at. Quick experiments can reveal if new attempts are hitting the mark. It’s also essential to ensure the chosen metric accurately reflects the true goals. If a metric proves inadequate, timely adjustments are necessary. For instance, focusing solely on test set accuracy might cause the algorithm to overfit the specific dataset, hindering generalization. In such cases, adjusting the evaluation metric or designing other methods to assess generalization is crucial.

3. Satisficing and Optimizing Metrics: Balancing Multiple Goals

In machine learning, balancing multiple objectives is often crucial. To manage this, we can define Satisficing Metrics and Optimizing Metrics.

Consider an image classification task where we care about both accuracy and efficiency. We can set accuracy as the optimizing metric to maximize, while setting efficiency, such as classification speed, as the satisficing metric, ensuring it stays below an acceptable threshold (e.g., 100 milliseconds).

When dealing with multiple goals, we choose one as the optimizing metric and aim to maximize it. The remaining goals become satisficing metrics, where we only need them to meet a set threshold. This approach helps us quickly and automatically select the best model that optimizes the primary goal while satisfying all other requirements.

Defining satisficing and optimizing metrics is similar to setting “hard” and “soft” targets. Hard targets are essential baselines, whereas soft targets are goals we strive to achieve. By setting these metrics appropriately, we can effectively handle multi-objective tasks. However, metrics can sometimes conflict. For instance, in recommendation systems, increasing click-through rates might reduce conversion rates. Therefore, it’s important to prioritize the main optimization goal based on business needs.

4. Training, Development, and Test Sets: Defining Goal Stages

Splitting data into training, development, and test sets is a fundamental practice in machine learning. These sets correspond to different stages:

• Training Set: Used for iterative optimization to achieve the best performance on training data.
• Development Set: Used for hyperparameter tuning to attain the best generalization on validation data.
• Test Set: Used for the final evaluation of the model’s generalization in real-world applications.

The development set serves as a benchmark for the team, guiding the expected generalization performance. Once the development set and evaluation metrics are established, the team can experiment and evaluate various models, selecting the best one.

It’s crucial for the development and test sets to come from the same distribution, to ensure consistent performance. Traditionally, 70% of the data might be used for training, with the remaining 30% for testing. In the era of big data, we can use a larger portion (e.g., 98%) for training and the rest for development and testing.

By defining these three stages, we can optimize the model from different angles:

• During the training stage, we can use techniques that perform well on training data, such as strong regularization and extensive data augmentation.
• In the development stage, we select models and hyperparameters that generalize well.
• Finally, in the test stage, we evaluate the model’s real-world generalization performance.

5. When to Adjust the Development Set and Metrics

The development set and evaluation metrics set the foundation for our goals. However, sometimes we find that these initial settings are inappropriate and need adjustment. For instance, in a cat classification task, if Algorithm A has a low error rate but frequently misclassifies inappropriate images (such as pornographic images) as cats, we need to adjust the evaluation metrics to assign a higher weight to these errors. Similarly, if the development set consists of high-quality internet images, but real-world applications involve lower-quality user-uploaded images, the development set should be adjusted to better match real-world conditions.

Even if the initial definition of the development set and evaluation metrics isn’t perfect, it’s important to set them to accelerate optimization iterations. When they become unsuitable, adjustments are necessary. Adjusting these metrics is like re-routing a navigation path when the original route no longer leads to the destination effectively.

Adjustments are needed under the following circumstances:

• When the model performs well on the development set but poorly in real-world applications
• When the evaluation metrics do not accurately reflect actual requirements
• When new data is obtained or issues with the original development set are identified
• When business needs change

Generally, if there is a significant discrepancy between the model’s development set performance and real-world performance, it’s time to consider adjustments.

6. Human Performance: A Benchmark for Evaluation and Optimization

In recent years, many machine learning teams have benchmarked their algorithms against human performance. This is because advancements in deep learning have made it possible for machine learning to surpass human capabilities, and aiming for human-level performance can make the design process more efficient. Human error rates on specific tasks can serve as an estimate of the Bayesian optimal error rate. When an algorithm’s performance approaches human levels, further improvements become more challenging.

Human performance provides a valuable benchmark for evaluation and optimization. By comparing the algorithm’s bias and variance issues to human performance, more informed strategies can be developed.

For example, if the algorithm’s error rate on the training set is much higher than that of humans, efforts should focus on reducing bias. If it is close to human levels, reducing variance should be prioritized. Once human performance is surpassed, assessing avoidable bias and variance becomes difficult, requiring new strategies for further improvement. Many structured data tasks have already achieved performance levels beyond human capabilities.

The benefits of using human performance as a benchmark include:

• It provides a direct measure of complex problem-solving, unaffected by specific datasets.
• It integrates multiple dimensions of information, such as prior knowledge and intuition.
• It reflects the inherent difficulty of a task.

However, there are limitations:

• Human performance can have high variance.
• Humans are influenced by factors such as training time and attention span.
• It is challenging for humans to perform consistently on large-scale data.
• Cognitive biases can affect human performance.

Overall, using human performance as a benchmark can provide excellent guidance for algorithm evaluation and optimization, but it’s important to be aware of its limitations and maintain realistic expectations.

7. The Key Role of Error Analysis in Improving Model Performance

When a machine learning model’s performance hasn’t reached human levels, error analysis is crucial for determining effective optimization strategies. For instance, if a cat image classification model frequently misclassifies dogs as cats, the intuitive response might be to collect more dog images for training. However, before investing significant time and resources, it’s essential to verify this strategy through error analysis.

By randomly selecting 100 misclassified images from the validation set for manual inspection, we can assess the proportion of misclassified dogs. If only 5% of errors are due to dogs being misclassified as cats, solving this issue will only reduce the error rate from 10% to 9.5%. This limited improvement suggests the need to reconsider the focus on dog classification.

Error analysis should examine various error sources, such as difficulties with large cats or issues caused by blurry images. Creating a table to list each error source and its frequency can help prioritize which issues to address. This approach predicts the potential performance gains for each optimization direction, guiding us on where to concentrate efforts. It’s important to recognize that error analysis identifies potential improvements but doesn’t mandate specific actions. New error types may emerge during analysis, requiring adjustments to the strategy. Overall, error analysis provides a framework for quantifying and comparing different optimization strategies, making it a powerful tool.

To enhance error analysis, we can study misclassifications from multiple angles, such as identifying common error regions and comparing errors across different models and combinations. Highlighting misclassified areas within images can reveal if the model relies too heavily on local features at the expense of global information. Comparing error patterns between models can uncover systemic biases, informing model integration strategies.

Quantitative and qualitative analyses from various dimensions help pinpoint the next optimization focus more accurately. Error analysis demands patience and insight, but the rewards are significant, allowing us to optimize our machine learning systems more effectively.

8. Improving Data Quality: Cleaning and Managing Incorrect Annotations

In supervised learning, training sets consist of input features (X) and corresponding labels (Y). However, some labels may be incorrect in practice. We need to balance the benefits and costs of cleaning this "dirty data." While deep learning models can tolerate random labeling errors to some extent, systematic errors, like mislabeling all white dogs as cats, significantly impact performance. Correcting such errors should be a priority. For mislabeled training data, the decision to clean extensively depends on the error rate and dataset size. If the dataset is large and the error rate is low, extensive cleaning might be unnecessary. However, mislabeled data in development and test sets can skew performance evaluation and should be meticulously corrected. During error analysis, we can track these errors to assess their impact. Significant errors that affect judgment warrant correction.

The cleaning process involves reviewing correctly and incorrectly predicted samples and addressing errors in the development and test sets. The extent of training set cleanup depends on the situation. Cleaning mislabeled data involves a careful cost-benefit analysis. To ensure quality, we can perform sample checks by re-labeling a random subset of the training data and comparing it to the original labels. This consistency check informs further cleaning decisions. Unsupervised learning methods can also detect anomalies in the training set, potentially identifying mislabeled samples. However, manual verification remains necessary to confirm true outliers.

Overall, evaluating the types, scale, and impact of labeling errors across datasets helps develop a cleaning strategy that is both cost-effective and quality-assured. Close collaboration between data scientists, domain experts, and labelers is essential to enhance data quality.

9. Efficient Iterative Optimization: Quickly Building and Continuously Improving Models

When developing new machine learning applications, our goal is to rapidly achieve reliable initial results. Using speech recognition as an example, we can see how efficient construction and continuous iterative optimization can create a functional system. Speech recognition optimization can target various aspects, such as noise reduction, dialect recognition, and children’s speech. Prioritizing these optimization directions can be challenging. We recommend first quickly building a basic system, then using bias and error analysis to determine optimization priorities.

Start by preparing a simple development set and evaluation metrics, then build a basic speech recognition system. Through bias, variance, and error analysis, we can identify major issues. For example, if many errors stem from distant speech, addressing this first might yield significant improvements.

Quickly building a basic system allows us to establish a functional version early on, which we can then use for analysis to guide further optimization. The system doesn’t need to be overly complex at first; a simple version is sufficient. With extensive experience or prior research, starting with a more complex system is also viable. However, for new problems, it’s often best to quickly build a basic system and iterate, avoiding overdesign and "analysis paralysis." After the initial build, set clear, quantifiable evaluation metrics and iteration goals. These can be for the overall system or specific components, but they must be measurable to compare the effectiveness of different iterations.

Additionally, control the time spent on each iteration to avoid endless optimization. Focus first on optimizations with the most significant improvements, then delve into more advanced optimizations. Quick construction and continuous iteration are invaluable skills in machine learning engineering.

10. Handling Training and Testing Set Distribution Mismatch

In deep learning, we often combine data from multiple sources to create a training set, but these sources might not match the actual application scenario’s distribution, impacting the model’s generalization. Here’s how to analyze and address this issue.

Set up an additional "training-development set" to reflect the model’s performance on real-world data. If the error on this set is much higher than on the training set, it indicates insufficient generalization due to distribution differences.

While increasing the data volume of the training-development set can reduce variance, there is no systematic solution for distribution mismatch. Common approaches include aligning the training data distribution with the development and test data, collecting more similar data, and using pre-trained models in related fields. For instance, if the development set has a lot of noise, adding similar synthetic noisy audio can make the training data more representative. The quality and diversity of synthetic data are crucial to avoid overfitting.

To visually analyze distribution differences, perform statistical comparisons like mean and variance calculations to check for biases. Use visualization tools like t-SNE to map high-dimensional data to a two-dimensional plane and see if training and test data cluster differently.

These analyses help identify which attributes cause distribution mismatches and their severity. Combining domain knowledge allows targeted solutions.

In summary, error analysis helps identify distribution mismatch issues, and synthetic data generation is an effective solution. Careful evaluation of synthetic data effectiveness is essential for building models with strong generalization capabilities.

11. Enhancing Model Generalization with Transfer Learning and Multi-task Learning

In deep learning, we often face the challenge of limited training data. Transfer learning and multi-task learning are two techniques that can help improve a model’s generalization by leveraging data from related tasks.

Transfer learning involves training a model on one task (the source task) and then transferring its knowledge to another task (the target task). This is particularly useful when the source task has ample data, while the target task has insufficient data. For example, a model can be pre-trained on a large-scale image classification task and then adapted for medical image analysis. Implementing transfer learning can involve directly using the pre-trained model, using only parts of its network layers, or using the source task model as a regularization term. It’s crucial to note that directly using the pre-trained model may lead to overfitting, so often, some layers are frozen and only the higher layers are trained.

Multi-task learning, on the other hand, trains a model to handle multiple related tasks simultaneously. For example, an autonomous driving system might need to detect vehicles, pedestrians, and traffic lights. Compared to training separate models for each task, multi-task learning allows the tasks to share features, thereby enhancing generalization. Typically, lower layers of the model share feature extraction, while higher layers handle specific tasks independently. Shared feature extraction layers are trained through gradient accumulation, balancing different tasks’ loss functions to prevent any single task from dominating.

Overall, transfer learning and multi-task learning are powerful tools for improving model generalization with limited data. Transfer learning is ideal for situations with insufficient target task data, while multi-task learning requires sufficient data for each task to support training. The choice between the two depends on the specific scenario.

12. Choosing Between End-to-End and Step-by-Step Methods in Deep Learning

End-to-end deep learning simplifies traditional systems by using a single neural network to map inputs directly to outputs. This approach can be highly effective but has limitations. Deciding whether to use end-to-end or step-by-step methods depends on the problem’s characteristics.

The primary advantage of end-to-end learning is its ability to train complex models with large amounts of data, eliminating the need for manually designed features. This avoids human design limitations and simplifies processing by removing intermediate steps. However, end-to-end methods require large datasets. For data-limited problems, step-by-step methods can incorporate human knowledge to enhance performance. Additionally, end-to-end approaches exclude the possibility of integrating effective manually designed components.

Not all problems are suitable for end-to-end deep learning. For complex but data-scarce issues, step-by-step methods may be more appropriate. For example, in facial recognition for security checks, performing face detection before recognition might be more effective. The decision between end-to-end and step-by-step approaches should consider task complexity, data availability, and problem variability. For simpler tasks with abundant data, end-to-end methods may be advantageous.

However, for more complex tasks, especially those requiring semantic understanding, step-by-step methods remain essential due to data and model constraints. Additionally, if the problem domain frequently changes, the modularity and interpretability of step-by-step methods can offer advantages.

In summary, the choice between end-to-end and step-by-step methods should be based on the problem’s specifics and data availability. End-to-end methods are not always superior to those incorporating human expertise. Leveraging data scale and learning capabilities is crucial to maximizing deep learning benefits.

Conclusion

This blog has explored how to effectively set goals for machine learning projects, evaluate model performance, and optimize accordingly. This includes task definition, selecting suitable evaluation metrics, balancing multiple objectives, the importance of data partitioning, and knowing when to adjust development sets and metrics. We also discussed the significance of using human performance as a benchmark.

In machine learning, choosing the right evaluation metrics and datasets, setting appropriate optimization goals, and properly assessing model performance are critical. These practices help us understand model behavior, identify shortcomings, and optimize effectively.

However, machine learning is not static. As technology evolves and new data emerges, we may need to revisit our models, metrics, and goals. Understanding these foundational concepts and applying them flexibly is key.