Section 1.1: Data Collection and Labeling

Collecting data can be a costly endeavor in terms of both time and finances, especially when dealing with unique issues that lack readily available datasets. In such cases, we must gather the data ourselves.

The most significant expenses often stem from labeling data for supervised learning tasks, particularly when done manually. For instance, if we aim to identify supermarkets in a city but lack the necessary data, a team might decide to deploy a vehicle equipped with cameras to capture images of the area. This method of data acquisition is not only expensive but also requires significant resources for manual labeling.

Additionally, this category of challenge includes issues with data quality, which can negatively affect our project's success. Poor raw data quality and inadequate labeling can lead to unreliable results.

Section 1.2: The Impact of Noise

Noise in datasets refers to irrelevant or erroneous information that can distort the data. Examples of noise include blurred images, background sounds in audio recordings, or poorly formatted text.

“Noise is often a random process that corrupts each example independently of other examples in a collection.” — Burkov, A. Machine Learning Engineering, Page 44.

Noisy data can become particularly problematic when working with small datasets relative to the complexity of the problem. This often results in overfitting, where the model learns to recognize the noise rather than the actual patterns, leading to poor performance on new data.

Section 1.3: Low Predictive Power

Experiencing consistent poor performance across multiple algorithms on a dataset can be frustrating. Often, we do not realize that we have a low predictive power issue until we have thoroughly explored various modeling options.

Low predictive power may stem from two main factors: the model's inadequacy in capturing the complexities of the data, or the dataset itself lacking sufficient information to enable effective learning.

Section 2.1: Understanding Bias

Bias refers to a systematic error that can skew results in one direction or another, often leading to unfair or closed-minded conclusions. In data science, biases can arise for numerous reasons, warranting a deeper exploration in a separate discussion.

Section 2.2: The Challenge of Outdated Examples

In the world of MLOps, a model's performance typically declines over time once deployed. This phenomenon, known as concept drift, occurs when the statistical properties of the target variable shift in unforeseen ways, resulting in decreased accuracy of predictions.

Section 2.3: Identifying Outliers

Outliers are data points that significantly deviate from the majority of examples in a dataset. Their identification can vary based on the chosen metrics, such as Euclidean distance. Some models, like Linear Regression, are particularly sensitive to outliers, while others can manage them more effectively.

Section 2.4: The Issue of Data Leakage

Data leakage occurs when information not included in the training data is inadvertently used to train a model, leading to misleadingly optimistic results. Recognizing data leakage is crucial, as it can indicate that the model's performance is not genuinely reflective of its predictive capabilities.