[AI] Weekly Paper #2: Learning Paradigms, Loss, Bias-Variance, and Cross-Validation

For the second weekly paper, I reviewed supervised and unsupervised learning, loss functions, the bias-variance trade-off, and K-fold cross-validation. Writing one at the end of each week also helps me review what I learned and check which parts I understand and which I do not. I can then study the gaps while putting the paper together. :)

1. What is the difference between supervised and unsupervised learning?

Supervised learning uses training data that includes both inputs and correct answers. The model learns the relationship between them. For example, when training a model to distinguish cats from dogs, each image is provided with a cat or dog label.

Unsupervised learning uses input data without answer labels. It examines the distribution of the data to find naturally separated groups or recurring patterns. Its results cannot be checked against known answers in the same direct way as supervised-learning accuracy, but they can still be evaluated by criteria such as how compact and well-separated the clusters are or whether the result serves the purpose of the analysis. Because much of the data available on the internet has no human-provided labels, unsupervised and self-supervised learning are also important approaches.

---

2. What is a loss function, and why is it important?

A loss function calculates a numerical error between the correct target and the model’s prediction during machine-learning or deep-learning training. It matters because gradient descent adjusts the model’s weights in a direction that reduces this loss, allowing the model to produce predictions closer to the targets.

A common regression loss is mean squared error, or MSE. It squares the difference between the target and the prediction for each sample, then averages those squared errors across the full dataset.

$$\operatorname{MSE} = \frac{1}{n} \sum_{i=1}^{n} \left(y_i-\hat{y}_i\right)^2$$

Here, $y_i$ is the true value for the $i$-th sample, and $\hat{y}_i$ is the model’s prediction. The farther the prediction is from the true value, the larger the loss becomes.

Ridge and Lasso add weight penalties to this loss. I wrote separately about how loss and regularization connect in why feature scaling matters before regularization.

A low loss on the training data does not guarantee good performance on new data. Validation performance must also be checked to determine whether the model has overfit the training set.

---

3. What are bias and variance, and how are they related?

Suppose the same problem is learned repeatedly from different training datasets. Bias is high when the average of the resulting predictions remains far from the correct target. Variance is high when the prediction for the same input changes substantially whenever the training dataset changes.

The archery-target example makes the distinction easier to explain. If the center of the target is the correct answer, bias is low when the center of the arrows is close to that answer. Variance is low when the arrows are tightly grouped together.

The same idea applies to fitted prediction lines. Across many possible training datasets, their average position can be represented as an average prediction line. A model has high bias and tends toward underfitting when this line is far from the underlying relationship. A model has high variance and tends toward overfitting when the individual lines change greatly around that average as the training data changes.

Bias and variance often trade off as model complexity changes. Under squared error, prediction error includes squared bias and variance as well as irreducible noise in the data. The goal is therefore not to minimize only bias or only variance, but to balance them so that error remains low on new data.

An underfit model performs poorly on both its training and test data. Put simply, it has not learned enough. An overfit model can perform extremely well on the training data but poorly on the test data because it has learned accidental patterns and noise found only in the training set.

For example, suppose every one-billion-won apartment in the training data happens to have yellow curtains. If the model treats that coincidence as an important feature, it may later undervalue a one-billion-won apartment with blue curtains simply because the curtains are not yellow.

---

4. What should be considered when choosing K for K-fold cross-validation?

K-fold cross-validation divides the full dataset into K equally sized groups called folds. One fold is used for validation while the other K-1 folds are used for training, and the validation fold is changed until the model has been evaluated K times. The model is trained from scratch for every run. The K evaluation results are then averaged to check whether performance remains stable on data that was not used for that run’s training.

The two main considerations when choosing K are the amount of data and the computational cost.

When both the dataset and K are small, each training run may receive too little data. If a dataset contains 30 samples and K is 2, each fold contains 15 samples. One fold is reserved for validation, so the model is trained on only the remaining 15 samples.

Increasing K raises the computational cost. With one million samples and K set to 10, each fold contains 100,000 samples. Each run trains on nine folds, or 900,000 samples, and validates on the remaining 100,000. Under the simplified assumption that the model processes each training sample once per run, the total training-data workload is nine million sample-processings.

With K set to 5, each fold contains 200,000 samples. Each run trains on four folds, or 800,000 samples, and the process is repeated five times. Under the same simplified calculation, the total training-data workload is four million sample-processings. Actual computational cost does not always scale directly with the number of samples because it also depends on the model and training procedure, but increasing K means training the model from scratch more times and generally increases the total cost.

The two cases can be visualized as follows. Each [Training] or [Validation] cell represents one fold.

K = 5

Each fold: 200,000 samples

Run 1  [Validation][Training][Training][Training][Training]
Run 2  [Training][Validation][Training][Training][Training]
Run 3  [Training][Training][Validation][Training][Training]
Run 4  [Training][Training][Training][Validation][Training]
Run 5  [Training][Training][Training][Training][Validation]

Training data per run: 800,000 samples
Total workload: 800,000 × 5 runs = 4,000,000 sample-processings

K = 10

Each fold: 100,000 samples
... = six middle training folds omitted

Run 1   [Validation][Training][Training] ... [Training]
Run 2   [Training][Validation][Training] ... [Training]
                                  ...
Run 10  [Training][Training][Training] ... [Validation]

Training data per run: 900,000 samples
Total workload: 900,000 × 10 runs = 9,000,000 sample-processings

With K set to 10, each run uses more training data and the model is trained from scratch twice as many times as with K set to 5. In this simplified example, the total training-data workload is 2.25 times larger.

Conversely, using K = 5 reduces that workload to less than half of the K = 10 case. In practice, K should be selected by considering the amount of available data together with training time and infrastructure cost.