Code › ai-engineering-study

Feature Scaling and Regularization

How feature scale affects model training and why Ridge, Lasso, and Elastic Net depend on comparable inputs

codeit-ai-sprintcodeit-ai-main-courseaibootcampai-engineeringmachine-learningfeature-scalingregularizationridgelassoelastic-netlearning

Today I wanted to sort out scaling, normalization, standardization, and regularization in one place. Before that, a quick aside. My teammates and I shared our weekly papers today, and I realized that my explanation had become too long without stating the main point clearly enough. The better order is to define the concept first, state that definition cleanly, and then move into the supporting details. This topic also connects to the weekly papers, so it was worth mentioning here.

Scaling and regularization are useful to consider together for linear models. Regularization penalizes the size of the weights, and when input features use very different units, comparing those weights fairly becomes difficult.

For example, when predicting house prices, the number of rooms may range from 1 to 5, while the area may range from tens to hundreds of square meters. A feature with a larger numerical range is not necessarily more important, but distance-based calculations can be dominated by that range. Gradient descent can also move unevenly because each feature changes at a different magnitude, which can slow down the path toward the minimum.

Separating Three Similar Terms

Normalization, standardization, and regularization have similar names, and their Korean translations can overlap. The instructor also mentioned this distinction in class. In particular, normalization and regularization are sometimes translated with the same Korean term, so I am separating the meanings here.

  • Normalization changes the size or range of data according to a fixed rule. It can mean more than Min-Max scaling, including transformations that make a vector length equal to 1, so in this post I use Min-Max scaling when that specific method is meant.
  • Standardization subtracts the mean from each value and divides by the standard deviation.
  • Regularization is not a preprocessing step that changes the data. It adds a weight penalty to the loss function.

Min-Max scaling and standardization both belong to feature scaling. Regularization methods such as Ridge, Lasso, and Elastic Net use the scaled data to control model complexity. In short, scaling transforms the input data, while regularization constrains how the model learns its weights.

Scaling Feature Magnitudes

Standardization subtracts the mean from each value and divides by the standard deviation.

z=x−μσz = \frac{x-\mu}{\sigma}

The transformed feature has a distribution with mean 0 and standard deviation 1. Outliers still affect the mean and standard deviation, but this method is commonly used with models that are sensitive to feature magnitudes, such as linear regression, logistic regression, and SVMs.

Normalization can mean several things, but the Min-Max scaling often introduced in class maps the minimum value to 0 and the maximum value to 1.

x′=x−xminxmax−xminx' = \frac{x-x_{min}}{x_{max}-x_{min}}

The advantage is that the range becomes fixed. The trade-off is that the minimum and maximum determine the transformation, so outliers can have a large effect. Neither standardization nor Min-Max scaling is always better. The choice should follow the data distribution and the model being used.

A scaler should be fitted on the training data only, not on the full dataset. If the mean, minimum, or maximum is calculated using the test data as well, information that should be unavailable during training leaks into the training process. The scaler should learn its reference values from the training data, then apply the same transformation to the validation and test data to avoid data leakage.

I also covered Min-Max transformation and the Z-score formula in the data preprocessing section of the first weekly paper.

Regularization Keeps Weights from Growing Too Large

When a model follows even small noise in the training data, it can perform well on that data but poorly on new data. Regularization limits excessive model complexity by adding a penalty for weight size to the cost function.

I covered how a loss function calculates the difference between actual and predicted values using MSE in the second weekly paper’s loss-function section. Regularization adds a weight penalty to that existing loss.

Ridge uses L2 regularization. It adds the sum of squared weights as a penalty, so large weights receive a stronger constraint. It generally shrinks weights overall, but usually does not make them exactly zero.

J(w)=Loss(w)+λ∑j=1dwj2J(w) = \text{Loss}(w) + \lambda \sum_{j=1}^{d}w_j^2

Lasso uses L1 regularization. It adds the sum of absolute weight values as a penalty. Because it can push some weights to zero, it can also reduce the number of features the model actually uses.

J(w)=Loss(w)+λ∑j=1d∣wj∣J(w) = \text{Loss}(w) + \lambda \sum_{j=1}^{d}|w_j|

Elastic Net combines L1 and L2 regularization. It pairs Lasso’s feature-selection effect with Ridge’s more stable weight shrinkage, and it can tune the balance between the two when many features are correlated.

The value that controls regularization strength is lambda. In scikit-learn’s Ridge and Lasso implementations, this is usually named alpha. If the value is too small, regularization has little effect. If it is too large, even useful relationships can become too weak, causing underfitting.

Why Scaling Comes First

Features with different units, such as area and room count, can need weights of different sizes even when they both contribute to the same prediction. Regularization directly compares the absolute values or squared values of those weights, so without scaling, a feature can receive a larger penalty because of its unit rather than because of its real influence.

For models that are sensitive to feature magnitude, the order matters. Split the data first, fit the scaler on the training data, transform the data with that scaler, and then train the regularized model. In scikit-learn, a pipeline can keep these steps together and reduce the chance of data leakage.

from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

model = make_pipeline(
    StandardScaler(),
    Ridge(alpha=1.0),
)

Today I reviewed the theory behind scaling features before applying regularization to improve model training. The lecture emphasized that model performance can be shaped heavily by data preprocessing, sometimes more than by changing the algorithm itself. That makes preprocessing a part I need to organize and understand more carefully.