Code › ai-engineering-study
[AI] Weekly Paper #3 - Decision Trees, Boosting, and Dimensionality Reduction
The third weekly paper on decision trees, major boosting models, and the difference between PCA and factor analysis
This third weekly paper covers decision trees, boosting, and dimensionality reduction. I started with the strengths and weaknesses of decision trees, moved into boosting methods that build trees sequentially, and finished by comparing two approaches to dimensionality reduction.
1. What are the advantages and disadvantages of decision trees?
A decision tree is a supervised learning model that divides data into regions by applying conditions to its features. A leaf node then produces a class or a predicted value. That definition sounds abstract, but a loan review example makes it more concrete: the tree might ask whether income exceeds a threshold and whether the applicant has a history of late payments.
Its clearest advantage is interpretability. A prediction can be traced through the conditions that produced it, and the tree itself can be visualized. Decision trees also learn nonlinear relationships and feature interactions without requiring those relationships to be written as formulas. Because they do not compare features through distance, they generally do not require normalization or standardization.
The main weakness is overfitting. A deep tree can keep splitting on small differences and noise in the training data. Trees also have high variance: a small change in the dataset can produce a different split near the top and reshape the rest of the tree. Maximum depth, minimum samples per leaf, and pruning are common ways to control this behavior.
Decision trees are also greedy learners. At each step, they choose the split that produces the largest immediate reduction in impurity or loss. They do not revisit every previous decision to search globally across all possible tree structures, so the resulting tree is not guaranteed to be the best possible tree.
2. What kind of ensemble method is boosting?
The word ensemble comes from French, where it means together. In machine learning, an ensemble combines predictions from multiple models. Boosting builds this ensemble sequentially from weak learners, models whose predictive power is limited on their own. Each new learner is trained to compensate for mistakes left by the current ensemble.
This is different from bagging, where models can be trained independently and combined by averaging or voting. Boosting makes each stage depend on the result of the previous stages. That dependency limits full parallelism across trees, but it also allows many shallow trees to represent complex nonlinear relationships.
Boosting often performs well on tabular data. The trade-off is that sequential training can take longer, and learning rate, tree count, and tree depth interact with one another. An ensemble that grows too complex can also begin to fit noise.
Gradient Boosting
Gradient Boosting calculates a direction that reduces the current loss and adds a weak learner that approximates that correction. In regression, this is often introduced as fitting the residuals left by the current model. More generally, each new tree approximates the negative gradient of the chosen loss function.
The update can be written as:
Here, is the ensemble built so far, is the new weak learner that corrects the remaining error, and is the learning rate that controls how much the new learner contributes.
The method supports different differentiable loss functions and captures nonlinear relationships well. Its sequential nature can make training slow, however, and too many deep trees can overfit. A smaller learning rate makes each update more conservative, but it usually requires more trees.
XGBoost
XGBoost extends Gradient Boosting with a regularized objective and optimization techniques designed for efficient tree construction. It uses both first- and second-order derivative information, penalizes tree complexity, handles missing and sparse values, and supports pruning and parallel computation. The trees are still added sequentially; the parallel work happens mainly inside operations such as evaluating split candidates.
It offers strong performance on tabular data and many controls for managing overfitting. Those controls also create a larger hyperparameter surface, so configuration is less straightforward. Training time and memory usage still matter as datasets grow.
AdaBoost
AdaBoost begins by assigning the same weight to every training sample. After each round, it increases the weight of incorrectly predicted samples so the next weak learner pays more attention to them. The final prediction combines the learners through a weighted vote or weighted sum.
The mechanism is relatively easy to follow, and several simple weak learners can outperform a single model. The same weighting rule can become a weakness when the data contains outliers or incorrect labels, because difficult or noisy samples may receive increasingly large influence.
LightGBM
LightGBM bins continuous feature values into histograms to evaluate split candidates efficiently. Instead of expanding every node at the same depth, it uses leaf-wise growth and splits the leaf expected to reduce the loss the most.
This design provides fast training and lower memory usage on large datasets. It can also work directly with categorical features without requiring one-hot encoding. Leaf-wise growth can create a deep branch quickly, though, which increases the risk of overfitting on small datasets. The number of leaves, minimum data per leaf, and maximum depth therefore need attention.
CatBoost
CatBoost is a Gradient Boosting model designed around categorical features. Rather than requiring every categorical value to be one-hot encoded in advance, it calculates categorical statistics internally. It uses data ordering to prevent the current row’s target from leaking into the statistic used for that same row.
Ordered Boosting applies a related constraint. For each sample, the training signal is calculated from data that appears earlier in a random ordering, which keeps the sample’s own target from entering its update prematurely.
This reduces preprocessing work when a dataset contains many categorical features. The additional categorical statistics and ordered procedure can increase training time depending on the data and configuration. GPU training can also be nondeterministic because floating-point sums may be accumulated in a different order.
In short, AdaBoost increases the influence of incorrectly predicted samples, while Gradient Boosting adds trees in a direction that reduces the loss. XGBoost extends that process with regularization and systems optimizations. LightGBM focuses on histogram-based training and leaf-wise growth, while CatBoost focuses on handling categorical features without target leakage.
3. What is the difference between PCA and factor analysis?
I have not covered this topic in class yet, so I researched it separately for this paper.
Principal component analysis and factor analysis both represent many observed features with fewer variables. Their objectives and assumptions, however, are different.
PCA creates new axes called principal components as linear combinations of the original features. The first component captures the direction with the greatest variance. Each following component is orthogonal to the earlier components and captures as much of the remaining variance as possible. Its main purpose is to reduce the number of features while preserving as much variation in the dataset as possible.
Suppose a customer dataset contains purchase amount, purchase frequency, visit count, time spent, and coupon usage. PCA can compress these correlated values into a smaller number of components that preserve most of their variation. A component is still a mathematical combination of observed features; it does not necessarily correspond to a real concept such as loyalty or purchase intent.
Factor analysis starts with a different assumption. It treats correlations among observed features as effects of a smaller number of unobserved latent factors. In the customer example, purchase amount and purchase frequency may move together, while visit count and time spent may move together. Factor analysis asks whether latent factors such as purchasing tendency or engagement can explain those shared patterns.
Under this model, variation in each observed feature is divided into variation shared through common factors and variation unique to that feature, including measurement noise. PCA looks for axes that preserve total variance. Factor analysis tries to explain common covariance through latent factors.
| Category | PCA | Factor analysis |
|---|---|---|
| Main objective | Reduce dimensions while preserving variance | Explain correlations through latent factors |
| Output | Principal components formed from observed features | Latent factors that influence observed features |
| Treatment of variance | Uses total variance in the data | Separates common variance from feature-specific variance and noise |
| Common use | Compression, visualization, noise reduction, preprocessing | Finding latent constructs behind survey items or behavioral indicators |
PCA is a natural first choice when the goal is to reduce the number of model inputs or visualize high-dimensional data. Factor analysis is better suited to questions about hidden constructs that may explain correlations among survey items or behavioral measurements.