The Wisdom of Crowds

A single expert can be wrong. Average many independent estimates and the errors tend to cancel.

Models work the same way. One tree is unstable, Lecture 4’s closing complaint. Average many trees fitted to different versions of the data and the noise largely cancels while the signal survives.

Ensembles work because independent errors cancel. The more independent the trees, the more averaging removes: the difference between bagging and random forests turns on that word.

Bagging: The Big Idea

Problem from Lecture 4: one tree has high variance. Bagging (Breiman 1996) averages it away:

1. Draw \(B\) bootstrap samples (with replacement, the same size as the data)
2. Fit one deep decision tree to each
3. Average the \(B\) predictions: \(\hat{y}_{\text{bag}} = \tfrac{1}{B}\sum_{b=1}^{B} \hat{f}^{(b)}(\mathbf{x})\)

Why it works, and where it stops. With \(\sigma^2\) one tree’s variance and \(\rho\) the correlation between two trees’ predictions,

\[\mathrm{Var}(\bar{f}) = \rho\sigma^2 + \tfrac{1-\rho}{B}\sigma^2 .\]

More trees shrink the second term; the first never moves.

More Trees, or Less Alike Trees

Divide by \(\sigma^2\) and each \(\rho\) draws its own curve, flattening onto a floor at \(\rho\):

Growing more trees only slides you along a curve. The floor is set by how alike the trees are, not by how many you grow: to get below it, make them less alike.

Out-of-Bag (OOB) Error: Free Cross-Validation

A bootstrap sample holds about 63% of the distinct rows. Score each row with only the trees that never saw it (oob_score=True): no refitting. But look at who scores it:

Weekly CA_1 FOODS forest. Gray: in that tree’s bootstrap sample; white: out of bag.

The Problem Bagging Does Not Fully Solve

If one feature dominates (dow daily, the four-week lag weekly), nearly every bagged tree splits on it first and \(\rho\) stays high. The random forest fix (Breiman 2001): at each split, consider only a random \(m < p\) of the features.

Lab 5 (daily CA_1 FOODS), 300 trees each. The fix spreads the roots, and its trees average better (417 to 254, against 398 to 248), but they start worse: with one dominant feature among ten, sqrt loses.

Random Forest Hyperparameters

The default is plain bagging. scikit-learn’s regressor considers every feature at every split; 'sqrt' is the classifier’s default. So set max_features yourself: it is the dial for \(\rho\), and the only one that really needs tuning. Start with n_estimators=500, min_samples_leaf=5.

Set random_state=42: a forest is random twice over, in the bootstrap samples and in the feature subsets.

Random Forest Feature Importance (Molnar 2022)

From the next slide’s forest; Lecture 4’s single tree gave the four-week lag 0.62. A screening device only: MDI is biased toward features with many split points. permutation_importance (shuffle a column, measure the damage) is the honest alternative.

Random Forest in Python

from sklearn.ensemble import RandomForestRegressor

rf = RandomForestRegressor(n_estimators=500, max_features='sqrt',
                           min_samples_leaf=5, oob_score=True,
                           random_state=42).fit(X_tr, y_tr)
print(rf.oob_score_)
importances = rf.feature_importances_

The forest cuts RMSE by 23% against a single deep tree. Both beat seasonal naive, but they forecast only one week ahead; Lectures 1–2 forecast a whole year.

One week ahead, SES and ARIMA beat 1,660 too, and a forest on the series’ lags alone scores 1,231: most of the gain is last week’s actual sales, not price or SNAP.

Boosting vs. Bagging: The Core Difference

The intuition. Forecast 1 says $50,000 against an actual $60,000; forecast 2 predicts the error, +$10,000, and the sum is right. Averaging never fixes a bias like that.

The Gradient Boosting Algorithm

Trees are shallow (depth 3 to 6): each captures a little of what remains.

The learning rate \(\eta\) is the main dial: 0.01–0.1 is slow and robust, above 0.3 risks overshooting. Use a small \(\eta\), many trees and early stopping, or boosting fits your noise.

Boosting, Tree by Tree

The algorithm on one feature, so the fit can be drawn: weekday electricity load against day of the year (Lecture 3’s data). Depth-3 trees, \(\eta = 0.3\), starting from the mean (dashed):

Each tree fits what the curve still misses, so the year’s W emerges a piece at a time. By 50 trees the training RMSE is 2.89 GW, against 2.86 for the best any day-of-year curve can do: what remains is noise this feature cannot explain.

XGBoost: Three Advances (Chen and Guestrin 2016)

1. Newton step. Uses the loss’s curvature as well as its slope. For squared error the curvature is constant, so this is the ordinary residual fit; it matters for log-loss or Poisson.
2. Explicit regularization. Penalties on leaf count and leaf size, inside the objective.
3. Column subsampling, borrowed from random forests.

reg_lambda is Ridge and reg_alpha is LASSO, on leaf weights (next week). gamma is a price per leaf, like Lecture 4’s ccp_alpha.

XGBoost dominated Kaggle tabular competitions in 2014–2018.

XGBoost: Key Hyperparameters

XGBoost in Python

model = xgb.XGBRegressor(n_estimators=1000, learning_rate=0.05,
                         max_depth=4, subsample=0.8, colsample_bytree=0.8,
                         early_stopping_rounds=50, random_state=42)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)

eval_set is what makes early_stopping_rounds work: XGBoost watches validation RMSE and halts when it stops improving, which is how you get away with setting n_estimators far higher than you need.

The validation set must respect time. Build it with TimeSeriesSplit and hold out the final stretch as a true test set.

On macOS xgboost also needs OpenMP: brew install libomp, or the import fails.

Tree Methods: Which One When

Random Forest is the better default: robust, almost no tuning, OOB error for free. Move to XGBoost when you can afford to tune and need the last few points — and budget for that tuning, because an untuned XGBoost often loses to a default forest.

Why No Tree Can Extrapolate

A forest’s prediction is a weighted average of training targets, so it can never leave [training min, training max]. A forest trained on US retail sales through 2019, then given each later month’s true lags:

It never forecasts above $450bn; by mid-2026 sales are $665bn. Boosted trees go flat beyond their last split too. On a trending series, keep a model that can extrapolate.

Lecture 5: Key Takeaways

Next: Regularization & Model Selection.