5  Tree Ensembles: Random Forests & Boosted Trees

Lecture 5 of ECON 8310.

5.1 Lecture Outline

  1. From Single Trees to Ensembles
  2. Bagging: Bootstrap Aggregating
  3. Random Forests: Decorrelating the Trees
  4. Feature Importance and Interpretation
  5. Gradient Boosting
  6. XGBoost
  7. Key Takeaways

Five parts. From single trees to ensembles. Bagging. Random forests. Feature importance. Then gradient boosting and XGBoost.

5.2 From Single Trees to Ensembles

Averaging many imperfect models can outperform any single model.

Part one, and it starts with an observation that has nothing to do with statistics.

5.2.1 The Wisdom of Crowds

A single expert can be wrong. Average many independent estimates and the errors tend to cancel.

Classic Example: Guessing the Ox’s Weight

At a 1907 county fair, 800 people guessed the weight of an ox. No individual was very accurate — but the average of all 800 guesses came within one pound of the true weight.

The same principle transfers directly to models. A single decision tree is unstable and noisy — that was Lecture 4’s closing complaint. Train many trees on different versions of the data and average them, and the noise largely cancels while the signal survives.

Ensembles work because independent errors cancel. And that word is doing real work: the more independent the individual predictions, the more variance averaging removes. Hold onto it — the entire difference between bagging and random forests turns on it.

In 1907, at a county fair, eight hundred people guessed the weight of an ox.

Francis Galton collected the guesses expecting to show that the crowd was foolish. Individually they were — almost nobody was close. But the average of all eight hundred guesses came within one pound of the true weight.

Why does that work? Because the errors were independent. Some people guessed high, some low, for unrelated reasons, and averaging cancelled them.

Now transfer that to models. A single decision tree is noisy — it splits somewhere slightly arbitrary and everything downstream inherits that arbitrariness. But if you train many trees on different versions of the data, each is noisy in a different way. Average them and the noise largely cancels while the signal, which they agree on, survives. I want you to notice the word doing all the work: independent.

Galton’s crowd worked because the guesses were unrelated. If eight hundred people had all overheard the same wrong estimate, averaging would have gained nothing.

Hold onto that. The entire difference between bagging and random forests — the whole reason random forests exist — turns on that one word.

5.3 Bagging: Bootstrap Aggregating

Train many trees on bootstrap samples; average their predictions.

Part two. Bootstrap aggregating.

5.3.1 Bagging: The Big Idea

Problem from Lecture 4: a single decision tree has very high variance — change a few training points and you get a completely different tree.

Bagging solution [@Breiman1996]:

  1. Draw \(B\) bootstrap samples (with replacement, same size as the original)
  2. Fit one deep decision tree on each
  3. Average all \(B\) predictions: \(\hat{y}_{\text{bag}} = \tfrac{1}{B}\sum_{b=1}^{B} \hat{f}^{(b)}(\mathbf{x})\)

Why it works. The variance of the averaged forecast is

\[\mathrm{Var}(\bar{f}) = \rho\sigma^2 + \tfrac{1-\rho}{B}\sigma^2,\]

where \(\rho\) is the pairwise correlation between tree predictions. The second term shrinks as \(B\) grows — so more trees never hurts. But the first term does not move at all.

That leaves a floor at \(\rho\sigma^2\), set by how alike the trees are rather than by how many you grow. Averaging a thousand nearly identical trees buys you almost nothing, and that is the problem the next section solves.

Three steps.

Draw B bootstrap samples — sample your training data with replacement, same size as the original. Because it’s with replacement, each sample gets some observations twice and misses others entirely. That’s the point: each sample is a slightly different view of the data.

Fit one deep tree on each. Deep — we’re not pruning. We want each tree to be low-bias and high-variance, because averaging is about to handle the variance. Then average all B predictions.

Now the equation, which is the most important thing on this slide. The variance of the averaged forecast is rho sigma squared, plus one-minus-rho over B, sigma squared. Two terms. Look at what happens as B — the number of trees — grows.

The second term has B in the denominator, so it shrinks toward zero. That’s why more trees never hurts. The first term has no B in it at all. It does not move.

So you have a floor at rho sigma squared, where rho is the pairwise correlation between the trees’ predictions.

Read what that says. If your trees are highly correlated — rho near one — the floor is near sigma squared, and averaging a thousand of them buys you essentially nothing. You have grown a thousand trees and reproduced one. That’s Galton’s crowd overhearing each other. And it’s exactly the problem the next section solves.

5.3.2 Out-of-Bag (OOB) Error: Free Cross-Validation

Each bootstrap sample draws \(n\) observations with replacement, so it contains about 63% of the distinct training points. The remaining 37% are out-of-bag for that tree — a free test set, different for every tree:

  1. For each observation \(i\), collect predictions only from the trees where \(i\) was out-of-bag
  2. Average them to get \(\hat{y}_i^{\text{OOB}}\)
  3. Compute the RMSE across all observations

Every one of those predictions came from a tree that never saw the observation — so the OOB RMSE is an honest out-of-sample estimate, and it lands close to 5-fold CV without refitting anything. In sklearn it is one argument: RandomForestRegressor(oob_score=True).

Below about 1,000 observations the OOB estimate gets noisy, because each observation is out-of-bag for too few trees. On a weekly business series — a few hundred rows — use proper walk-forward CV instead.

A pleasant side effect of bootstrapping.

Each bootstrap sample draws n observations with replacement, so some get picked more than once and some not at all. Work through the probability and you find each sample contains about sixty-three percent of the distinct training points. Which means roughly thirty-seven percent are left out of any given tree. Those are the out-of-bag observations.

Here’s the trick. For observation i, collect predictions only from the trees where i was out-of-bag. Average those. Compute the RMSE across all observations.

Every one of those predictions came from a tree that never saw that observation. So the OOB error is an honest out-of-sample estimate — and you got it without setting aside a validation set or refitting anything. It falls out of the bootstrapping you were doing anyway. In sklearn it’s one argument: oob_score=True. One caution, and it’s the one that matters for this course.

Below about a thousand observations, OOB gets noisy — each observation is out-of-bag for too few trees, so its OOB prediction is an average over a small, unstable set.

Our weekly series have two hundred seventy-seven observations. That is well under a thousand. So on this data, use proper walk-forward cross-validation. OOB is a genuine convenience on large datasets and a trap on small ones, and business time series are almost always small.

5.4 Random Forests: Decorrelating the Trees

Restrict each split to a random subset of features so trees are less alike.

Part three. Decorrelating the trees.

5.4.1 The Problem Bagging Does Not Fully Solve

Recall the variance of the averaged forecast: \(\mathrm{Var}(\bar{f}) = \rho\sigma^2 + \tfrac{1-\rho}{B}\sigma^2\). More trees shrink the second term to nothing. The first term never moves.

The correlation problem. If one feature dominates — and in forecasting it always does, because last year’s value is the best single predictor — then nearly every tree splits on it at the root. The trees end up near-identical, \(\rho\) stays large, and bagging buys you very little.

The Random Forest fix [@Breiman2001] is almost absurdly simple: at each split, consider only a random subset of \(m < p\) features, and take the best split among those.

Sometimes the dominant feature is not in the candidate set, so the tree is forced to find its second-best story. Across hundreds of trees that produces genuinely different structures — \(\rho\) falls, and the floor falls with it. You deliberately make each individual tree worse in order to make the ensemble better.

Back to the floor.

Why would trees end up correlated? Because of the greedy algorithm from last week. At the root, CART picks the single best split. If one feature is clearly the strongest predictor, then every tree picks it, and every tree starts the same way.

And in forecasting, one feature is essentially always dominant — last year’s value, or last week’s. So this isn’t an edge case. It’s the normal situation. The trees end up near-identical. Rho stays large. Bagging buys you very little. The Random Forest fix, from Breiman in 2001, is almost absurdly simple.

At each split, don’t consider all p features. Consider a random subset of m, and take the best split among those. That’s it. That’s the whole method.

Think about what it does. Sometimes the dominant feature simply isn’t in the candidate set. The tree is forced to find its second-best story — and then a third tree finds a different one. Across hundreds of trees you get genuinely different structures. Rho falls, and the floor falls with it.

And here’s the part I find genuinely elegant: you are deliberately making each individual tree worse in order to make the ensemble better. Any single random forest tree is a poorer model than a single bagged tree. It was handicapped. But the ensemble beats it, because the handicap bought independence. That is one of the more counterintuitive ideas in applied statistics, and it works.

5.4.2 Random Forest Hyperparameters

Parameter sklearn name Default Effect
Number of trees n_estimators 100 More = lower variance; plateaus ~500
Max features/split max_features 'sqrt' Lower = more decorrelated
Max depth max_depth None Limit only to reduce memory
Min samples/leaf min_samples_leaf 1 Higher = smoother predictions
OOB score oob_score False True gives a free CV estimate

Start with n_estimators=500, max_features='sqrt', min_samples_leaf=5, oob_score=True. Note how few genuinely need tuning — max_features is the dial controlling \(\rho\), and the rest is close to fire-and-forget. That is most of why random forests make a good default.

Set random_state=42: a forest is random twice over, in the bootstrap samples and in the feature subsets.

Five parameters, and the striking thing is how few need attention.

n_estimators — number of trees. More is better and plateaus around five hundred. There is no overfitting risk from adding trees; that’s the first term of the variance equation, which doesn’t care.

max_features — this is the one that matters. It’s m from the previous slide. It is the dial controlling rho, and therefore the dial controlling the floor. Lower means more decorrelated trees. The 'sqrt' default is a reasonable starting guess, not a law. max_depth — leave it at None. We want deep trees. min_samples_leaf — raise it to about five for smoother predictions. oob_score — set True if your dataset is large enough for it to mean anything.

Start with five hundred trees, sqrt features, min_samples_leaf of five.

And note the practical consequence: essentially one parameter needs real tuning. Compare that to what’s coming with XGBoost, where you’ll be juggling six. A large part of why random forests are the right default is that they are very hard to misconfigure.

Set random_state=42. A forest is random twice over — in the bootstrap samples and in the feature subsets — so without a seed your results move between runs and you’ll waste an hour wondering why.

5.5 Feature Importance and Interpretation

Which predictors matter most? RF gives a data-driven answer.

Part four. Which predictors matter?

5.5.1 Random Forest Feature Importance [@Molnar2022]

Mean Decrease in Impurity (MDI)

For each feature \(j\), average the total reduction in node impurity across every split on \(j\), weighted by the fraction of observations reaching that node. Normalize to sum to 1.

Feature (CA_1 FOODS, weekly) Importance
units_lag4 0.45
units_lag1 0.18
units_lag52 0.11
snap_days 0.08
units_lag2 0.04
avg_price 0.04

Read it as a screening device, not an explanation. MDI is biased toward features with many split points. permutation_importance — shuffle a column, see how much accuracy falls — is the honest alternative.

Mean decrease in impurity. For each feature, sum the variance reduction across every split where it was used, weighted by how many observations passed through that node. Normalize to one.

On our CA_1 food sales the ranking comes out: four-week lag at zero point four five, last week at zero point one eight, same week last year at zero point one one, and SNAP benefit days at zero point zero eight — ahead of price.

That SNAP result is worth pausing on. Nobody told the model that food sales spike when benefits are disbursed. It found it, and ranked it above price. That is a real business finding, and no linear specification would have surfaced it unless someone already suspected it. But read this as a screening device, not an explanation, because MDI is biased.

The bias is toward features with many possible split points. A continuous variable like price offers hundreds of thresholds to try. A binary flag offers one. The continuous variable gets more chances to look useful — not because it carries more signal, but because it had more opportunities.

permutation_importance is the honest alternative: shuffle one column and measure how much accuracy falls. If shuffling it doesn’t hurt, it wasn’t doing anything. That’s a direct measurement rather than an accounting artifact.

Use MDI to narrow forty candidates to eight. Use permutation importance before you put a ranking in front of anyone who’ll act on it.

5.5.2 Random Forest in Python

sklearn Code Template

from sklearn.ensemble import RandomForestRegressor

rf = RandomForestRegressor(n_estimators=500, max_features='sqrt',
                           min_samples_leaf=5, oob_score=True,
                           random_state=42).fit(X_tr, y_tr)
print(rf.oob_score_)
importances = rf.feature_importances_
Model (CA_1 FOODS, 52-week test) Test RMSE
Seasonal naïve 1,660
Single deep tree 1,415
Random Forest 1,020

The forest cuts RMSE roughly 28% against a single deep tree, and is the first method in this course to beat the seasonal naive benchmark on this series.

What changed is not a better model of the series but a richer feature set — lags, price, and SNAP days are information ETS and ARIMA never had.

The code is four lines, and the results table is the interesting part.

On CA_1 food sales, fifty-two week test: seasonal naive at 1,660. A single deep tree at 1,415. The random forest at 1,020.

Two things.

The forest cuts RMSE about twenty-eight percent against the single tree. That’s the variance reduction, bought purely by averaging — same algorithm, same features, five hundred of them instead of one.

And it is the first method in this course to beat the seasonal naive benchmark on this series. In Homework 1 you found that exponential smoothing couldn’t. ARIMA couldn’t either. This does, comfortably.

Now — be careful about the conclusion you draw, because the obvious one is wrong.

What changed is not that random forests model time series better than ARIMA. It’s that we gave this model information the others never had: lags, price, SNAP days, event flags. ETS and ARIMA only ever saw the series’ own history.

That’s the real lesson of the second half of this course. The gains from here are coming substantially from feature engineering, not from model sophistication. A random forest with one lag feature would lose to ARIMA. A random forest with a well-built feature set wins.

5.6 Gradient Boosting

Instead of averaging independent trees, fit each new tree to the errors of all previous trees.

Part five, and we change direction entirely.

5.6.1 Boosting vs. Bagging: The Core Difference

Bagging (Random Forest):

  • Trees built in parallel, independently
  • Each tree predicts \(y\) directly
  • Averaging reduces variance
  • More trees never reduces bias

Boosting:

  • Trees built sequentially
  • Each fits the errors of the ensemble so far
  • Reduces bias as well as variance
  • Will overfit if trees are too deep or too many

The intuition. Forecast 1 says $50,000; actual is $60,000, so the error is +$10,000. Forecast 2 does not predict sales at all — it predicts that error, and says “the first model was short by $10,000.” Add them: $60,000. Each stage corrects what remains, which is why boosting can fix a systematic bias that averaging never touches.

Bagging builds trees in parallel and independently. Each tries to predict y directly. Averaging reduces variance. And critically — adding more trees never reduces bias. If your trees are systematically wrong in the same direction, averaging preserves that exactly.

Boosting builds trees sequentially. Each new tree fits the errors of everything built so far. That reduces bias as well as variance — and it can overfit, which bagging essentially cannot. The intuition on the slide is the clearest version I know. Forecast one says fifty thousand dollars. The actual is sixty thousand. The error is plus ten thousand.

Now, forecast two does not try to predict sales at all. It tries to predict that error. It says: “the first model was short by about ten thousand.” Add them together: fifty plus ten is sixty.

Each stage corrects what remains. And notice this fixes something averaging cannot touch — if every bagged tree under-predicts December by the same amount, the average under-predicts December. Boosting sees that residual pattern and builds a tree specifically for it. Bagging attacks variance. Boosting attacks bias. That sentence is the whole slide.

5.6.2 The Gradient Boosting Algorithm

Gradient Boosting (squared-error loss)

Initialize \(F_0(\mathbf{x}) = \bar{y}\). For \(b = 1, \ldots, B\):

  1. Compute residuals: \(r_i^{(b)} = y_i - F_{b-1}(\mathbf{x}_i)\)
  2. Fit a shallow tree \(T_b\) to \(\{(\mathbf{x}_i, r_i^{(b)})\}\)
  3. Update: \(F_b = F_{b-1} + \eta\, T_b\)

Final prediction: \(\hat{y} = F_B(\mathbf{x})\)

The trees are deliberately shallow — depth 3 to 6 — because each one only has to capture a small piece of the remaining error. The ensemble converges slowly and on purpose.

The learning rate \(\eta\) is the main dial. Small values (0.01–0.1) take a tiny step per tree: slower, needs more trees, far more robust. Large values (\(> 0.3\)) move fast and risk overshooting. The reliable recipe is a small \(\eta\), many trees, and early stopping — without which boosting will happily fit your noise.

Formally, three steps in a loop.

Start with the mean. Then repeatedly: compute the residuals, fit a shallow tree to those residuals, and add a fraction of that tree to the running prediction.

Two design choices worth understanding.

The trees are deliberately shallow — depth three to six. That’s the opposite of bagging, where we grew them deep. Here each tree only needs to capture a small piece of the remaining error, so a small tree is enough. A deep tree would fit the residual noise, and the next iteration would be trying to correct noise.

The learning rate eta is the main dial. After fitting a tree to the residuals, you don’t add the whole thing — you add eta times it, where eta might be zero point zero five. Which means you deliberately under-correct at every step. Why on earth would you do that?

Because each tree is fitted to the residuals of one particular sample, so part of what it sees is noise. Take the full correction and you bake that noise in. Take five percent of it, five hundred times, and the noise averages out while the signal accumulates.

The reliable recipe is: small eta, many trees, and early stopping. Without early stopping, boosting will keep reducing training error until it has fitted your noise perfectly. Unlike bagging, this method genuinely can overfit — and it will if you let it.

5.7 XGBoost

Three improvements over vanilla boosting: second-order optimization, explicit regularization, and column subsampling.

The version that won everything.

5.7.1 XGBoost: Three Advances [@Chen2016]

  1. Newton step (second-order optimization). Uses the gradient and the curvature of the loss to choose each tree’s direction, so it converges in fewer trees than first-order boosting.
  2. Explicit regularization. Penalizes the number of leaves (\(\gamma\)) and the magnitude of leaf weights (\(\lambda\)) directly in the objective — so overfitting is controlled by the loss function rather than only by early stopping.
  3. Column subsampling. Borrowed straight from random forests: sample features per tree and per split, decorrelating the ensemble.

Notice that the second advance is exactly the Ridge and LASSO idea from next week, applied to leaf weights instead of regression coefficients. reg_lambda is an \(\ell_2\) penalty; gamma penalizes complexity the way \(\ell_1\) does.

XGBoost dominated Kaggle tabular competitions from 2014–2018. LightGBM is faster on large data; CatBoost handles categoricals natively.

Three improvements over vanilla gradient boosting.

The Newton step. Vanilla boosting uses the gradient — the direction of steepest descent. XGBoost also uses the second derivative, the curvature. Knowing how sharply the loss is bending lets you take a better-sized step, so it converges in fewer trees.

Explicit regularization. This is the important one. XGBoost puts penalties directly in the objective — gamma on the number of leaves, lambda on the magnitude of the leaf weights. So overfitting is controlled by the loss function itself, not only by stopping early.

Column subsampling. Borrowed straight from random forests: sample features per tree and per split. The decorrelation idea from slide nine, imported into boosting.

Notice what that second advance actually is.

reg_lambda penalizes the sum of squared leaf weights. That is an L2 penalty — it is Ridge regression, applied to leaf weights instead of regression coefficients. And gamma penalizes the count of leaves, which is what LASSO does to nonzero coefficients. You are seeing next week’s lecture, one week early, wearing different clothes.

That’s not a coincidence. Regularization is not a linear-model topic. It is how every flexible model is kept honest, and you’ll meet it a fourth time in Lecture 7 as weight decay and dropout.

XGBoost dominated Kaggle tabular competitions from 2014 to 2018. LightGBM is faster on large data; CatBoost handles categorical features natively. The ideas are the same.

5.7.2 XGBoost: Key Hyperparameters

Parameter XGB name Typical Effect
Number of trees n_estimators 500–2000 More + small \(\eta\) = better
Learning rate learning_rate 0.01–0.1 Smaller = more robust
Tree depth max_depth 3–6 Shallow preferred
Row subsample subsample 0.7–0.9 Fraction of training rows per tree
Col. subsample colsample_bytree 0.7–0.9 Feature fraction per tree
L2 leaf penalty reg_lambda 1 Ridge on leaf weights

Practical starting point: learning_rate=0.05, max_depth=4, n_estimators=1000 with early stopping on validation RMSE. Tune max_depth and subsample via TimeSeriesSplit.

Six parameters, against the random forest’s one that mattered. This is the cost of the extra accuracy.

Number of trees, five hundred to two thousand — high, because you’re pairing it with early stopping. Learning rate zero point zero one to zero point one. Depth three to six, shallow. Row subsample and column subsample around zero point eight. And reg_lambda, the Ridge penalty on leaf weights.

The two that interact most are learning_rate and n_estimators, and they trade off directly. Halve the learning rate and you need roughly twice the trees. Small eta with many trees is almost always better than large eta with few — you’re just paying for it in compute.

Practical starting point: learning rate zero point zero five, depth four, a thousand trees with early stopping on validation RMSE. Then tune depth and subsample by TimeSeriesSplit.

5.7.3 XGBoost in Python

Code Template

model = xgb.XGBRegressor(n_estimators=1000, learning_rate=0.05,
                         max_depth=4, subsample=0.8, colsample_bytree=0.8,
                         early_stopping_rounds=50, random_state=42)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)

eval_set is what makes early_stopping_rounds work: XGBoost watches validation RMSE and halts when it stops improving, which is how you get away with setting n_estimators far higher than you need.

The validation set must respect time. Build it with TimeSeriesSplit and hold out the final stretch as a true test set.

On macOS xgboost also needs OpenMP: brew install libomp, or the import fails.

The code, and one argument that matters more than it looks.

eval_set is what makes early_stopping_rounds work. XGBoost watches RMSE on that set and halts when it stops improving. That’s how you get away with setting a thousand trees — you’re not committing to a thousand, you’re setting a ceiling and letting the data decide.

And the validation set has to respect time. Build it with TimeSeriesSplit, hold out the final stretch as a true test set, and never let a random split choose which weeks the model sees.

If you use a random split here, early stopping will pick the iteration that best predicts weeks the model has effectively already seen. You’ll stop too late, ship an overfitted model, and your diagnostics will look excellent.

One practical note that will save you an evening. On macOS, xgboost needs the OpenMP runtime. If you get an error about libomp.dylib not loading, run brew install libomp. The Python package installs fine and then fails at import, which makes it look like a code problem when it’s a system library problem.

5.8 Key Takeaways and Roadmap

Bagging reduces variance; boosting reduces bias. Together they cover most of what tabular forecasting needs.

Let’s pull it together.

5.8.1 Tree Methods: Which One When

Method Training Interpretable Tuning Accuracy
Decision Tree Single tree High Low Baseline
Random Forest Parallel (\(B\) trees) Medium Low Good
XGBoost Sequential (\(B\) trees) Low Higher Best

Random Forest is the better default: robust, almost no tuning, OOB error for free. Move to XGBoost when you can afford to tune and need the last few points — and budget for that tuning, because an untuned XGBoost often loses to a default forest.

Neither can extrapolate beyond the training range. A tree predicts the mean of a leaf, and no leaf contains values it never saw. On a trending series, both will flatten out exactly when you need them to keep climbing.

Three rows.

Decision tree — single tree, highly interpretable, minimal tuning, baseline accuracy. Random forest — parallel, medium interpretability, low tuning, good accuracy. XGBoost — sequential, low interpretability, higher tuning, best accuracy.

My advice is that random forest is the better default, and I want to be clear about why, because “XGBoost is best” is what the accuracy column says.

A random forest is robust, needs almost no tuning, and hands you OOB error for free. It is genuinely hard to make one catastrophically wrong.

Move to XGBoost when you can afford to tune and you need the last few points. And budget for that tuning honestly — an untuned XGBoost frequently loses to a default random forest. The accuracy in that column is the accuracy of a well-tuned XGBoost, and that word is carrying weight. And the warning box applies to every row.

Neither can extrapolate. A tree predicts the mean of a leaf, and no leaf contains values it never saw. On a trending series, both flatten out exactly when you need them to keep climbing.

We put a number on that last week: CA_1 food sales grew twenty-nine percent over five years. Train on the first four, and the highest number a tree can output is a leaf mean from the older, lower range.

So the classical methods did not become obsolete today. If your series trends and you need to forecast beyond its historical range, an ARIMA with drift will do something a forest simply cannot.

5.8.2 Lecture 5: Key Takeaways

  1. Ensembles work because independent errors cancel. Averaging many imperfect models beats any single model.

  2. Bagging averages \(B\) trees fit to bootstrap samples; variance falls as \(B\) grows but floors at \(\rho\sigma^2\). Random Forests subsample features at each split to cut \(\rho\) — the key gain over plain bagging.

  3. OOB error is free cross-validation: each observation sits out ~37% of trees, giving an unbiased generalization estimate.

  4. Boosting inverts the idea — trees are built sequentially, each fitting the ensemble’s remaining errors. Bagging attacks variance; boosting attacks bias. XGBoost adds a Newton step, penalties on leaf count and weights, and column subsampling.

  5. Tune by cross-validation, always with TimeSeriesSplit — a random split leaks the future into the past — and pair boosting with early stopping.

Next: Regularization & Model Selection — choosing among many predictors by shrinking coefficients rather than counting them.

Five things. One. Ensembles work because independent errors cancel. The word independent is load-bearing.

Two. Bagging averages B trees on bootstrap samples. Variance falls as B grows but floors at rho sigma squared. Random forests subsample features at each split to cut rho — the key gain over plain bagging. Three. OOB error is free cross-validation — but it needs a large dataset, and ours is not.

Four. Boosting inverts the idea. Trees built sequentially, each fitting what remains. Bagging attacks variance; boosting attacks bias. XGBoost adds a Newton step, explicit penalties, and column subsampling. Five. Tune by cross-validation, always with TimeSeriesSplit, and pair boosting with early stopping.

And one thing that isn’t on the slide. The jump from ARIMA to random forest on our data was large — sixteen sixty down to ten twenty. Most of that came from the feature set, not the algorithm. Remember that when you’re tempted to reach for a more complicated model instead of a better predictor.

5.8.3 References

Reading is ISLP section 8.2 — bagging, random forests, and boosting — plus the XGBoost “Introduction to Boosted Trees” tutorial, which derives the regularized objective we waved at on slide 18.

Next week: Regularization and Model Selection. We take the penalty idea that showed up inside XGBoost and give it a lecture of its own. See you then.

5.9 Timing guide

Segment Slides Target
Opening & wisdom of crowds 1–4 ~4 min
Bagging & OOB 5–7 ~6 min
Random forests 8–10 ~6 min
Feature importance & Python 11–13 ~6 min
Boosting & XGBoost 14–20 ~9 min
Close 21–24 ~4 min

This is the longest deck in the course. If you must compress, slides 10 and 19 — the two hyperparameter tables — survive trimming best. Do not compress slides 6, 9, or 15: the variance floor, the decorrelation fix, and the bagging-versus-boosting distinction are what the whole lecture is built on.