The Limits of Linear Models

Three things a constant slope cannot represent:

• Interactions — the holiday-season advertising effect is not a constant slope
• Threshold effects — spending may be flat until unemployment drops below 5%, then jump
• Asymmetry — a rise and a fall of the same size move the forecast equally, even when the real response is lopsided

Decision trees capture nonlinearity and interactions automatically, without your having to specify them in advance.

The Bias–Variance Tradeoff (James et al. 2023)

Weekly CA_1 FOODS, 20 bootstrap refits per model. Stumps (one-question trees) agree and miss: bias. Deep trees center on the truth but scatter: variance. (The miss includes noise, so it is not pure bias.) Goal: the complexity that balances the two, visible only out-of-sample.

What Is a Decision Tree?

A flowchart of yes/no questions about the features, ending in a prediction. Here is the real depth-2 tree for weekly CA_1 FOODS units:

Non-parametric · nonlinear (captures thresholds) · interpretable · scale-invariant. A single deep tree overfits severely; ensembles, the next lecture, are the fix.

\(y_{t-4}\): units four weeks earlier; \(y_{t-52}\): the same week last year. Fitted on the 173 weeks before the test year.

The Same Tree, Drawn as Rectangles

Each question is a straight cut parallel to an axis. Each leaf is a rectangle, shaded by its mean; every week inside it gets that forecast.

Last year’s sales cut only the right half: they matter only when recent demand is already high. That is an interaction, and nobody specified it.

The CART Algorithm (Regression) (Breiman et al. 1984)

Greedy Is Not Optimal — and That Matters

CART is exhaustive at each node: it tries every feature and every threshold. But it is greedy across nodes: it never looks ahead. An illustration:

CART takes the 40%. It cannot see the 50% coming: it scores one split at a time.

So a fitted tree is a sequence of locally best choices, not the optimal tree (finding that is intractable). It can miss real structure, and a small change in the data can send the greedy sequence down a different path: the instability we turn to next.

Depth Controls the Bias–Variance Tradeoff

A stump asks one question: high bias, low variance. A full tree, one week per leaf, memorizes the sample: zero training error, high variance.

Weekly CA_1 FOODS, one week ahead (seasonal naive: 1,660). Training error falls to zero; test error stops improving after depth 3 (only 173 weeks). Read the shape, not the minimum: picking depth from the test year is a leak. Unstable, too: one changed week can flip the root split, and everything below it. Tune by TimeSeriesSplit.

Two Ways to Stop a Tree Overfitting

max_depth caps the tree in advance. Two other controls do the job better.

Choose either one by TimeSeriesSplit cross-validation, never by the test year.

Cost-Complexity Pruning: A Line Touching the Curve

Pruning minimizes \(\sum_{\text{leaves}} \text{SSE} + \alpha\,|T|\): the fit, plus a price \(\alpha\) for each of the \(|T|\) leaves. Each black dot is the best pruned tree for some range of \(\alpha\).

Push a line of slope \(-\alpha\) up from below: the first dot it touches is the tree \(\alpha\) picks. A bigger \(\alpha\) tilts it and picks a smaller tree. The 4-leaf tree is the flowchart from earlier.

The same idea as Lecture 3’s smoothing penalty and Lecture 6’s Ridge and LASSO; \(\alpha\) is chosen by cross-validation. scikit-learn’s ccp_alpha is \(\alpha/n\).

Decision Tree Regression in Python

from sklearn.tree import DecisionTreeRegressor

X, y = weekly[FEATURES], weekly['units']   # lags, SNAP, events, price

tree = DecisionTreeRegressor(max_depth=5, min_samples_leaf=10, random_state=42)
tree.fit(X_train, y_train)
y_hat = tree.predict(X_test)

Cross-validate with TimeSeriesSplit, never KFold — random folds leak the future into the training set.

Feature Importance (Molnar 2022)

From the previous slide’s depth-5 tree. Read it as a list of what the tree leaned on: a fast way to screen predictors.

But impurity importance is biased toward features with many split points. Permutation importance gives a more reliable ranking; we return to it in Lecture 5.

Interpreting and Communicating Tree Predictions

A tree’s main advantage is that you can explain why a specific prediction was made.

Produce the path with sklearn.tree.plot_tree(tree) or export_text(tree, feature_names=[...]).

This is where business buy-in comes from — a manager can check that the tree is splitting on variables they consider sensible.

When a Single Tree Falls Short

The ensemble solution: average many trees fitted to bootstrap samples. Variance falls sharply and feature importance survives, typically for 20–40% lower RMSE than one tree.

Lecture 4: Key Takeaways

Next: Tree Ensembles — variance reduction through bootstrap aggregation, and sequential error correction with boosting.