import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor, export_text, plot_tree
import warnings
warnings.filterwarnings('ignore')
from pathlib import Path
def data_path(name):
"""Find the processed data file whether you render from the repo root or from
this file's own folder."""
for base in (Path.cwd(), *Path.cwd().parents):
p = base / "data" / "processed" / name
if p.exists():
return p
raise FileNotFoundError(
f"Could not find data/processed/{name}. "
"Run `python scripts/prep_m5.py` from the repository root first."
)
RANDOM_STATE = 42
daily = pd.read_csv(data_path("m5_daily.csv"), parse_dates=["date"])
daily["dow"] = daily["date"].dt.dayofweek # 0 = Monday
daily["doy"] = daily["date"].dt.dayofyear
FEATURES = ["units_lag1", "units_lag7", "units_lag14", "units_lag28", "units_lag365",
"snap", "is_event", "dow", "doy", "avg_price"]
df = daily.dropna(subset=FEATURES).copy()
N_TEST = 365
train, test = df.iloc[:-N_TEST], df.iloc[-N_TEST:]
X_train, y_train = train[FEATURES].values, train["units"].values
X_test, y_test = test[FEATURES].values, test["units"].values
rmse = lambda a, b: float(np.sqrt(np.mean((np.asarray(a) - np.asarray(b)) ** 2)))
print(f"{len(df)} usable days — train {len(train)}, test {len(test)}")
print(f"benchmark (same day last year): RMSE {rmse(y_test, test.units_lag365.values):,.0f}")4 Lab 4: Decision Trees — Reading One, and Finding Where It Breaks
Run this one live: the executable lab.
4.1 What we are doing
Today the course changes character. Everything so far specified a structure and estimated it. A tree is handed a pile of features and finds the structure itself.
We use the same series you just forecast in Homework 2 — daily units for CA_1 FOODS. Same store, same aisle, same target. Last week you fitted a GAM to it. Today you fit a tree, and the comparison is the point: you will be able to say what each method bought you on identical data.
By the end you will have done four things: read a fitted tree out loud, watched the bias–variance tradeoff happen in a table, discovered why a single tree is unstable, and found the one place trees reliably fail.
Suggested time: ~40 minutes · You will need: pandas, numpy, matplotlib, scikit-learn, and data/processed/m5_daily.csv (run python scripts/prep_m5.py if it is missing).
4.2 Setup
Notice what just happened to the data. There is no date index in X_train. Every row is an independent example, and everything the model knows about time is sitting in the lag columns. That reframing — a time series becomes a flat table — is what lets every method from here to Lecture 9 be applied to forecasting at all.
Walkthrough
1576 usable days — train 1211, test 365
benchmark (same day last year): RMSE 625Just over four years of daily data, holding out the final year — May 2015 through May 2016.
The benchmark is the same one you have used since Lecture 1: predict today with the same day last year. 625 units. Write it down. Every number today gets measured against it.
Now look at what we handed the model, because this is the important part of the setup.
Ten columns: five lags, a SNAP flag, an event flag, day of week, day of year, and price. And no date index. There is no time in X_train at all. Every row is an independent example, and everything the model knows about when it is, is sitting in those columns because we put it there.
That reframing — a time series flattened into an ordinary table — is what lets everything from here to Lecture 9 be applied to forecasting at all. Trees, forests, boosting, neural networks: none of them know what a time series is. They all consume this table.
4.3 Step 1 — Fit a tree and read it out loud
Start deliberately shallow, because a depth-3 tree is small enough to read.
tree3 = DecisionTreeRegressor(max_depth=3, random_state=RANDOM_STATE).fit(X_train, y_train)
print(export_text(tree3, feature_names=FEATURES, decimals=0))
print(f"\ntest RMSE: {rmse(y_test, tree3.predict(X_test)):,.0f}")fig, ax = plt.subplots(figsize=(13, 5))
plot_tree(tree3, feature_names=FEATURES, filled=True, rounded=True, fontsize=7, ax=ax)
plt.tight_layout()Put the root split into business language — not dow <= 4.50 but what that means about this store.
In the room: read it aloud to the person next to you. Saying it to someone is what exposes a sentence you do not actually understand.
On your own: write the sentence in a comment, then add a prediction you can check immediately:
# the root split separates ____ from ____
# I expect the higher-sales side to be dow <= 4.50 / dow > 4.50, because ____Then read the two node values off the tree plot above and see whether you were right. A guess you wrote down and checked teaches you more than a split you merely read.
Then look at the importances:
imp = pd.Series(tree3.feature_importances_, index=FEATURES).sort_values(ascending=False)
print(imp[imp > 0].round(3).to_string())
Compare this with the lecture slide
Lecture 4’s importance table was computed on weekly CA_1 FOODS, and there the four-week lag dominated. Here, on daily data for the same store and aisle, something else wins outright.
Same store. Same product. Different frequency, different answer. Say why before moving on.
Walkthrough
Depth 3, deliberately. Eight leaves is small enough to read in full, and you should read it in full, because being able to do that is most of what a tree is for.
|--- dow <= 4
| |--- units_lag28 <= 2596
...
|--- dow > 4
| |--- units_lag1 <= 3351
...
test RMSE: 337The root split is dow <= 4. That is Monday-through-Friday on one side, Saturday and Sunday on the other — and the tree found it without being told that a week has a weekend in it.
Try it
Pause here and say the whole path out loud, in business language rather than in inequalities. Not “dow less than or equal to 4.5, then units_lag28 above 2596.” Something closer to: it’s a weekday, and four weeks ago was already busy, and last week was busy too, so predict about 2,950.
If you can do that for one leaf, you can do it for a store manager, and that is a thing no model in Lectures 1 through 3 could offer you.
One branch worth reading before we move on. On the weekend side, on days when yesterday was already very busy, the tree splits on is_event — and it predicts 4,041 when there is no event and 3,583 when there is. The event lowers the forecast by about 450 units. Events in M5 include days like Christmas, when the store is shut or nearly so. The tree has no idea what Christmas is. It found the hole in the data anyway.
And the accuracy: 337 against a benchmark of 625. Roughly a 46% improvement, from eight leaves you can print on a slide.
dow 0.789
units_lag28 0.106
units_lag1 0.064
units_lag7 0.030
is_event 0.011Day of week takes seventy-nine percent of the importance. On its own, day-of-week means explain about 66% of the variance in training units — weekday average 2,632, weekend average 3,773, a gap of 43%. There is no mystery about why the tree grabbed it first.
Try it
Lecture 4’s importance table, on the weekly version of this same store and aisle, had units_lag4 at 0.62 and day-of-week nowhere — it isn’t in the table at all. Here, on daily data, dow wins outright with 0.79 and the four-week lag drops to 0.11. Same store. Same product. Opposite answer. Pause and work out why before I say it.
Because weekly data has no day-of-week variation left in it. Aggregating to a week averages the weekend and the weekdays together and destroys the largest single signal in the series. It was never that the four-week lag mattered more; it was that the thing that mattered most had already been summed away.
That is worth more than it looks. Feature importance is a statement about the table you built, not about the business. Change the frequency and the ranking inverts.
4.4 Step 2 — Watch the bias–variance tradeoff happen
Lecture 4 drew this as a curve. Now produce it.
rows = []
for depth in [1, 2, 3, 5, 8, 12, None]:
t = DecisionTreeRegressor(max_depth=depth, random_state=RANDOM_STATE).fit(X_train, y_train)
rows.append({"depth": str(depth),
"leaves": t.get_n_leaves(),
"train_RMSE": round(rmse(y_train, t.predict(X_train))),
"test_RMSE": round(rmse(y_test, t.predict(X_test)))})
sweep = pd.DataFrame(rows)
print(sweep.to_string(index=False))fig, ax = plt.subplots(figsize=(7, 3.5))
ax.plot(range(len(sweep)), sweep.train_RMSE, marker="o", label="train")
ax.plot(range(len(sweep)), sweep.test_RMSE, marker="o", label="test")
ax.set_xticks(range(len(sweep))); ax.set_xticklabels(sweep.depth)
ax.set_xlabel("max_depth"); ax.set_ylabel("RMSE"); ax.legend()
plt.tight_layout()Two things to say out loud. What does the training error do as depth increases, and what number does it reach at the end? And where does the test error bottom out?
The gap between those two curves is overfitting, drawn to scale.
Walkthrough
Lecture 4 drew this as a curve on a slide. Here it is as an actual table on your own data.
depth leaves train_RMSE test_RMSE
1 2 443 413
2 4 388 369
3 8 354 337
5 32 284 303
8 179 184 313
12 698 84 385
None 1200 0 406Read the two columns separately.
Training error falls the whole way down and ends at zero. Not near zero — zero. Look at the leaf count on that row: 1,200 leaves for 1,211 training days. The tree has given almost every single day its own leaf and memorized the answer. A model that has memorized the training set has a training error of zero by construction, and that number tells you nothing whatsoever.
Test error does something else. It falls to 303 at depth 5, and then turns around and climbs back to 406 — worse than the depth-2 tree with four leaves.
The gap between the two columns is overfitting, drawn to scale. At depth 5 it’s 284 against 303, almost nothing. At full depth it’s 0 against 406.
Two things to carry out of this table.
First, the model that fits the training data best is the worst model in the table on data it has not seen. That is not a subtlety, that is the whole lesson, and it will repeat in every lecture for the rest of the course.
Second — and this is the practical one — the depth-5 tree beats the fully grown tree, and the only thing separating them is one keyword argument. Nothing about the data changed. max_depth is the entire difference between 303 and 406.
4.5 Step 3 — Why one tree is not enough
Lecture 4 claimed a tree is unstable: change the data slightly and the structure changes. Test it properly, using bootstrap resamples — draw a sample of the same size with replacement, refit, and repeat.
rng = np.random.RandomState(RANDOM_STATE)
roots, preds = {}, []
for b in range(20):
idx = rng.choice(len(X_train), len(X_train), replace=True)
t = DecisionTreeRegressor(max_depth=5, random_state=RANDOM_STATE).fit(X_train[idx], y_train[idx])
root = [l.strip() for l in export_text(t, feature_names=FEATURES).split("\n")
if l.startswith("|---")][0]
roots[root] = roots.get(root, 0) + 1
preds.append(t.predict(X_test))
P = np.array(preds)
print("root splits across 20 bootstrap trees:")
for k, v in sorted(roots.items(), key=lambda kv: -kv[1]):
print(f" {v:2d} x {k}")
print(f"\nper-day spread of predictions across the 20 trees:")
print(f" average sd {P.std(0).mean():,.0f} units worst day {P.std(0).max():,.0f} units")The root may well be stable — one feature can be dominant enough that every resample finds it first. That is not the same as the tree being stable. Look at the spread of predictions: on a typical day the twenty trees disagree by a meaningful margin, and on the worst day they disagree enormously. All that disagreement lives below the root.
Now the payoff, and it is one line:
single = np.mean([rmse(y_test, p) for p in P])
averaged = rmse(y_test, P.mean(0))
print(f"average RMSE of a single bootstrap tree : {single:,.0f}")
print(f"RMSE of the 20 trees averaged together : {averaged:,.0f}")
print(f"improvement: {100*(single-averaged)/single:.0f}%")You just built a bagged ensemble by hand — bootstrap aggregating, in three lines. Nothing was tuned and no new information was added; the gain came entirely from averaging away the disagreement you measured above. It is not yet a Random Forest: a forest adds one more idea, which is to stop every tree from looking at dow first. Twenty trees that agree perfectly on their root are twenty trees making correlated mistakes. That is next week.
Walkthrough
Lecture 4 claimed a tree is unstable — nudge the data and the structure changes. Let’s test that properly rather than take it on faith.
Twenty bootstrap resamples: draw 1,211 rows with replacement, refit a depth-5 tree, record what it split on first, and keep its test predictions.
root splits across 20 bootstrap trees:
20 x |--- dow <= 4.50
per-day spread of predictions across the 20 trees:
average sd 160 units worst day 1,137 unitsNow, that first result looks like it refutes the lecture. Twenty out of twenty found the same root. The root is perfectly stable.
It is stable because day-of-week is overwhelming — a 43% gap survives any resample you can draw. A stable root is not a stable tree. Look at the second number: on a typical test day the twenty trees disagree by 160 units, and on the worst day they are spread over 1,137 units — about a third of an average day’s sales. Every bit of that disagreement lives below the root, in the splits you cannot see from this printout.
That is the instability the lecture was talking about, and you can only see it by looking at predictions, not at structure.
average RMSE of a single bootstrap tree : 324
RMSE of the 20 trees averaged together : 266
improvement: 18%Eighteen percent. No tuning, no new features, no extra data — the twenty trees saw less data than the original fit, since a bootstrap sample leaves out about a third of the rows each time.
And note where 266 sits against Step 2’s table: the best single tree you found by sweeping depth was 303. Averaging twenty individually worse trees — 324 on average — produced something better than the best one of them and better than the best tree in the whole sweep.
That is the result the next lecture is built on, and it is worth being precise about the mechanism. Nothing here reduced any individual tree’s bias. What fell was the variance you measured two cells ago: the disagreement averages out, the signal does not.
What you have built is called bagging — bootstrap aggregating. It is not yet a random forest. A forest adds one more idea, which is to stop the trees from all looking at day-of-week first, and you can already see why that might matter: twenty trees that agree perfectly on their root are twenty trees making correlated mistakes. That is Lecture 5, and it is one keyword argument.
4.6 Step 4 — Where trees reliably fail
A tree predicts the mean of the training rows in a leaf. So every prediction it can ever make is an average of values it has already seen.
full = DecisionTreeRegressor(random_state=RANDOM_STATE).fit(X_train, y_train)
p = full.predict(X_test)
print(f"training target range : [{y_train.min():,.0f}, {y_train.max():,.0f}]")
print(f"prediction range : [{p.min():,.0f}, {p.max():,.0f}]")The predictions cannot leave the training range — not “usually do not,” cannot. Now watch what that does at the extremes.
hi = y_test > np.percentile(y_test, 90)
lo = y_test < np.percentile(y_test, 10)
print(f"busiest 10% of test days (n={hi.sum()}): actual {y_test[hi].mean():,.0f}"
f" predicted {p[hi].mean():,.0f} under by {y_test[hi].mean()-p[hi].mean():,.0f}")
print(f"quietest 10% of test days (n={lo.sum()}): actual {y_test[lo].mean():,.0f}"
f" predicted {p[lo].mean():,.0f} over by {p[lo].mean()-y_test[lo].mean():,.0f}")Your turn: the two errors above point in opposite directions. Plot predicted against actual for the test set, add the 45-degree line, and see the shape this produces.
# ---- your turn: uncomment and fill in the two blanks ----
# fig, ax = plt.subplots(figsize=(4.5, 4.5))
# ax.scatter(___, ___, s=6, alpha=0.4) # <- actual on x, predicted on y
# lims = [y_test.min(), y_test.max()]
# ax.plot(lims, lims, "k--", lw=1)
# ax.set_xlabel("actual units"); ax.set_ylabel("predicted units")
# plt.tight_layout()
Walkthrough
Every lecture in this course should end by telling you where the method breaks. Here is where a tree breaks, and it follows from one sentence: a tree predicts the mean of the training rows in a leaf. Every prediction it will ever make is an average of values it has already seen.
training target range : [0, 5,016]
prediction range : [0, 4,527]The prediction range sits inside the training range, and it has to — not “usually does,” cannot. A tree can never forecast a value above the largest one in its training data, because there is no arithmetic in it that could produce one. No trend term, no extrapolation, nothing. If this store grows 20% next year, the tree will predict last year’s ceiling forever.
busiest 10% of test days (n=37): actual 4,115 predicted 3,929 under by 186
quietest 10% of test days (n=37): actual 2,030 predicted 2,237 over by 207Both ends are wrong, and — this is the point — they are wrong in opposite directions. The busy days are under-forecast by 186 units. The quiet days are over-forecast by 207. The model is pulling everything toward the middle.
Be precise about what this is and is not. The hard ceiling did not bite this year: the busiest test day was 4,717 units and the training set had a 5,016-unit day, so nothing was clipped. What you are seeing is the softer, more common version — leaf averaging shrinks the extremes inward. It is not a large effect in absolute terms next to a full tree’s RMSE of 406. It is a systematic one, which is worse, because it will not average out over the year and it is exactly the days you care most about.
Try it
Uncomment the scatter block and fill in the two blanks — actual on x, predicted on y, with the 45-degree line. Pause the video, run it, and come back when you can describe the shape.
You should be looking at a cloud that is flatter than the 45-degree line. Below the line on the right, above it on the left, hinged in the middle. Once you have seen that shape you will recognize it everywhere, in every method that predicts by averaging.
4.7 Before you leave
You should have found that the tree systematically under-predicts the busiest days and over-predicts the quietest ones, and that averaging twenty unstable trees beat any one of them.
Discuss (the failure): A tree pulls its predictions toward the middle at both ends. Give the mechanical reason — think about what a leaf prediction actually is.
Then the operational version: this store is planning inventory. Which of those two errors costs more, and does your answer change if the product is milk rather than canned goods?
Discuss (the comparison): You forecast this exact series with a GAM last week. Which method would you rather hand to a store manager, and what specifically would each one let you say that the other would not?
Solutions posted to Canvas after class.
Walkthrough
Try it
Two questions on the discussion board before the next session.
First, the failure. You have just seen a tree pull its predictions toward the middle at both ends. Give me the mechanical reason — and I mean mechanical, in terms of what a leaf prediction literally is, not “the model is conservative.”
Then the operational half. This store is planning inventory. One of those two errors costs more than the other: under-forecasting a busy day means an empty shelf and a lost sale; over-forecasting a quiet day means stock sitting in the back. Which is worse — and does your answer change if the product is milk rather than canned goods?
Second, the comparison. You fitted a GAM to this exact series in Homework 2, and a tree to it today. Which one would you actually hand to a store manager, and what specifically would each let you say that the other would not?
Be concrete. The GAM gives you a smooth curve per feature and a partial-dependence plot you can show someone. The tree gives you a path — weekday, four weeks ago was busy, yesterday was busy, predict 2,950 — that a manager can follow and argue with. Those are genuinely different kinds of explanation, and one of them will suit an audience better than the other.
One last thing before you close the file. Look back at Step 2’s table and notice what is not in this lab: we chose depth 5 by looking at the test set. Every number in that column was computed on the year we held out, and then we picked the winner from it. Lecture 1 told you what that is worth. Fixing it properly needs walk-forward validation on the training data alone, and that is Homework 3’s job.
Solutions for the coded parts go up on Canvas after the deadline. Next week: what happens when you stop the twenty trees from all looking at day-of-week first.
4.8 What your run should produce
Your numbers will differ where a model is stochastic; these are the values this lab produces on the pinned environment.
Deterministic; verified across repeated runs. random_state=42 on every fit; the bootstrap uses a seeded RandomState.
| Quantity | Value |
|---|---|
| Data | 1,576 usable days (train 1,211, test 365), 2015-05-24 → 2016-05-22 |
| Benchmark — same day last year | RMSE 625 |
| Depth-3 tree | test RMSE 337 · root split dow <= 4 |
| Depth-3 importances | dow 0.789 · units_lag28 0.106 · units_lag1 0.064 · units_lag7 0.030 · is_event 0.011 |
| Lecture 4 slide (weekly, same series) | units_lag4 0.62 dominant; dow absent |
| Weekday vs weekend mean (train) | 2,632 vs 3,773 — a 43% gap; dow alone explains 66% of training variance |
| Depth sweep — train / test RMSE | 1: 443/413 · 2: 388/369 · 3: 354/337 · 5: 284/303 · 8: 184/313 · 12: 84/385 · None: 0/406 |
| Leaves at full depth | 1,200 (for 1,211 training rows) |
| Bootstrap root splits | 20 / 20 identical (dow <= 4.50) |
| Prediction spread across 20 trees | average sd 160 units · worst day 1,137 |
| Single bootstrap tree (mean) | RMSE 324 |
| Twenty trees averaged | RMSE 266 — an 18% improvement |
| Full tree — training target range | [0, 5,016] |
| Full tree — prediction range | [0, 4,527] |
| Busiest 10% (n=37) | actual 4,115 · predicted 3,929 · under by 186 |
| Quietest 10% (n=37) | actual 2,030 · predicted 2,237 · over by 207 |