Lab 4: Decision Trees — Reading One, and Finding Where It Breaks
ECON 8310: Business Forecasting — University of Nebraska at Omaha
What we are doing
Today the course changes character. Everything so far specified a structure and estimated it. A tree is handed a pile of features and finds the structure itself.
We use the same series you just forecast in Homework 2 — daily units for CA_1 FOODS. Same store, same aisle, same target. Last week you fitted a GAM to it. Today you fit a tree, and the comparison is the point: you will be able to say what each method bought you on identical data.
By the end you will have done four things: read a fitted tree out loud, watched the bias–variance tradeoff happen in a table, discovered why a single tree is unstable, and found the one place trees reliably fail.
Suggested time: ~40 minutes · You will need:pandas, numpy, matplotlib, scikit-learn, and data/processed/m5_daily.csv (run python scripts/prep_m5.py if it is missing).
Setup
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltfrom sklearn.tree import DecisionTreeRegressor, export_text, plot_treeimport warningswarnings.filterwarnings('ignore')from pathlib import Pathdef data_path(name):"""Find the processed data file whether you render from the repo root or from this file's own folder."""for base in (Path.cwd(), *Path.cwd().parents): p = base /"data"/"processed"/ nameif p.exists():return praiseFileNotFoundError(f"Could not find data/processed/{name}. ""Run `python scripts/prep_m5.py` from the repository root first." )RANDOM_STATE =42daily = pd.read_csv(data_path("m5_daily.csv"), parse_dates=["date"])daily["dow"] = daily["date"].dt.dayofweek # 0 = Mondaydaily["doy"] = daily["date"].dt.dayofyearFEATURES = ["units_lag1", "units_lag7", "units_lag14", "units_lag28", "units_lag365","snap", "is_event", "dow", "doy", "avg_price"]df = daily.dropna(subset=FEATURES).copy()N_TEST =365train, test = df.iloc[:-N_TEST], df.iloc[-N_TEST:]X_train, y_train = train[FEATURES].values, train["units"].valuesX_test, y_test = test[FEATURES].values, test["units"].valuesrmse =lambda a, b: float(np.sqrt(np.mean((np.asarray(a) - np.asarray(b)) **2)))print(f"{len(df)} usable days — train {len(train)}, test {len(test)}")print(f"benchmark (same day last year): RMSE {rmse(y_test, test.units_lag365.values):,.0f}")
1576 usable days — train 1211, test 365
benchmark (same day last year): RMSE 625
Notice what just happened to the data. There is no date index in X_train. Every row is an independent example, and everything the model knows about time is sitting in the lag columns. That reframing — a time series becomes a flat table — is what lets every method from here to Lecture 9 be applied to forecasting at all.
Step 1 — Fit a tree and read it out loud
Start deliberately shallow, because a depth-3 tree is small enough to read.
Put the root split into business language — not dow <= 4.50 but what that means about this store.
In the room: read it aloud to the person next to you. Saying it to someone is what exposes a sentence you do not actually understand. On your own: write the sentence in a comment, then add a prediction you can check immediately:
# the root split separates ____ from ____
# I expect the higher-sales side to be dow <= 4.50 / dow > 4.50, because ____
Then read the two node values off the tree plot above and see whether you were right. A guess you wrote down and checked teaches you more than a split you merely read.
Lecture 4’s importance table was computed on weekly CA_1 FOODS, and there the four-week lag dominated. Here, on daily data for the same store and aisle, something else wins outright.
Same store. Same product. Different frequency, different answer. Say why before moving on.
Step 2 — Watch the bias–variance tradeoff happen
Lecture 4 drew this as a curve for the weekly series. Now measure it on your own daily data and draw the same curve.
Two things to say out loud. What does the training error do as depth increases, and what number does it reach at the end? And where does the test error bottom out?
The gap between those two curves is overfitting, drawn to scale.
Step 3 — Why one tree is not enough
Lecture 4 claimed a tree is unstable: change the data slightly and the structure changes. Test it properly, using bootstrap resamples — draw a sample of the same size with replacement, refit, and repeat.
rng = np.random.RandomState(RANDOM_STATE)roots, preds = {}, []for b inrange(20): idx = rng.choice(len(X_train), len(X_train), replace=True) t = DecisionTreeRegressor(max_depth=5, random_state=RANDOM_STATE).fit(X_train[idx], y_train[idx]) root = [l.strip() for l in export_text(t, feature_names=FEATURES).split("\n")if l.startswith("|---")][0] roots[root] = roots.get(root, 0) +1 preds.append(t.predict(X_test))P = np.array(preds)print("root splits across 20 bootstrap trees:")for k, v insorted(roots.items(), key=lambda kv: -kv[1]):print(f" {v:2d} x {k}")print(f"\nper-day spread of predictions across the 20 trees:")print(f" average sd {P.std(0).mean():,.0f} units worst day {P.std(0).max():,.0f} units")
root splits across 20 bootstrap trees:
20 x |--- dow <= 4.50
per-day spread of predictions across the 20 trees:
average sd 160 units worst day 1,137 units
The root may well be stable — one feature can be dominant enough that every resample finds it first. That is not the same as the tree being stable. Look at the spread of predictions: on a typical day the twenty trees disagree by a meaningful margin, and on the worst day they disagree enormously. All that disagreement lives below the root.
Now the payoff, and it is one line:
single = np.mean([rmse(y_test, p) for p in P])averaged = rmse(y_test, P.mean(0))print(f"average RMSE of a single bootstrap tree : {single:,.0f}")print(f"RMSE of the 20 trees averaged together : {averaged:,.0f}")print(f"improvement: {100*(single-averaged)/single:.0f}%")
average RMSE of a single bootstrap tree : 324
RMSE of the 20 trees averaged together : 266
improvement: 18%
You just built a bagged ensemble by hand — bootstrap aggregating, in three lines. Nothing was tuned and no new information was added; the gain came entirely from averaging away the disagreement you measured above. It is not yet a Random Forest: a forest adds one more idea, which is to stop every tree from looking at dow first. Twenty trees that agree perfectly on their root are twenty trees making correlated mistakes. That is next week.
Step 4 — Where trees reliably fail
A tree predicts the mean of the training rows in a leaf. So every prediction it can ever make is an average of values it has already seen.
full = DecisionTreeRegressor(random_state=RANDOM_STATE).fit(X_train, y_train)p = full.predict(X_test)print(f"training target range : [{y_train.min():,.0f}, {y_train.max():,.0f}]")print(f"prediction range : [{p.min():,.0f}, {p.max():,.0f}]")
training target range : [0, 5,016]
prediction range : [0, 4,527]
The predictions cannot leave the training range — not “usually do not,” cannot. That is the one place trees reliably fail: a series that grows past its history. This test year does not, so the ceiling never binds here. Keep that in mind before you blame the next result on it.
hi = y_test > np.percentile(y_test, 90)lo = y_test < np.percentile(y_test, 10)print(f"busiest 10% of test days (n={hi.sum()}): actual {y_test[hi].mean():,.0f}"f" predicted {p[hi].mean():,.0f} under by {y_test[hi].mean()-p[hi].mean():,.0f}")print(f"quietest 10% of test days (n={lo.sum()}): actual {y_test[lo].mean():,.0f}"f" predicted {p[lo].mean():,.0f} over by {p[lo].mean()-y_test[lo].mean():,.0f}")
busiest 10% of test days (n=37): actual 4,115 predicted 3,929 under by 186
quietest 10% of test days (n=37): actual 2,030 predicted 2,237 over by 207
The two errors point in opposite directions. Before you blame the tree, look at how we picked these days: by their actual sales. Picked that way, any imperfect forecaster errs in exactly this pattern. It is called regression to the mean, and a linear regression on the same features (day of week as dummies), which can extrapolate, pulls toward the middle even harder.
Your turn: plot predicted against actual for the test set, add the 45-degree line, and see the shape this produces.
# ---- your turn: uncomment and fill in the two blanks ----# fig, ax = plt.subplots(figsize=(4.5, 4.5))# ax.scatter(___, ___, s=6, alpha=0.4) # <- actual on x, predicted on y# lims = [y_test.min(), y_test.max()]# ax.plot(lims, lims, "k--", lw=1)# ax.set_xlabel("actual units"); ax.set_ylabel("predicted units")# plt.tight_layout()
Before you leave
You should have found that a tree can never predict outside its training range, that the pull toward the middle at both ends is regression to the mean rather than a tree failure, and that averaging twenty unstable trees beat any one of them.
Discuss (the pull to the middle): The busiest test days were under-forecast and the quietest over-forecast. Why does any imperfect forecaster show this when days are chosen by their actual sales? What would you expect if you chose the busiest days by the tree’s prediction instead?
Then the operational version: this store is planning inventory. Which of those two errors costs more, and does your answer change if the product is milk rather than canned goods?
Discuss (the comparison): You forecast this exact series with a GAM last week. Which method would you rather hand to a store manager, and what specifically would each one let you say that the other would not?
