1  Lab 1: Benchmarks and Exponential Smoothing

Run this one live: the executable lab.

1.1 What we are doing

Lecture 1 made a claim that sounded like housekeeping: always beat a benchmark first. In the next 35 minutes you are going to test that claim on a real Walmart series, and then watch it fail on a different one.

The business question: you are forecasting weekly HOUSEHOLD-goods demand at store TX_1 to set inventory a year out. Which method should you actually ship?

Suggested time: ~35 minutes · You will need: pandas, numpy, matplotlib, statsmodels, and data/processed/m5_weekly.csv (run python scripts/prep_m5.py if it is missing).

1.2 Setup

Nothing to fill in here — run it and move on.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from statsmodels.tsa.holtwinters import ExponentialSmoothing, SimpleExpSmoothing, Holt
import warnings
warnings.filterwarnings('ignore')

from pathlib import Path

def data_path(name):
    """Find the processed data file whether you render from the repo root or from
    this file's own folder."""
    for base in (Path.cwd(), *Path.cwd().parents):
        p = base / "data" / "processed" / name
        if p.exists():
            return p
    raise FileNotFoundError(
        f"Could not find data/processed/{name}. "
        "Run `python scripts/prep_m5.py` from the repository root first."
    )

weekly = pd.read_csv(data_path("m5_weekly.csv"), parse_dates=["week_start"])

def get_series(store, cat):
    return (weekly[(weekly.store_id == store) & (weekly.cat_id == cat)]
            .set_index("week_start")["units"].sort_index().astype(float))

y = get_series("TX_1", "HOUSEHOLD")

N_TEST = 52
train, test = y.iloc[:-N_TEST], y.iloc[-N_TEST:]
print(f"{len(y)} weeks total — train {len(train)}, test {len(test)}")

1.3 Step 1 — Look at it first

Never fit anything before you have seen the series.

fig, ax = plt.subplots(figsize=(11, 3.5))
ax.plot(train.index, train.values, label="train", lw=1.4)
ax.plot(test.index, test.values, label="test (held out)", lw=1.4)
ax.set_ylabel("units / week"); ax.set_title("TX_1 HOUSEHOLD")
ax.legend(); plt.tight_layout()

Notice: the series roughly doubles over five years, and the level is still moving at the end. Hold that thought — it is the whole lab.

Never fit anything before you’ve looked at the series. This takes ten seconds and it will save you an hour.

Two things. First, the level roughly doubles across five years — this store’s household aisle is growing, and growing substantially. Second, and this is the one to hold onto: the level is still moving at the end of the training window. It hasn’t flattened out. Whatever we fit has to say something about where the level goes next.

That single observation is the whole lab. Everything that happens in the next twenty minutes is a consequence of the level moving.

1.4 Step 2 — Establish the benchmark

Before any model, the thing to beat. Seasonal naive says: this week next year looks like this week last year.

def rmse(actual, pred):
    return float(np.sqrt(np.mean((np.asarray(actual) - np.asarray(pred)) ** 2)))

def mae(actual, pred):
    return float(np.mean(np.abs(np.asarray(actual) - np.asarray(pred))))

seasonal_naive = y.shift(52).iloc[-N_TEST:].values

results = {"Seasonal naive": (rmse(test, seasonal_naive), mae(test, seasonal_naive))}
print(f"Seasonal naive  RMSE {results['Seasonal naive'][0]:,.0f}")

Your turn: before running anything else, write down a number — what RMSE do you think a proper exponential smoothing model will get? You will check it in Step 4.

Before any model, the thing to beat.

Seasonal naive says: this week next year looks like this week last year. That’s it. One line — y.shift(52). No parameters, no fitting, no estimation.

Seasonal naive  RMSE 973

Nine hundred and seventy-three. Remember that number, because everything from here is measured against it. If a model can’t beat 973, it does not deserve to be in production no matter how sophisticated it is.

Try it

Pause here. Before you run anything else, write down a number — an actual number, on paper or in a comment. What RMSE do you think a proper exponential smoothing model will get on this series?

I’m asking you to commit because in Step 4 you’re going to check it, and the gap between what you expected and what happened is worth more to you than the result itself. Guessing privately and then quietly adjusting is not the same exercise. Write it down. Then come back.

1.5 Step 3 — Fit the three models

All three calls come straight from Lecture 1. Run this, then predict the ordering before you look at Step 4 — write your guess down.

# SES — level only.
ses_fit = SimpleExpSmoothing(train).fit()
results["SES"] = (rmse(test, ses_fit.forecast(N_TEST)),
                  mae(test, ses_fit.forecast(N_TEST)))

# Holt — level + trend. Undamped for now; we come back to damping in Step 5.
holt_fit = Holt(train).fit()
results["Holt"] = (rmse(test, holt_fit.forecast(N_TEST)),
                   mae(test, holt_fit.forecast(N_TEST)))

# Holt-Winters — level + trend + seasonality, additive, period 52.
hw_fit = ExponentialSmoothing(train, trend="add",
                              seasonal="add", seasonal_periods=52).fit()
results["Holt-Winters"] = (rmse(test, hw_fit.forecast(N_TEST)),
                           mae(test, hw_fit.forecast(N_TEST)))
One constraint to respect

Holt-Winters must estimate a seasonal index for each of the 52 weeks, and it needs two full cycles — 104 observations — to initialize. You have 225 training weeks, so you are fine here. You will not always be.

(Screen: run the three-model cell. It takes a few seconds — Holt-Winters is estimating 52 seasonal indices.)

Three models, three lines, all straight from Lecture 1. SES — level only. No trend, no seasonality. The simplest thing that could work.

Holt — level plus trend. Undamped for now; we come back to that in Step 5, and the fact that I’m flagging it should tell you something. Holt-Winters — level, trend, and seasonality, additive, period 52.

One constraint worth respecting. Holt-Winters estimates a seasonal index for every one of the 52 weeks, and it needs two full cycles — 104 observations — just to initialize. You have 225 training weeks, so you’re fine. You will not always be fine. When you get to a series with three years of weekly data and you’re wondering why Holt-Winters won’t fit, this is why.

Try it

Before you scroll to Step 4: predict the ordering. Which of these three wins, which comes last? Write it down next to your number from Step 2.

Most people rank them by sophistication — Holt-Winters first, because it models the most. See whether you did.

1.6 Step 4 — Compare honestly

tbl = (pd.DataFrame(results, index=["RMSE", "MAE"]).T
         .sort_values("RMSE").round(0).astype(int))
print(tbl.to_string())

Stop before reading on: did any model beat the benchmark? By how much? And was it the model you expected in Step 2?

In the room: say your answer to the person next to you, and make them say theirs.
On your own: score yourself against what you already wrote down. Scroll back to the RMSE number from Step 2 and the ordering from Step 3, and write two lines in a comment:

# guess ____, actual ____, off by ____
# ordering: right / wrong — first model I misplaced was ____

Grading your own prediction in writing is the part that matters. It is what makes the next result inform you rather than just agree with you.

                RMSE   MAE
SES              427   352
Holt-Winters     885   814
Seasonal naive   973   848
Holt            1166  1029

Read that from the bottom.

Holt is last. Eleven sixty-six. Worse than the benchmark — worse than shifting the series back a year and calling it a day. Holt is the model that adds a trend to SES, and this series has an obvious trend. It should have helped. It made things twenty percent worse than doing nothing.

Holt-Winters is third. Eight eighty-five. It beats the benchmark, but only just, and it loses badly to a model with no trend and no seasonality in it at all. This is the most sophisticated thing on the table and it’s mid.

SES wins, and it isn’t close. Four twenty-seven against a benchmark of 973 — less than half the error. The simplest model on the list, the one that only tracks the level, beat everything.

Now compare that to what you wrote down. If you predicted Holt-Winters would win because it models the most structure, you are in good company and you have just learned the most useful thing in this lab: more structure is not more accuracy. Every parameter Holt-Winters estimates is a parameter that can be estimated badly, and 52 seasonal indices from 225 observations is a lot of estimating.

Try it

Pause and answer these in the discussion thread for Week 1:

  1. Did any model beat the benchmark, and by how much?
  2. Was the winner the model you predicted in Step 3?
  3. Holt is worse than doing nothing. Before you watch Step 5 — what’s your hypothesis for why?

Post before you continue. The next step gives away the answer to number three, and the guess is worth more than the answer.

1.7 Step 5 — Break it, then fix it

Look at where Holt’s forecast goes.

fig, ax = plt.subplots(figsize=(11, 3.5))
ax.plot(test.index, test.values, label="actual", lw=1.6)
ax.plot(test.index, ses_fit.forecast(N_TEST), label="SES", ls="--")
ax.plot(test.index, holt_fit.forecast(N_TEST), label="Holt (undamped)", ls="--")
ax.set_ylabel("units / week"); ax.legend(); plt.tight_layout()

Lecture 1 warned that undamped linear extrapolation gets unrealistic past about six periods. You are forecasting 52.

Your turn: refit Holt with a damped trend and add it to the table. One keyword argument.

# ---- your turn: uncomment and fill in the one keyword argument ----
# holt_damped = Holt(train, ___).fit()             # <- damped_trend=True
#
# results["Holt (damped)"] = (rmse(test, holt_damped.forecast(N_TEST)),
#                             mae(test, holt_damped.forecast(N_TEST)))
#
# print(pd.DataFrame(results, index=["RMSE", "MAE"]).T
#         .sort_values("RMSE").round(0).astype(int).to_string())
# print(f"estimated phi: {holt_damped.params['damping_trend']:.3f}")

Report the value of \(\phi\). What does it mean that it came out below 1?

Here’s what went wrong. Look at where Holt’s forecast goes.

That’s a straight line, and it does not stop. Holt estimated a positive trend from the training data — correctly, the series was growing — and then extrapolated it linearly for fifty-two consecutive weeks. By the end of the horizon it’s forecasting a level the store has never reached.

Lecture 1 warned that undamped linear extrapolation stops being credible after about six periods. You are forecasting fifty-two. The model isn’t wrong about the trend; it’s wrong about the trend continuing forever.

Meanwhile SES, the dashed flat line, just says “about here” — and about-here turns out to be a much better guess than a confident straight line pointed at the ceiling.

Try it

Your turn, and it’s genuinely one keyword argument. Uncomment the block and fill in the blank:

holt_damped = Holt(train, damped_trend=True).fit()

That’s the fix. Add it to the table, print the result, and print the estimated phi. Pause here and do it — it’s thirty seconds of typing and you’ll remember it because you typed it.

Holt (damped)    426   351
SES              427   352

Phi comes out at 0.800. Two things here, and the second one is the honest one.

First: damping rescued the model completely. Eleven sixty-six down to four twenty-six. Same model, same data, same trend estimate — one parameter that says “the trend decays as you forecast further out” moved it from worst on the table to best. Phi of 0.8 means each step ahead gets eighty percent of the previous step’s trend, so the forecast bends toward flat instead of running off. That is what phi below 1 means, and it’s the answer to the question in the lab.

Second — and don’t let this slide past — damped Holt beat SES by one unit of RMSE. Four twenty-six against four twenty-seven. That is not a win. On a different held-out year it could easily flip. If you report “the damped model is best” you’re reporting noise as a finding.

What actually happened is that damped Holt and SES are the same answer arrived at two ways. Damping the trend hard enough is close to not having a trend. The honest summary is: on this series, models that keep the forecast near the current level beat models that extrapolate, and the two ways of doing that tie.

1.8 Step 6 — Same store. Different aisle. Opposite answer.

Change one word — HOUSEHOLD becomes FOODS. Same store, same code, same horizon.

y2 = get_series("TX_1", "FOODS")
train2, test2 = y2.iloc[:-N_TEST], y2.iloc[-N_TEST:]

sn2 = y2.shift(52).iloc[-N_TEST:].values
ses2 = SimpleExpSmoothing(train2).fit()

print(f"TX_1 FOODS   seasonal naive RMSE {rmse(test2, sn2):,.0f}")
print(f"TX_1 FOODS   SES            RMSE {rmse(test2, ses2.forecast(N_TEST)):,.0f}")

Notice: the ranking flips. On the household aisle an exponential smoother won comfortably. On the food aisle of the same store, the benchmark wins and SES is about 13% worse.

The reason is visible in one number. For TX_1 HOUSEHOLD the test year sits about 15% above the year before it, so seasonal naive carries a stale level forward and loses badly. For TX_1 FOODS the level barely moves — about 1.5% down — and last year is an excellent guess.

So it is not the store, and it is not the method. It is whether the level moved.

Change one word. HOUSEHOLD becomes FOODS. Same store, same code, same horizon, same fifty-two weeks held out.

TX_1 FOODS   seasonal naive RMSE 1308
TX_1 FOODS   SES            RMSE 1474

The ranking flips.

On the household aisle, SES beat the benchmark by more than half. On the food aisle of the same store, the benchmark wins and SES is about thirteen percent worse. Nothing changed except which shelf we’re looking at.

So what’s different? It isn’t the store. It isn’t the method — the code is character-for-character identical apart from one string.

It’s visible in one number. For household, the test year sits about fifteen percent above the year before it. The level moved, so seasonal naive carried a stale level forward and lost badly — and a method that tracks the level won.

For foods, the level barely moved — about one and a half percent down. When the level doesn’t move, last year is an excellent guess, and there’s nothing for a level-tracking model to add. It is not the store, and it is not the method. It is whether the level moved.

That’s the sentence to leave with. And notice what it does to the claim we opened with — “always beat a benchmark first” isn’t process hygiene. It’s the only thing that told us the answer was different on these two series. If you’d fit SES on household, seen it win, and rolled it out across the store, you’d have shipped a model that’s thirteen percent worse than doing nothing on the food aisle, and nothing in your workflow would have told you.

1.9 Before you leave

Discuss: You now have two series where the best method is different. You are responsible for all 30 series in this dataset and you have to pick one approach to run in production every month.

What do you actually do — and what would you need to measure to decide? “Fit everything and pick the winner per series” is an answer, but say what it costs you.

Solutions posted to Canvas after class.

Try it

Here’s the question, and it’s the one that shows up in your job rather than in your homework.

You now have two series where the best method is different. You are responsible for all thirty series in this dataset, and you have to pick one approach to run in production every month. What do you actually do — and what would you need to measure to decide?

“Fit everything and pick the winner per series” is a legitimate answer. But say what it costs you: thirty model selections a month, each one a chance to overfit the held-out year; a pipeline that can silently switch methods between runs; and no single story to tell the person who asks why the forecast changed.

Post your answer in the Week 1 thread before the next lecture. Two or three sentences. There isn’t a key for this one — I want to see how you reason about it, and we’ll pick it up in Lecture 2 when the methods get more complicated and the temptation gets worse. Solutions for the coded parts go up on Canvas after the deadline. See you in Lecture 2.

1.10 What your run should produce

Your numbers will differ where a model is stochastic; these are the values this lab produces on the pinned environment.

Everything the lab prints, for checking against your screen. These are deterministic; if yours differ, the data changed.

Quantity Value
Series length 277 weeks (train 225, test 52)
Seasonal naive RMSE 973
SES RMSE / MAE 427 / 352
Holt (undamped) RMSE / MAE 1166 / 1029
Holt-Winters RMSE / MAE 885 / 814
Holt (damped) RMSE / MAE 426 / 351
Estimated phi 0.800
SES estimated alpha 0.621
TX_1 FOODS seasonal naive RMSE 1308
TX_1 FOODS SES RMSE 1474 (12.7% worse)
HOUSEHOLD test year vs prior year +14.5%
FOODS test year vs prior year −1.5%

Note for the instructor. Holt-Winters emits a ConvergenceWarning on this series — the optimizer does not fully converge while estimating 52 seasonal indices. The lab suppresses warnings globally in Setup, so students will not see it. It does not change the conclusion (Holt-Winters loses to SES either way), but it is a second, quieter piece of evidence for the same point: that model is being asked to estimate more than this sample supports.