import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import warnings
warnings.filterwarnings('ignore')
from pathlib import Path
def data_path(name):
for base in (Path.cwd(), *Path.cwd().parents):
p = base / "data" / "processed" / name
if p.exists():
return p
raise FileNotFoundError(
f"Could not find data/processed/{name}. "
"Run `python scripts/prep_m5.py` from the repository root first."
)
weekly = (pd.read_csv(data_path("m5_weekly.csv"), parse_dates=["week_start"])
.sort_values(["store_id", "cat_id", "week_start"]))
CHANNELS = ["units", "snap_days", "event_days", "avg_price"]
WINDOW = 26
CUTOFF = np.datetime64("2015-05-17")
rmse = lambda a, b: float(np.sqrt(np.mean((np.asarray(a) - np.asarray(b)) ** 2)))
print(f"{weekly.groupby(['store_id','cat_id']).ngroups} series, {len(weekly):,} rows")7 Lab 7: Your First Network — and Why One Run Proves Nothing
Run this one live: the executable lab.
7.1 What we are doing
Two things today, and the second matters more than the first.
First, you will build the machinery Homework 5 needs: turning a panel of 30 series into windowed training examples, and training a feedforward network on them. That scaffolding is the part of Homework 5 most likely to eat an evening, so we do it together.
Second, you will train the same model several times and discover that it gives you a different answer every time. Then you will find out how large that variation is, and what it means for every neural network result you will ever read — including your own.
Suggested time: ~40 minutes · You will need: pandas, numpy, matplotlib, torch, and data/processed/m5_weekly.csv.
7.2 Setup
7.3 Step 1 — From a panel to windows
A network cannot read a dataframe. It needs a stack of fixed-size examples.
For each series, slide a 26-week window along the history: the window is the input, the units in the following week is the target.
X, Y, D = [], [], []
for _, g in weekly.groupby(["store_id", "cat_id"]):
arr = g[CHANNELS].values.astype("float32")
dates = g.week_start.values
for i in range(len(g) - WINDOW):
X.append(arr[i:i+WINDOW]) # (26, 4)
Y.append(arr[i+WINDOW, 0]) # next week's units
D.append(dates[i+WINDOW]) # the target's date decides train/test
X = np.stack(X); Y = np.array(Y, dtype="float32"); D = np.array(D)
train_mask, test_mask = D < CUTOFF, D >= CUTOFF
print(f"X shape {X.shape} (windows, weeks, channels)")
print(f"train {train_mask.sum():,} test {test_mask.sum():,}")Notice the window count is far larger than the row count. Every week is the end of one window and the interior of twenty-five others, so 8,310 rows become ~7,500 overlapping examples. That overlap is why the effective sample size is much smaller than it looks.
Now standardize — using training statistics only.
mu = X[train_mask].mean(axis=(0, 1), keepdims=True)
sd = X[train_mask].std(axis=(0, 1), keepdims=True) + 1e-8
y_mu, y_sd = Y[train_mask].mean(), Y[train_mask].std()
Xn, Yn = (X - mu) / sd, (Y - y_mu) / y_sd
# flatten each window for a feedforward network: (26, 4) -> 104 features
X_tr = torch.tensor(Xn[train_mask].reshape(train_mask.sum(), -1))
y_tr = torch.tensor(Yn[train_mask])
X_te = torch.tensor(Xn[test_mask].reshape(test_mask.sum(), -1))
print(f"flattened to {X_tr.shape[1]} features per example")This block is what Homework 5 Part 1 Q1 asks for. Keep it.
Walkthrough
30 series, 8,310 rowsX shape (7530, 26, 4) (windows, weeks, channels)
train 5,940 test 1,590Here is the reframing that every method from here to the end of the course depends on.
A network cannot read a dataframe. It needs a stack of identically-shaped examples. So for each series we slide a 26-week window along the history: those 26 weeks are the input, the units in week 27 are the target, and the target’s date decides whether the example is training or test.
Four channels — units, SNAP days, event days, average price. Twenty-six weeks. So each example is a 26-by-4 block.
Now notice the count, because it is a trap dressed as good news. 8,310 rows became 7,530 examples. It looks like you have seven and a half thousand independent observations. You do not. Every week appears in twenty-six different windows — once as a target, twenty-five times as interior context. The examples overlap almost completely with their neighbors.
So the effective sample size is far smaller than 7,530, and that matters for how much you should trust anything this network tells you. Half of Homework 5’s difficulty is remembering this.
flattened to 104 features per exampleTwo details, both of which are graded in Homework 5.
The statistics come from training data only. mu and sd are computed under train_mask and then applied to everything. Standardizing with statistics computed over the full dataset leaks test-period information into training, and it will not raise an error — it will just quietly make your results better than they are.
And then we flatten. Twenty-six weeks by four channels becomes 104 numbers in a row. A feedforward network has no notion of sequence; it sees 104 unordered features. Hold that thought — it is the answer to one of the closing questions, and it is what Lecture 8 exists to fix. This block is Homework 5 Part 1 Question 1. Keep it.
7.4 Step 2 — Train a network
def make_model(n_features, seed):
torch.manual_seed(seed)
return nn.Sequential(
nn.Linear(n_features, 128), nn.ReLU(), nn.Dropout(0.2),
nn.Linear(128, 64), nn.ReLU(), nn.Dropout(0.2),
nn.Linear(64, 1))
def train(model, epochs=60, lr=5e-4, weight_decay=0.0):
opt = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=weight_decay)
lossfn = nn.MSELoss()
loader = torch.utils.data.DataLoader(
torch.utils.data.TensorDataset(X_tr, y_tr), batch_size=64, shuffle=False)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
lossfn(model(xb).squeeze(-1), yb).backward()
opt.step()
model.eval()
with torch.no_grad():
return rmse(Y[test_mask], model(X_te).squeeze(-1).numpy() * y_sd + y_mu)
model = make_model(X_tr.shape[1], seed=1)
print(f"parameters: {sum(p.numel() for p in model.parameters()):,}")
print(f"test RMSE: {train(model):,.0f}")Note shuffle=False. Almost every PyTorch tutorial sets it True. For a time series it destroys the ordering, and nothing warns you.
Compare that RMSE against your records. XGBoost managed 781 on this panel in Homework 4, and a 46-coefficient LASSO managed 744. Your network has around 22,000 parameters.
Walkthrough
parameters: 21,761
test RMSE: 1,109Three layers — 104 in, 128, 64, one out — with ReLU and dropout between them. Twenty-one thousand seven hundred and sixty-one parameters, trained on 5,940 overlapping examples.
One line deserves attention. shuffle=False in the DataLoader. Almost every PyTorch tutorial you will find sets it to True, because for images it is right. For a time series it destroys the ordering, and — this is the theme of the lab — nothing warns you. No error, no warning, just a number that is wrong in a way you cannot see.
Now the result. 1,109 on the test weeks, against a seasonal-naive benchmark of 2,152. So the network is roughly twice as good as the benchmark. On a panel where a typical week is about 9,300 units, that is an error of about 12%.
And now the comparison you should not skip. On this same panel, in Homework 4, XGBoost managed 781 and a 46-coefficient LASSO managed 744. Twenty-two thousand parameters lost to forty-six coefficients. Not narrowly — by about 45%.
I am not going to soften that, and you should not expect the rest of the lab to rescue it. It is the honest result on this dataset, and why is one of your discussion questions. But hold the number 1,109 loosely for about ninety seconds, because Step 3 is going to tell you how much that particular number is worth.
7.5 Step 3 — Run it again
Same code, same data, same settings. Only the random seed changes.
for seed in (1, 2, 3):
print(f"seed {seed}: test RMSE {train(make_model(X_tr.shape[1], seed)):,.0f}")Say out loud what just happened, because it is the point of the lab. Nothing about the problem changed. The differences you see come entirely from random weight initialization and dropout.
Walkthrough
Same code. Same data. Same settings. Same machine. The only thing that changes is the random seed — which controls the initial weights and which units dropout switches off.
seed 1: test RMSE 1,109
seed 2: test RMSE 1,176
seed 3: test RMSE 946
Try it
Pause the video and say what just happened, in one sentence, before I do. Nothing about the problem changed and the answer moved by 230 RMSE — from 946 to 1,176.
Sit with the size of that. 230 is not a rounding error. It is about 20% of the result. It is larger than most differences you will ever see quoted between two competing architectures in a blog post.
And notice which seed won. Seed 3 gives 946 — the best number anywhere in this lab. If I had run seed 3 first and stopped there, I would have reported 946 and believed it. If I had run seed 2 first, I would have reported 1,176 and believed that. The reported number would depend entirely on an integer nobody thinks of as part of the experiment. That is the whole lab. Everything from here is working out the consequences.
7.6 Step 4 — So how would you compare two settings?
Here is the trap. Suppose you want to know whether weight decay helps. You try three values, train once each, and pick the winner.
single = {wd: train(make_model(X_tr.shape[1], seed=1), weight_decay=wd)
for wd in (0.0, 1e-4, 1e-2)}
for wd, r in single.items():
print(f"weight_decay={wd:<7} test RMSE {r:,.0f}")Now do it honestly — five seeds per setting, and look at the spread as well as the mean.
rows = []
for wd in (0.0, 1e-4, 1e-2):
runs = [train(make_model(X_tr.shape[1], s), weight_decay=wd) for s in (1, 2, 3, 4, 5)]
rows.append({"weight_decay": f"{wd:g}", "mean": round(np.mean(runs)),
"sd": round(np.std(runs)), "min": round(min(runs)), "max": round(max(runs))})
print(pd.DataFrame(rows).to_string(index=False))
Compare the two tables before you go on
When this was run while writing the lab, the single-seed experiment made 1e-4 look like the winner — about 100 RMSE better than no weight decay at all. On five seeds the two means differed by 4.
That 100-point advantage was not a small effect measured imprecisely. It did not exist.
The five-seed table also shows something the single run cannot: 1e-2 is not just worse on average, its spread is three times wider. A setting that sometimes returns 1,050 and sometimes 1,792 is a different kind of bad from one that reliably returns 1,240 — and you would never learn that from one run.
Your turn: using your own two tables, state the smallest RMSE difference you would be willing to call real.
# ---- your turn: uncomment and fill in ----
# spread = ___ # <- a number from your five-seed table
# print(f"differences smaller than about {spread:,.0f} RMSE are not evidence")
Walkthrough
Here is the trap, and it is the single most common mistake in applied deep learning. You want to know whether weight decay helps. Three values, one training run each, pick the winner.
weight_decay=0.0 test RMSE 1,109
weight_decay=0.0001 test RMSE 1,007
weight_decay=0.01 test RMSE 1,142Clear result. 1e-4 wins by 102 RMSE over no weight decay at all. Nine percent. You would write that down, you would put it in a slide, and honestly most people would put it in a paper.
Now do it honestly. Five seeds per setting, and report the spread as well as the mean.
weight_decay mean sd min max
0 1084 86 946 1176
0.0001 1088 134 991 1343
0.01 1241 278 1050 1792The two means differ by four.
Not “the effect was smaller than we thought.” Not “the advantage was modest.” The 102-point advantage that the single-seed experiment found did not exist. Seed 1 happened to be a good draw for 1e-4 and an average draw for zero, and that is the entire finding.
Look closer and it is worse than that. On min and on max, no weight decay is better than 1e-4 — 946 against 991, 1,176 against 1,343. The setting with the worse spread has the marginally better mean, by four. There is no effect here in any direction.
Now the column the single run could never have produced: sd.
1e-2 is worse on average — 1,241 — but the important number is 278, more than three times the spread of no weight decay at all. It returned 1,050 once and 1,792 another time. That is a different kind of bad from a setting that reliably lands at 1,240. One is a model that is mediocre; the other is a model you cannot make promises about. You cannot see that in one run. Ever.
Try it
Fill in the blank and answer the question in your own words: what is the smallest RMSE difference you would be willing to call real on this problem?
There is no single right answer, but there is a defensible way to reason. The run-to-run standard deviation at your best setting is about 86. Two single runs differing by less than that are telling you nothing at all. If you want to claim a difference of that size is real, you need several seeds per setting and you need to compare the distributions, not the winners.
Which means: a difference under roughly 100 RMSE, from single runs, is not evidence. And you will notice that this disqualifies almost every architecture comparison you have ever read.
7.7 Step 5 — Break it on purpose
One line, no error message.
def train_broken(model, epochs=60, lr=5e-4):
opt = torch.optim.Adam(model.parameters(), lr=lr)
lossfn = nn.MSELoss()
loader = torch.utils.data.DataLoader(
torch.utils.data.TensorDataset(X_tr, y_tr), batch_size=64, shuffle=False)
for _ in range(epochs):
model.train()
for xb, yb in loader:
# opt.zero_grad() <-- deleted
lossfn(model(xb).squeeze(-1), yb).backward()
opt.step()
model.eval()
with torch.no_grad():
return rmse(Y[test_mask], model(X_te).squeeze(-1).numpy() * y_sd + y_mu)
print(f"with zero_grad(): {train(make_model(X_tr.shape[1], 1)):,.0f}")
print(f"without zero_grad(): {train_broken(make_model(X_tr.shape[1], 1)):,.0f}")Without zero_grad(), PyTorch accumulates gradients across batches instead of replacing them, so every update uses a sum of all gradients seen so far in that epoch.
No exception. No warning. Just a model that trains badly — and note how badly. The broken version lands at 7,325, which is more than three times worse than the seasonal-naive benchmark of 2,152. A one-line omission turned a working model into something far worse than predicting last year’s value.
Walkthrough
One last thing, and it is the same theme in a different costume.
Look at train_broken. One line is commented out — opt.zero_grad(). Everything else is identical.
with zero_grad(): 1,109
without zero_grad(): 7,325PyTorch accumulates gradients by default. It does not replace them. If you never zero them, every update is driven by the sum of every gradient seen so far in that epoch — so the steps get larger and larger and increasingly unrelated to the batch in front of you.
The result is 7,325. The seasonal-naive benchmark is 2,152. The broken model is three and a half times worse than predicting last year’s value.
And the point is not the number, it is everything that did not happen. No exception. No warning. No NaN. The loop ran to completion in the normal amount of time and returned a float. If you had not had a benchmark to compare against, you would have had no way to know.
That is why Lecture 1 spent a whole session on benchmarks, and it is why every scoreboard in this course starts with seasonal naive. A benchmark is not there to flatter your model. It is the only thing standing between you and a silent bug.
7.8 Before you leave
Discuss (the seeds): You now know a single training run tells you very little. But papers, blog posts and vendor benchmarks routinely report exactly one. When you read “our model achieves RMSE 1,050,” what should you want to know before believing it beats a baseline at 1,100?
Discuss (the comparison): Your network has roughly 22,000 parameters and lost to a 46-coefficient LASSO from Homework 4. Give one reason that is not simply “neural networks need more data” — think about what the FFN can and cannot see in a flattened window.
Discuss (the practice): You will hit this again in Homework 5, where you compare five architectures. Given today, what will you do differently when one of them comes out 80 RMSE ahead of another?
Solutions posted to Canvas after class.
Walkthrough
Try it
Three questions for the board.
The seeds. You now know a single run tells you very little. But papers, blog posts, and vendor benchmarks routinely report exactly one. When you read “our model achieves RMSE 1,050”, what do you need to know before believing it beats a baseline at 1,100? Be specific about what you would ask for — and notice that today, a 50-point gap is well inside one standard deviation.
The comparison. Twenty-two thousand parameters lost to a forty-six-coefficient LASSO, 1,084 against 744. Give one reason that is not “neural networks need more data.” Think about what your network can actually see: 104 unordered numbers. It does not know that feature 1 and feature 5 are the same channel a week apart, or that the channels are four different quantities. It has to learn the layout of its own input from 5,940 heavily overlapping examples — and the LASSO was handed that structure by whoever engineered the features.
The practice. In Homework 5 you compare five architectures. Given today, what will you actually do when one comes out 80 RMSE ahead of another?
I’ll tell you what the honest answer is, because it is also the assignment’s answer: you say the difference is inside the noise, and you say so even when the architecture you like is ahead. Homework 5 Part 3 asks you to rank five architectures. Some of those gaps will be real and some will be seeds. Knowing which is which is the skill being tested.
Solutions for the coded parts go up on Canvas after the deadline. Next week: a network that knows its 104 inputs came from a sequence.
7.9 What your run should produce
Your numbers will differ where a model is stochastic; these are the values this lab produces on the pinned environment.
Deterministic — torch.manual_seed per model, shuffle=False in the loader. Seed 1 returns 1,109 on every run. Whole lab ≈ 95 seconds, of which Step 4’s five-seed table is ~60.
| Quantity | Value |
|---|---|
| Panel | 30 series, 8,310 rows → 7,530 windows (train 5,940, test 1,590) |
| Window | 26 weeks × 4 channels → flattened to 104 features |
| Model | 3 layers (128, 64, 1), dropout 0.2 — 21,761 parameters |
| Seed 1 / 2 / 3 | 1,109 · 1,176 · 946 (range 230) |
| Single-seed weight decay | 0.0: 1,109 · 1e-4: 1,007 · 1e-2: 1,142 → apparent 102-point win |
| Five seeds, wd = 0 | mean 1,084 · sd 86 · min 946 · max 1,176 |
| Five seeds, wd = 1e-4 | mean 1,088 · sd 134 · min 991 · max 1,343 |
| Five seeds, wd = 1e-2 | mean 1,241 · sd 278 · min 1,050 · max 1,792 |
| Difference in means, 0 vs 1e-4 | 4 |
Without zero_grad() |
7,325 (vs 1,109) |
| Seasonal naive on these test weeks | 2,152 — recomputed and verified |
| Homework 4 on this panel | LASSO (46 features) 744 · XGBoost 781 |
| Mean weekly units, test period | 9,301 |