import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import math
import torch
import torch.nn as nn
import warnings
warnings.filterwarnings('ignore')
from pathlib import Path
def data_path(name):
for base in (Path.cwd(), *Path.cwd().parents):
p = base / "data" / "processed" / name
if p.exists():
return p
raise FileNotFoundError(
f"Could not find data/processed/{name}. "
"Run `python scripts/prep_m5.py` from the repository root first.")
weekly = (pd.read_csv(data_path("m5_weekly.csv"), parse_dates=["week_start"])
.sort_values(["store_id", "cat_id", "week_start"]))
CHANNELS, WINDOW = ["units", "snap_days", "event_days", "avg_price"], 52
CUTOFF = np.datetime64("2015-05-17")
rmse = lambda a, b: float(np.sqrt(np.mean((np.asarray(a) - np.asarray(b)) ** 2)))
X, Y, D = [], [], []
for _, g in weekly.groupby(["store_id", "cat_id"]):
arr, dates = g[CHANNELS].values.astype("float32"), g.week_start.values
for i in range(len(g) - WINDOW):
X.append(arr[i:i+WINDOW]); Y.append(arr[i+WINDOW, 0]); D.append(dates[i+WINDOW])
X = np.stack(X); Y = np.array(Y, dtype="float32"); D = np.array(D)
tr, te = D < CUTOFF, D >= CUTOFF
mu, sd = X[tr].mean((0,1), keepdims=True), X[tr].std((0,1), keepdims=True) + 1e-8
ym, ys = Y[tr].mean(), Y[tr].std()
Xn, Yn = (X - mu)/sd, (Y - ym)/ys
X_tr, y_tr, X_te = torch.tensor(Xn[tr]), torch.tensor(Yn[tr]), torch.tensor(Xn[te])
print(f"{X.shape[0]:,} windows of {WINDOW} weeks — train {tr.sum():,}, test {te.sum():,}")10 Lab 9 Part 2: Attention — and the Line You Cannot Forget
Run this one live: the executable lab.
10.1 What we are doing
Last week’s recurrent models read the window one step at a time. Attention does not: every position looks at every other position at once, so week 3 reaches week 50 in a single operation rather than through forty-seven state updates.
Today you build a Transformer encoder, then delete one line from it and watch it fall apart. That line is the one PyTorch does not add for you, and forgetting it is the most common way this architecture fails silently.
Suggested time: ~40 minutes · You will need: pandas, numpy, matplotlib, torch, and data/processed/m5_weekly.csv.
This lab is slow
Attention is \(O(T^2)\) in sequence length, and we use a 52-week window. A single fit takes about 70 seconds and Step 3 trains four of them. Start the cell, read while it runs.
10.2 Setup
Walkthrough
Same panel, same cutoff, same standardization. One difference from the recurrent lab: the window is fixed at 52 weeks throughout, because a long window is where attention is supposed to earn its keep. If direct long-range paths are worth anything, a full year of history is where we should see it.
That choice is also why this lab is slow. Attention compares every position with every other position, so the cost grows with the square of the window. Fifty-two weeks means 2,704 pairwise comparisons per example per head per layer.
10.3 Step 1 — Positional encoding, before the model
Attention computes a weighted average over positions. Averaging does not care about order: shuffle the input and every attention output is unchanged.
So position has to be added to the input itself. The standard device is a set of sinusoids of different frequencies.
def sinusoidal(seq_len, d_model):
pe = torch.zeros(seq_len, d_model)
pos = torch.arange(seq_len).unsqueeze(1).float()
div = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
pe[:, 0::2] = torch.sin(pos * div)
pe[:, 1::2] = torch.cos(pos * div)
return pe.unsqueeze(0)
pe = sinusoidal(WINDOW, 64)
fig, ax = plt.subplots(figsize=(9, 3))
im = ax.imshow(pe[0].T, aspect="auto", cmap="RdBu")
ax.set_xlabel("week in window"); ax.set_ylabel("encoding dimension")
plt.colorbar(im); plt.tight_layout()Read the picture. Every column — every week — has a distinct pattern down the dimensions, so the model can tell positions apart. Nearby columns look similar, so it can also learn relative distance rather than only absolute index.
Walkthrough
Start with the problem, because the solution makes no sense without it.
Attention computes a weighted average over positions. And an average does not care about order — reverse the sequence, shuffle it, deal it like a deck of cards, and every attention output is identical. The mechanism is permutation-invariant by construction.
That is a catastrophe for forecasting. “Sales were 3,000 last week and 9,000 a year ago” and “sales were 9,000 last week and 3,000 a year ago” are the same bag of numbers and completely different futures.
So position has to be put into the input itself, before attention ever sees it. The standard device is a set of sinusoids at different frequencies.
Try it
Pause on the heat map and check two properties before I name them.
One: every column is distinct. Each week has its own signature down the 64 dimensions, so the model can tell position 3 from position 47.
Two, and this is the part people miss: nearby columns look similar. Week 20 and week 21 have almost the same pattern; week 20 and week 50 do not. That means the encoding carries relative distance, not just an arbitrary label per slot. A model can learn “about four weeks apart” as a concept rather than memorizing every pair of indices — which is what makes the scheme work at positions it never saw in training.
That is why sinusoids rather than, say, just numbering the weeks 1 to 52.
10.4 Step 2 — Build the encoder
class TransformerForecaster(nn.Module):
def __init__(self, n_channels=4, d_model=64, nhead=4, layers=2, use_pe=True):
super().__init__()
self.use_pe = use_pe
self.inp = nn.Linear(n_channels, d_model)
self.register_buffer("pe", sinusoidal(WINDOW, d_model))
enc_layer = nn.TransformerEncoderLayer(
d_model=d_model, nhead=nhead, dim_feedforward=128,
dropout=0.1, batch_first=True)
self.enc = nn.TransformerEncoder(enc_layer, num_layers=layers)
self.fc = nn.Linear(d_model, 1)
def forward(self, x):
h = self.inp(x)
if self.use_pe:
h = h + self.pe # <- the line in question
return self.fc(self.enc(h)[:, -1, :]).squeeze(-1)
def train(use_pe, seed, epochs=30):
torch.manual_seed(seed)
model = TransformerForecaster(use_pe=use_pe)
opt = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)
lossfn = nn.MSELoss()
loader = torch.utils.data.DataLoader(
torch.utils.data.TensorDataset(X_tr, y_tr), batch_size=128, shuffle=False)
for _ in range(epochs):
model.train()
for xb, yb in loader:
opt.zero_grad()
lossfn(model(xb), yb).backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step()
model.eval()
with torch.no_grad():
return rmse(Y[te], model(X_te).numpy() * ys + ym), sum(p.numel() for p in model.parameters())
score, n_par = train(use_pe=True, seed=1)
print(f"Transformer — {n_par:,} parameters, test RMSE {score:,.0f}")Note the parameter count against last week’s models: the RNN had 4,545 and won.
Walkthrough
The whole architecture is four objects: a linear layer lifting 4 channels to 64 dimensions, the positional encoding, two TransformerEncoderLayers, and a linear head reading the final position.
That line is the subject of the lab. One addition. Everything else in this class is supplied by PyTorch.
Transformer — 67,329 parameters, test RMSE 1,013Sixty-seven thousand parameters. Hold that against last week: the vanilla RNN had 4,545 and scored 805. So attention arrives with fifteen times the parameters of the model it needs to beat.
10.5 Step 3 — Delete the line
Same model, same data, same seeds — use_pe=False removes the single addition in forward.
rows = []
for use_pe in (True, False):
runs = [train(use_pe, s)[0] for s in (1, 2)]
rows.append({"positional encoding": "yes" if use_pe else "no",
"mean": round(np.mean(runs)), "sd": round(np.std(runs)),
"runs": [round(r) for r in runs]})
print(pd.DataFrame(rows).to_string(index=False))
Nothing raised an exception
The model without positional encoding trained happily, converged, and returned forecasts. It was simply working from an unordered bag of 52 weeks — it could not tell last week from a year ago.
Compare the gap against the run-to-run spread in the sd column. This is not noise.
nn.TransformerEncoderLayer gives you attention, residuals, layer norm and the feed-forward sublayer. It does not give you position. That is yours to add, and nothing checks that you did.
Your turn: express the penalty as a percentage of the with-encoding RMSE.
# ---- your turn: uncomment and fill in ----
# df = pd.DataFrame(rows)
# penalty = ___ - ___
# print(f"cost of the missing line: {penalty:,.0f} RMSE "
# f"({100*penalty/df['mean'].iloc[0]:.0f}% worse)")
Walkthrough
Same model, same data, same two seeds. use_pe=False removes that one addition in forward and changes nothing else.
positional encoding mean sd runs
yes 990 23 [1013, 967]
no 1282 8 [1290, 1274]292 RMSE. About 29% worse, from deleting one line.
And now the sentence I want you to take out of this course, never mind this lab: nothing raised an exception.
The model without positional encoding trained for thirty epochs. The loss went down. It converged. It returned forecasts in the right units, in the right shape, with a plausible RMSE that is better than the seasonal-naive benchmark of 2,152. If you had no comparison to make, you would ship it.
What it was actually doing is working from an unordered bag of 52 weeks. It could not distinguish last week from a year ago. It knew what the year contained and nothing about when.
Look at the sd column before anyone asks. The two conditions have spreads of 23 and 8 against a gap of 292 — twelve to thirty-six times the noise. By the standard Lab 7 set, this is about as unambiguous as a result gets.
The practical warning: nn.TransformerEncoderLayer hands you attention, residual connections, layer normalization and the feed-forward sublayer. It does not hand you position, there is no default, and nothing checks that you added it. In a language model you would notice, because the output would be word salad. In a forecaster the output is a number, and numbers always look fine.
Try it
Fill in the blank and express the penalty as a percentage. You should get 292 RMSE, 29% worse.
10.6 Step 4 — Where attention lands
board = {
"LASSO, 46 features (HW4)": 744,
"XGBoost (HW4)": 781,
"Vanilla RNN (Lab 9 Pt 1)": 805,
"LSTM (Lab 9 Pt 1)": 998,
"Transformer (today)": round(np.mean([r for r in rows[0]["runs"]])),
"1D CNN (Lab 8)": 1032,
"Seasonal naive": 2152,
}
print(pd.Series(board).sort_values().to_string())The Transformer beats the CNN and lands level with the LSTM, on roughly fifteen times the parameters of the RNN that beat both.
Every advantage attention has is an advantage at scale. Parallel computation across positions matters when a sequence is thousands of steps and a corpus is billions of tokens. Direct long-range paths matter when the signal genuinely sits far back. Neither describes 5,900 windows of weekly retail demand.
Walkthrough
LASSO, 46 features (HW4) 744
XGBoost (HW4) 781
Vanilla RNN (Lab 9 Pt 1) 805
Transformer (today) 990
LSTM (Lab 9 Pt 1) 998
1D CNN (Lab 8) 1032
Seasonal naive 2152Read it honestly. The Transformer beats the CNN. It ties the LSTM — 990 against 998, a gap of 8 when the run-to-run spread is 23, so those two are indistinguishable. And it loses to the vanilla RNN by 185, on fifteen times the parameters.
Nothing here is a failed implementation. This is the architecture working correctly on a problem it was not designed for. And that is the point worth carrying. Every advantage attention has is an advantage at scale.
Parallel computation across positions is transformative when your sequence is thousands of steps and your corpus is billions of tokens — an RNN has to walk the sequence one step at a time, and a Transformer does not. Direct paths between distant positions matter when the signal genuinely sits far back and must survive the journey.
Neither describes 5,160 training windows of weekly retail demand, where the answer is mostly in the last few weeks. You are paying for a mechanism whose benefits do not exist at this size, with parameters that have to be estimated from data you do not have.
10.7 Before you leave
Discuss (the missing line): You have now met three failures that produce a plausible number and no error —
KFoldon a time series, a forgottenzero_grad(), and a missing positional encoding. What do the three have in common, and what practice would catch all of them?Discuss (the architecture): Describe a forecasting problem where you would expect the Transformer to beat everything else here. Be specific about sequence length and data volume.
Discuss (for Homework 5): Part 2 asks you to fit all three sequence models and explain the ranking. You have now seen it. What the lab does not answer is whether that ranking would survive on a different series — name one property of a series that would most likely flip it.
Solutions posted to Canvas after class.
Walkthrough
Try it
Three questions for the board.
The missing line. You have now met four failures that produce a plausible number and no error: KFold on a time series, a forgotten zero_grad(), a transposed axis, and a missing positional encoding. What do they have in common, and what single practice would have caught all four?
valid program that means something other than what you intended. Types cannot catch that. Only a benchmark you trust, computed on held-out time, can — which is why the seasonal-naive row has been on every scoreboard since Lecture 1.*
The architecture. Describe a forecasting problem where you would expect the Transformer to beat everything else here. Be specific about two things: sequence length and data volume.
For Homework 5. Part 2 asks you to fit all three sequence models and explain the ranking. You have now seen it. What the lab does not answer is whether the ranking would survive on a different series — name one property of a series that would most likely flip it.
And one closing thought that is worth more than the ranking. Everything you have built in the last four weeks — feedforward, convolutional, recurrent, attention — lost to a 46-coefficient LASSO from Homework 4. That is not an argument against neural networks. It is an argument about matching the model to the problem: 5,900 overlapping windows of weekly retail demand is not the regime any of these architectures was designed for, and the honest thing to do is say so and pick the model that fits the data you actually have.
That is the argument Lecture 13 makes with the whole course’s scoreboard in front of it.
10.8 What your run should produce
Your numbers will differ where a model is stochastic; these are the values this lab produces on the pinned environment.
Deterministic; torch.manual_seed per fit, shuffle=False. Slow — attention is O(T²) at a 52-week window; the lab states about 70 seconds per fit and Step 3 trains four.
| Quantity | Value |
|---|---|
| Windows | 6,750 of 52 weeks — train 5,160, test 1,590 |
| Transformer | 67,329 parameters (RNN: 4,545 — a 14.8× ratio) · seed 1 → 1,013 |
| With positional encoding | mean 990 · sd 23 · runs [1013, 967] |
| Without positional encoding | mean 1282 · sd 8 · runs [1290, 1274] |
| Penalty for the missing line | 292 RMSE — 29% worse; 12–36× the run-to-run spread |
| Attention comparisons per example | 52² = 2,704 per head per layer |
| Scoreboard | LASSO 744 · XGBoost 781 · RNN 805 · Transformer 990 · LSTM 998 · CNN 1,032 · naive 2,152 |
| Transformer vs LSTM | 8 apart against sd 23 — indistinguishable |
| Transformer vs RNN | 185 behind, on 14.8× the parameters |