8  Lab 8: Convolutions — What the Filter Can Actually See

Run this one live: the executable lab.

8.1 What we are doing

Last week you flattened a 26-week window into a vector of 104 numbers and fed it to a network. That model had no idea the window was ordered — week 1 and week 26 were just positions in a list.

A convolution does know. It slides a filter along the time axis and looks for local shapes.

Today you will build one, and then find out how far back it can actually see — which is considerably less than the window you hand it, and is the single most useful thing to understand about this architecture.

You will also use last week’s lesson: every comparison here runs on three seeds, because one run tells you nothing.

Suggested time: ~40 minutes · You will need: pandas, numpy, matplotlib, torch, and data/processed/m5_weekly.csv.

Two cells here are slow

Steps 3 and 4 each train several networks — about four minutes and about three minutes respectively on a laptop. Start the cell, read the text underneath it while it runs, and do not assume it has hung.

8.2 Setup

Same windowing as Lab 7, with one change that matters.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import warnings
warnings.filterwarnings('ignore')

from pathlib import Path

def data_path(name):
    for base in (Path.cwd(), *Path.cwd().parents):
        p = base / "data" / "processed" / name
        if p.exists():
            return p
    raise FileNotFoundError(
        f"Could not find data/processed/{name}. "
        "Run `python scripts/prep_m5.py` from the repository root first.")

weekly = (pd.read_csv(data_path("m5_weekly.csv"), parse_dates=["week_start"])
            .sort_values(["store_id", "cat_id", "week_start"]))
CHANNELS = ["units", "snap_days", "event_days", "avg_price"]
CUTOFF   = np.datetime64("2015-05-17")
rmse = lambda a, b: float(np.sqrt(np.mean((np.asarray(a) - np.asarray(b)) ** 2)))

def build(W):
    X, Y, D = [], [], []
    for _, g in weekly.groupby(["store_id", "cat_id"]):
        arr, dates = g[CHANNELS].values.astype("float32"), g.week_start.values
        for i in range(len(g) - W):
            X.append(arr[i:i+W]); Y.append(arr[i+W, 0]); D.append(dates[i+W])
    X = np.stack(X); Y = np.array(Y, dtype="float32"); D = np.array(D)
    tr, te = D < CUTOFF, D >= CUTOFF
    mu, sd = X[tr].mean((0,1), keepdims=True), X[tr].std((0,1), keepdims=True) + 1e-8
    ym, ys = Y[tr].mean(), Y[tr].std()
    Xn, Yn = (X - mu)/sd, (Y - ym)/ys
    # Conv1d wants (batch, channels, time) -- the transpose is the whole point
    return (torch.tensor(Xn[tr].transpose(0, 2, 1)), torch.tensor(Yn[tr]),
            torch.tensor(Xn[te].transpose(0, 2, 1)), Y[te], ym, ys)

DATA = {W: build(W) for W in (13, 26, 52)}
print("shapes at W=26:", DATA[26][0].shape, "(batch, channels, time)")
The transpose that bites everyone

Your DataLoader naturally yields (batch, time, channels). Conv1d demands (batch, channels, time).

Get it backwards and nothing crashes. The model convolves across the four channels instead of along time, learns something meaningless, and returns a number. Same failure mode as forgetting zero_grad() last week: no exception, just a wrong answer.

shapes at W=26: torch.Size([5940, 4, 26]) (batch, channels, time)

Same windowing as Lab 7, wrapped in a build() function so we can rebuild it at three window lengths. Same panel, same cutoff, same train/test split, statistics from training rows only. One change matters, and it is in the last line of build.

Conv1d wants . Your natural windowing produces (batch, time, channels)**. So there is a transpose, and the comment calls it “the whole point.”

Read this callout properly, because it is the same species of bug as last week’s missing zero_grad().

Get the transpose backwards and nothing crashes. The shapes are still valid — you have four of one thing and twenty-six of another, and Conv1d will happily convolve across the four channels as though they were the time axis. It will slide a filter over “units, SNAP days, event days, price” in that order, learn something with no meaning at all, and return a number that looks like an RMSE.

That is the third time this course has shown you the same shape of failure: shuffle=True on a time series, a missing zero_grad(), and now a transposed axis. None of them raise. All of them return a plausible number. This is what makes deep learning code different from regression code — the language cannot tell you that your tensor means the wrong thing.

8.3 Step 1 — How far back can the filter see?

Before building anything, work this out on paper.

Our network is Conv1d(4, 32, kernel_size=5) then Conv1d(32, 64, kernel_size=3).

  • A neuron in layer 1 sees 5 consecutive weeks.
  • A neuron in layer 2 sees 3 layer-1 neurons — which between them span \(3 + (5 - 1) = \mathbf{7}\) weeks.
def receptive_field(kernels):
    rf = 1
    for k in reversed(kernels):
        rf = rf + (k - 1)
    return rf

print(f"kernels (5, 3)      -> receptive field {receptive_field([5,3])} weeks")
print(f"kernels (5, 3, 3)   -> receptive field {receptive_field([5,3,3])} weeks")
print(f"kernels (5, 3, 3, 3)-> receptive field {receptive_field([5,3,3,3])} weeks")

Seven weeks. Not 26 — seven. Hand this model a 52-week window and each output neuron still sees only seven consecutive weeks of it.

Hold that fact. Step 4 tests what it costs.

Before we build anything, we do the arithmetic on paper. This is the calculation that decides whether the architecture can possibly work, and it takes ten seconds. Our network is Conv1d(4, 32, kernel_size=5) and then Conv1d(32, 64, kernel_size=3).

A neuron in the first layer sees five consecutive weeks — that is what kernel size 5 means.

A neuron in the second layer sees three first-layer neurons. But those three overlap: each sees five weeks, shifted by one. So between them they span 3 + (5 − 1) = seven weeks.

kernels (5, 3)      -> receptive field 7 weeks
kernels (5, 3, 3)   -> receptive field 9 weeks
kernels (5, 3, 3, 3)-> receptive field 11 weeks

Seven weeks. Not twenty-six. Seven.

And look at what stacking buys: each extra kernel-3 layer adds two weeks. Two. You go from seven to nine to eleven, one whole convolutional layer at a time.

Hand this network a 52-week window and every output neuron still sees seven consecutive weeks of it. The annual cycle — the thing that actually drives a retail series — is not merely hard for this model to find. It is outside what any neuron in it can look at. Hold that. Step 4 measures what it costs.

8.4 Step 2 — Build the CNN

def conv_out_len(W):
    return W - 4 - 2          # kernel 5, then kernel 3, no padding

def make_cnn(W, head, seed):
    torch.manual_seed(seed)
    body = [nn.Conv1d(4, 32, 5), nn.ReLU(), nn.Conv1d(32, 64, 3), nn.ReLU()]
    if head == "flatten":
        tail = [nn.Flatten(), nn.Linear(64 * conv_out_len(W), 1)]
    elif head == "avgpool":
        tail = [nn.AdaptiveAvgPool1d(1), nn.Flatten(), nn.Linear(64, 1)]
    else:
        tail = [nn.AdaptiveMaxPool1d(1), nn.Flatten(), nn.Linear(64, 1)]
    return nn.Sequential(*body, *tail)

def train(model, W, epochs=40):
    Xtr, Ytr, Xte, Yte, ym, ys = DATA[W]
    opt = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)
    lossfn = nn.MSELoss()
    loader = torch.utils.data.DataLoader(
        torch.utils.data.TensorDataset(Xtr, Ytr), batch_size=128, shuffle=False)
    for _ in range(epochs):
        model.train()
        for xb, yb in loader:
            opt.zero_grad(); lossfn(model(xb).squeeze(-1), yb).backward(); opt.step()
    model.eval()
    with torch.no_grad():
        return rmse(Yte, model(Xte).squeeze(-1).numpy() * ys + ym)

m = make_cnn(26, "flatten", seed=1)
print(f"parameters: {sum(p.numel() for p in m.parameters()):,}")
print(f"test RMSE:  {train(m, 26):,.0f}")
parameters: 8,161
test RMSE:  980

Two convolutional layers, then a head that flattens and maps to one output.

Compare that parameter count with last week: 8,161 against 21,761. The CNN is about a third the size of the feedforward network — because the filters are shared across all time positions instead of every input getting its own weight. And on this single seed it scores 980, against the FFN’s 1,109.

Smaller model, better number. That is the convolutional bargain: weight sharing buys you efficiency, as long as the thing you are looking for is local. Which — per Step 1 — means seven weeks local. Keep that qualifier attached.

8.5 Step 3 — Does pooling belong here?

Lecture 8 argued that pooling buys translation invariance — an image classifier should not care where in the frame the cat is — and that a forecaster usually should care where the spike was.

Test it. Three heads, three seeds each, because of last week.

head_runs = {}
rows = []
for head in ("flatten", "maxpool", "avgpool"):
    head_runs[head] = [train(make_cnn(26, head, s), 26) for s in (1, 2, 3)]
    rows.append({"head": head, "mean": round(np.mean(head_runs[head])),
                 "sd": round(np.std(head_runs[head])),
                 "runs": [round(r) for r in head_runs[head]]})
print(pd.DataFrame(rows).to_string(index=False))

Read the spread before the means. The run-to-run sd is around 50–80. The gap between flatten and avgpool is several hundred. That difference is real in a way last week’s weight decay comparison was not.

Flatten keeps all 64 channels at all 20 remaining time positions, so the final layer knows where each pattern was found. AdaptiveAvgPool1d(1) averages across time and throws that away.

Say it in one sentence: why would averaging over time be the right thing to do for classifying photographs and the wrong thing for forecasting demand?

Lecture 8 argued that pooling buys translation invariance. An image classifier should not care where in the frame the cat is; pooling throws away position and keeps the presence of the pattern. That is exactly right for photographs.

The claim was that a forecaster usually should care where the spike was. A demand spike three weeks ago and a demand spike twenty weeks ago are different facts about next week.

Three heads, three seeds each.

   head  mean  sd               runs
flatten  1032  80   [980, 1145, 972]
maxpool  1418  47 [1433, 1354, 1467]
avgpool  1534  81 [1643, 1448, 1509]

Read the spread before you read the means — that is the habit last week was for.

Run-to-run standard deviation is 47 to 81. The gap between flatten and avgpool is 502. That is six to ten times the noise. Unlike last week’s weight decay comparison, this difference is unambiguously real, and you can say so with three runs precisely because you measured the spread.

The mechanism is worth stating exactly. Flatten keeps all 64 channels at all 20 surviving time positions and hands the final layer 1,280 numbers — so the model knows where each pattern was found. AdaptiveAvgPool1d(1) collapses the time axis to a single average per channel: 64 numbers. It keeps “this pattern occurred” and discards “it occurred in week 22.”

Try it

Pause and answer: why is averaging over position right for classifying photographs and wrong for forecasting demand?

Because in a photograph, position is nuisance — the cat is a cat wherever it sits, and a model that has to learn “cat in the top left” and “cat in the middle” separately is wasting capacity. In a forecast, position is the signal. Recency is the single most informative thing about a lag, and average pooling deletes it.

Note that max pooling is less bad than average pooling — 1,418 against 1,534 — which fits: max at least preserves the strongest response rather than diluting it across the window. Both are far behind keeping the positions.

8.6 Step 4 — A longer window, the same receptive field

Now the experiment Step 1 set up. Keep the architecture fixed and vary only the window.

rows = []
for W in (13, 26, 52):
    # W=26 was already fitted in Step 3 -- reuse those three rather than paying twice
    runs = head_runs["flatten"] if W == 26 else [train(make_cnn(W, "flatten", s), W) for s in (1, 2, 3)]
    n_par = sum(p.numel() for p in make_cnn(W, "flatten", 1).parameters())
    rows.append({"window": W, "receptive_field": receptive_field([5,3]),
                 "parameters": n_par, "mean_RMSE": round(np.mean(runs)),
                 "sd": round(np.std(runs)), "runs": [round(r) for r in runs]})
print(pd.DataFrame(rows).to_string(index=False))
The result runs the wrong way, and that is the lesson

A longer window does not help. Read the ends of the table, not the middle.

13 weeks and 26 weeks sit inside each other’s run-to-run spread — a 52-point gap against a standard deviation of 80. That is exactly the kind of difference Lab 7 taught you not to trust, and you should not trust it here either. But 52 weeks is 123 RMSE worse than 13 weeks, a gap no seed accounts for: the worst 13-week run still beats the best 52-week run.

The receptive field is stuck at seven weeks no matter how much history you supply. What a longer window does add is parameters — the flatten layer grows with W, from 7,329 to 9,825 — so you get more capacity to overfit and no additional context to justify it.

More data into the model is not more information out of it, unless the architecture can reach it.

Your turn: using receptive_field(), find how many kernel-3 layers you would need to stack after the first kernel-5 layer to actually span 52 weeks.

# ---- your turn: uncomment and fill in ----
# k = [5]
# while receptive_field(k) < 52:
#     k.append(___)               # <- what kernel size?
# print(f"{len(k)} layers needed: kernels {k} -> {receptive_field(k)} weeks")

Now the experiment Step 1 set up, and it is the reason this lab exists.

Keep the architecture completely fixed — same two layers, same kernels, same seven-week receptive field — and vary only how much history you hand it. Thirteen weeks, twenty-six, fifty-two. Three seeds each, and we reuse Step 3’s three flatten runs for the 26-week row rather than paying for them twice.

 window  receptive_field  parameters  mean_RMSE  sd               runs
     13                7        7329        980  42   [1039, 960, 943]
     26                7        8161       1032  80    [980, 1145, 972]
     52                7        9825       1103  26  [1111, 1068, 1131]

Quadrupling the window made the forecast worse.

Now let me show you how to read this table honestly, because there are two different claims in it and only one of them is safe.

The middle comparison is not evidence. Thirteen weeks against twenty-six is 980 against 1,032 — a gap of 52, when the 26-week row’s own standard deviation is 80. Last week you decided that differences smaller than about 100 RMSE, from a handful of runs, are not findings. That rule applies to results you like as much as results you don’t. Those two windows are indistinguishable.

The ends are evidence. Thirteen weeks against fifty-two is 980 against 1,103 — 123 RMSE, and the spreads do not touch: the worst 13-week run, 1,039, still beats the best 52-week run, 1,068. Three seeds each, no overlap. That one you can report.

So the finding is: giving this architecture four times the history made it measurably worse, and giving it twice the history did nothing at all.

Why? Look at the two columns either side of the RMSE. The receptive field is 7 on every row — it does not move, because nothing about the architecture changed. The parameter count goes 7,329, 8,161, 9,825 — because the flatten layer’s input grows with the window.

So a longer window adds capacity and adds no reach. Every extra week you supply becomes more weights to fit and not one additional week that any neuron can actually look at. That is a recipe for overfitting, and the 52-week column is what it looks like.

More data into the model is not more information out of it, unless the architecture can reach it.

Try it

Fill in the blank in the next cell and find out how many kernel-3 layers you would have to stack after that first kernel-5 layer to span a full 52 weeks.

The answer is 24 — twenty-five convolutional layers in total, for a receptive field of 53 weeks. Twenty-three is not enough; it reaches 51.

Sit with that for a second. To let this architecture see one year, you would need a twenty-five-layer network — on 5,940 training examples that overlap almost completely. Nobody is going to do that, and if they did it would overfit spectacularly.

That is the honest limit of a plain 1D CNN for this problem, and it is why the next lecture is about a different mechanism rather than a deeper stack of the same one. Real practitioners reach for dilated convolutions here — filters with gaps in them, so the receptive field grows multiplicatively instead of by two per layer. That is a genuine third option and worth knowing the name of.

8.7 Step 5 — Where does it land?

board = {
    "LASSO, 46 features (HW4)": 744,
    "XGBoost (HW4)": 781,
    "FFN, flattened window (Lab 7)": 1084,
    "1D CNN, flatten head (today)": round(np.mean(head_runs["flatten"])),   # from Step 3
    "Seasonal naive": 2152,
}
print(pd.Series(board).sort_values().to_string())

The CNN is competitive with the feedforward network and well behind both tabular models — on a problem where the useful signal sits at lags the filter cannot reach.

LASSO, 46 features (HW4)          744
XGBoost (HW4)                     781
1D CNN, flatten head (today)     1032
FFN, flattened window (Lab 7)    1084
Seasonal naive                   2152

Say the honest version of this table.

The CNN at 1,032 and the FFN at 1,084 differ by 52 — and you measured the run-to-run spread at 80. So the correct statement is not “the CNN beat the feedforward network.” It is: the two are indistinguishable on this problem. One-third the parameters, a genuinely better inductive bias, ordering preserved — and it bought nothing measurable.

Both are well behind a 46-coefficient LASSO at 744.

That is not a failure of the lab and it is not a bad implementation. It is a result about the problem. The useful signal in weekly store sales sits at lags a seven-week filter cannot reach — last year’s same week, the annual shape — and no amount of care inside this architecture will recover something the receptive field excludes.

Which is the honest reason to move on to Lecture 9, and not a reason to conclude that convolutions are bad.

8.8 Before you leave

Discuss (the architecture): You have now seen two ways to fail to use a 52-week window: the FFN flattens it and loses the ordering, the CNN preserves ordering but can only see seven weeks at a time. Describe, without naming an architecture, what a model would need in order to use the whole window properly.

Discuss (the practice): Step 4’s result — longer window, worse forecast — is the kind of thing a practitioner discovers after a week of tuning. What one calculation, done in Step 1, would have predicted it?

Discuss (for Homework 5): Part 1 Q4 asks you to run this window experiment yourself and explain the limit. You now have the answer. What it does not tell you is what to change architecturally — name two options and say which you would try first.

Solutions posted to Canvas after class.

Try it

Three questions for the board.

The architecture. You have now seen two distinct ways to waste a window. The FFN flattens it and loses the ordering entirely. The CNN preserves ordering but can only look at seven weeks at a time. Without naming an architecture, describe what a model would need in order to use a whole 52-week window properly.

The practice. Step 4’s result — longer window, no better or worse forecast — is the sort of thing a practitioner discovers after a week of tuning. What one calculation, done in Step 1 in ten seconds, would have predicted it?

For Homework 5. Part 1 Question 4 asks you to run this window experiment and explain the limit. You now have the answer. What it does not tell you is what to change. Name two architectural options and say which you would try first — and note that “stack more conv layers” is one of them, so work out how many you would need before you recommend it.

Solutions for the coded parts go up on Canvas after the deadline. Next week: a model that carries state forward through the whole window.

8.9 What your run should produce

Your numbers will differ where a model is stochastic; these are the values this lab produces on the pinned environment.

Deterministic; torch.manual_seed per model, shuffle=False. Slowest lab in the course — about eight minutes end to end, with Step 3 ≈ 4 min (9 fits) and Step 4 ≈ 3 min (6 new fits).

Quantity Value
Tensor shape at W=26 (5940, 4, 26) — batch, channels, time
Receptive field (5,3): 7 weeks · (5,3,3): 9 · (5,3,3,3): 11 — +2 per layer
CNN, W=26, flatten head 8,161 parameters (FFN in Lab 7: 21,761) · seed 1 → 980
Head comparison, 3 seeds flatten 1032 (sd 80) · maxpool 1418 (sd 47) · avgpool 1534 (sd 81)
flatten vs avgpool gap 502 — six to ten times the run-to-run spread
Window, 3 seeds — 13 mean 980 · sd 42 · runs [1039, 960, 943] · 7,329 params
Window, 3 seeds — 26 mean 1032 · sd 80 · runs [980, 1145, 972] · 8,161 params
Window, 3 seeds — 52 mean 1103 · sd 26 · runs [1111, 1068, 1131] · 9,825 params
13 vs 52 123 RMSE, no overlap (worst 13 = 1,039 < best 52 = 1,068)
13 vs 26 52 RMSE — inside the noise, not a finding
Five-seed confirmation (not in the lab) 13: 975 ± 48 · 26: 984 ± 87 · 52: 1,094 ± 35
Layers to span 52 weeks 24 kernel-3 layers after the kernel-5 (25 total → 53 weeks)
Scoreboard LASSO 744 · XGBoost 781 · CNN 1032 · FFN 1084 · seasonal naive 2152