Why Convolution? A Dense Layer Wastes Its Weights

A \(224 \times 224\) image has 50,176 pixels. One hidden layer of 1,000 neurons on top of it needs 50 million weights — for one layer. Lecture 7’s whole network had 22,657.

And the waste is worse than the cost. That layer has no idea that neighboring pixels are related and distant ones usually are not. And it cannot reuse what it learns: a pattern learned in the top-left corner is stored separately from the same pattern in the bottom-right.

Why Convolution? Two Ideas

A reader does not memorize the letter “A” separately for every spot on the page. They learn it once and apply it everywhere. A filter does exactly that.

How a Convolutional Layer Works

For a sequence, a 1D convolution slides a filter of width \(K\) along the series and takes a dot product at each position:

\[(f * g)[t] = \sum_{k=0}^{K-1} f[k] \cdot g[t - k]\]

Each filter learns one local pattern — an edge, a spike, a seasonal dip. Its output is a feature map whose high values mark where that pattern was found.

Convolution, Step by Step

A length-10 series and a width-3 filter \([1, 5, 1]\). Slide the filter along and take a dot product at each position.

Valid keeps only full overlaps: \(10 - 3 + 1 = 8\) outputs. Same pads a zero at each end and keeps all 10. Max pooling then keeps the larger of each pair.

What a Filter Sees

The same operation in two dimensions: a \(3\times3\) filter slides over every pixel of a photograph. Two hand-set filters find two different kinds of structure.

18 March 2020: shelves emptied by a shock no forecast saw coming — more in Week 15.

Pooling: Downsampling Feature Maps

Pooling summarizes each small region of a feature map with one number: max pooling keeps the largest, average pooling the mean.

It buys fewer parameters downstream and translation invariance: shift the input slightly and the pooled output barely changes. The shelf edge survives.

LeNet to VGG: Depth and Small Filters

LeNet-5 (LeCun et al. 1998) set the template in 1998: convolution and pooling layers, then dense layers. It read handwritten digits on bank checks.

VGG (Simonyan and Zisserman 2015) scaled it up in 2014 with one simplification: only \(3\times3\) filters, stacked deep. Two stacked \(3\times3\) layers see what one \(5\times5\) sees, with fewer parameters and an extra nonlinearity.

VGG’s cost: 138 million parameters, mostly in the final dense layers.

Inception and ResNet: Depth That Trains

Inception (Szegedy et al. 2015) runs \(1\times1\), \(3\times3\) and \(5\times5\) convolutions in parallel and lets the network choose the scale; \(1\times1\) convolutions keep the cost down.

ResNet (He et al. 2016) fixed a different problem: past some depth, deeper plain networks trained worse — not overfitting, but an optimization difficulty. The fix is a residual or skip connection:

Each block learns only a change to \(\mathbf{x}\), and the \(+\,\mathbf{x}\) gives the gradient a direct path backward. It made 100-layer networks trainable, and it is in nearly every deep architecture since — including Lecture 9’s.

What a Trained CNN Learns

ResNet-18, trained on 1.2 million labeled images. Nobody designed its first layer — these are the 64 filters it arrived at, and what four of them find in the same photograph.

Many look like the edge filters we set by hand; the rest detect color. Learned, not engineered — the same promise a 1D CNN makes for a time series.

What Carries Over to Time Series

These architectures were built for images, on problems that look nothing like weekly demand forecasting. Three of their ideas transfer anyway.

One idea does not transfer, and it is the one we flagged: pooling for translation invariance. An image classifier should not care where the cat is. A forecaster usually should care where the spike was.

Why Use a 1D CNN for Forecasting?

A 1D CNN takes a window of the recent past — 26 weeks of units, SNAP days, events and price — and predicts the next week. You stop choosing lags: the filters learn which local shapes matter. It is far cheaper than a dense network (weight sharing again), and the same filters serve every store-category at once.

The catch is the receptive field. Stacked width-5 and width-3 filters see seven weeks; annual seasonality is 52 away. Add dilation, or feed the annual lag in as its own channel.

1D CNN Forecasting Pipeline in PyTorch

import torch.nn as nn

model = nn.Sequential(
    nn.Conv1d(n_channels, 32, kernel_size=5), nn.ReLU(),
    nn.Conv1d(32, 64, kernel_size=3),         nn.ReLU(),
    nn.AdaptiveAvgPool1d(1),   # collapses the time axis
    nn.Flatten(),
    nn.Linear(64, 1))

Input shape is (batch, channels, time): channels before time, the transpose of a dataframe. Lecture 7’s rules still apply: Adam with weight decay, zero_grad(), shuffle=False, a split by time.

AdaptiveAvgPool1d(1) is global average pooling: one mean per filter over the whole time axis. A flatten head skips it and gives every position its own weight. Watch what that does on the next slide.

Does It Actually Work?

Fitted on our weekly panel — 26-week windows, four channels, 30 store-category series, the same test block as Homework 4.

Every CNN variant loses to a 46-coefficient linear model — their 26-week window does not reach the annual lag LASSO was handed directly.

And swapping average pooling for Flatten, which keeps where each pattern occurred, is worth about 500 RMSE — three seeds each, against a spread of about 80.

CNN Architecture Summary

VGG, Inception and ResNet appeared a year before the papers cited for them.

Reach for a 1D CNN when local shape matters more than exact position, when you have many parallel series, or as a cheap feature extractor feeding something else.

Do not reach for one when the dominant signal is at a long lag, when you have well-engineered tabular features already, or when a stakeholder needs to read the model. On the evidence above, that covers most of what we forecast in this course.

Lecture 8: Key Takeaways

Next: RNNs and LSTMs — built to carry state through time.