8  CNN Architectures

Lecture 8 of ECON 8310.

8.1 Lecture Outline

1. Convolution: The Core Idea
2. Pooling and Dropout
3. CNN Architecture Evolution
4. 1D CNNs for Time Series
5. Key Takeaways

Four parts. Convolution itself — what a filter is and what it costs. Pooling, which is where I want you to be suspicious. Then twenty years of image architectures, compressed into the three ideas that outlived them. And then one-dimensional CNNs on our own time series, with an honest measurement at the end.

A word on why we’re spending time on image architectures in a forecasting course, since that’s a fair question to have.

Two reasons. The first is that three of their ideas transfer directly, and you’ll recognize one of them again next week inside a completely different model. The second is that the history is a worked example of the thing this course keeps insisting on: every one of these architectures won by encoding a better assumption about its data, not by being bigger. That’s the transferable lesson, and it happens to be easier to see in pictures than in demand curves.

8.2 Convolution: The Core Idea

Look at a small window, and look for the same thing everywhere. Two ideas, and they are the whole architecture.

Two ideas: look locally, and reuse what you learn everywhere.

8.2.1 Why Convolution?

Start with why a fully connected layer fails on structured input. A \(224 \times 224\) image has 50,176 pixels, so a single hidden layer of 1,000 neurons needs 50 million weights — for one layer. Lecture 7’s network had 22,657 parameters in total.

Worse than the cost is the waste. That layer ignores spatial structure: it has no notion that neighboring pixels are related and distant ones usually are not. And it does not generalize position — a pattern learned in the top-left corner is stored separately from the same pattern in the bottom-right.

A convolutional layer fixes both with two ideas. Local connectivity: each neuron sees only a small receptive field. Weight sharing: the same filter is applied at every position. Far fewer parameters, and a pattern learned anywhere is recognized everywhere.

The analogy that makes it stick: a reader does not memorize the letter “A” separately for every position on the page. They learn it once and apply it everywhere. A filter does exactly that.

Start with why a fully connected layer fails on structured input, because the failure is dramatic.

A two-twenty-four by two-twenty-four image has just over fifty thousand pixels. If you feed that into a single hidden layer of a thousand neurons, that layer needs fifty million weights. For one layer. Put that next to last week’s network, which had twenty-two thousand six hundred fifty-seven parameters in total. But the cost is not the worst part. The waste is.

That fully connected layer has no notion that neighboring pixels are related and distant ones usually are not — it treats pixel one and pixel fifty thousand as equally likely to interact. And it does not generalize position: a pattern it learns in the top-left corner is stored completely separately from the same pattern in the bottom-right. It has to learn everything twice.

A convolutional layer fixes both problems with two ideas. Local connectivity — each neuron sees only a small window, called its receptive field. And weight sharing — the same filter is applied at every position in the input. Far fewer parameters, and a pattern learned anywhere is recognized everywhere.

The analogy that makes it stick: you don’t memorize the letter “A” separately for every position on the page. You learn it once, and you apply that one recognizer everywhere you look. A filter does exactly that.

And notice this is the same kind of move we’ve made before, in a different costume. Weight sharing is a constraint. We are telling the model, before it sees any data, that a pattern occurring at position ten and the same pattern at position two hundred should be treated identically. That is a strong assumption, and it is what buys the parameter reduction.

Constraints that encode true structure make models better. Constraints that encode false structure make them worse, and no amount of training fixes it. Hold that thought — by the end of this lecture we will have an example of each, on the same dataset.

8.2.2 How a Convolutional Layer Works

For a sequence, a 1D convolution slides a filter of length \(K\) along the series and takes a dot product at each position:

\[(f * g)[t] = \sum_{k=0}^{K-1} f[k] \cdot g[t - k]\]

Kernel size	Filter length \(K\). Larger means a wider receptive field — more context per output, and more parameters.
Stride	How far the filter moves between positions. Stride 2 halves the output length.
Padding	Zeros at the edges. padding='same' keeps output length equal to input length.
Filters	How many different patterns to learn. This becomes the number of output channels.

Each filter learns one local pattern — an edge, a spike, a seasonal dip. Its output is a feature map whose high values mark where that pattern was found.

For a sequence — which is what we care about — a one-dimensional convolution slides a filter of length K along the series, and at each position it takes a dot product.

That’s the formula on the slide, and honestly the formula is less useful than the picture: a small window sliding along, computing one number at each stop.

Kernel size is the filter length. Bigger means a wider receptive field — more context per output — and more parameters. Stride is how far the filter jumps between positions. Stride two halves your output length.

Padding puts zeros at the edges. Setting padding to 'same' keeps the output the same length as the input, which saves you a lot of arithmetic.

Filters is how many different patterns you want to learn. That number becomes the channel count of the output.

Here’s what to hold onto. Each filter learns one local pattern — an edge, a spike, a seasonal dip. What comes out is called a feature map, and its high values mark the places where that pattern was found.

Two things follow from that, and they’re worth stating explicitly because they shape how you read the results later.

First, the filter’s weights are learned, not designed. You don’t tell it to look for a promotional spike. You give it a slot for thirty-two patterns and gradient descent decides what those patterns are. That is the “learned features” promise — the thing that’s supposed to replace your hand-built lag columns.

Second — and this is the constraint — a filter can only see K steps at a time. Stack two layers and the second one sees a window of windows, so the reach grows, but it grows slowly. This is called the receptive field, and I’m naming it now because it is going to be the single most important number in this lecture.

8.3 Pooling and Dropout

Pooling shrinks the feature map and buys robustness — at a price that matters more for forecasting than for images.

8.3.1 Pooling: Downsampling Feature Maps

Pooling reduces a feature map by summarizing each small region with a single number. Max pooling takes the largest value in the window; average pooling takes the mean. A \(2\times2\) max pool with stride 2 quarters the number of values.

Two things are bought. Fewer parameters downstream, because later layers see a smaller input. And translation invariance: shift the input slightly and the pooled output barely changes, so the network recognizes a cat whether it sits left or right of center.

Translation invariance is a feature for images and a problem for forecasting. If a demand spike moved three weeks later, that is not a nuisance to be smoothed away — it is the thing you are trying to predict. Pooling deliberately discards where in the window a pattern occurred, and for a time series the position is often the signal.

Hold that objection. We measure exactly what it costs later in this lecture.

Pooling reduces a feature map by summarizing each little region with one number. Max pooling takes the largest value in the window. Average pooling takes the mean. A two-by-two max pool with stride two cuts the number of values to a quarter.

You buy two things. Fewer parameters downstream, because the later layers see a smaller input. And translation invariance — shift the input slightly and the pooled output barely changes, so the network recognizes a cat whether it’s left or right of center. And here is where I want you to get suspicious. Translation invariance is a feature for images and a problem for forecasting.

Think about what it’s actually doing. If a demand spike moved three weeks later, pooling treats that as noise to be smoothed away. But that shift is not a nuisance — for a forecaster, when the spike happened is very often the entire thing you’re trying to predict.

Pooling deliberately discards where in the window a pattern occurred. For an image classifier that’s exactly right. For a time series, position is frequently the signal.

Hold that objection. Don’t resolve it yet. We’re going to measure precisely what it costs, later in this lecture.

8.4 CNN Architecture Evolution

Twenty years of image models, and the three ideas that outlived them.

8.4.1 LeNet to VGG: Depth and Small Filters

LeNet-5 [@LeCun1998] established the template in 1998: alternating convolution and pooling layers, then fully connected layers at the end. It had roughly 60,000 parameters and read handwritten digits on cheques.

VGG [@Simonyan2015] scaled that idea up in 2014 and made one sharp simplification: use only \(3\times3\) filters, but stack many of them. Two stacked \(3\times3\) layers see the same receptive field as one \(5\times5\), with fewer parameters and an extra nonlinearity in between.

The lesson that carried: depth built from small filters beats width built from large ones. It is cheaper in parameters and more expressive, and it is why almost every modern architecture is deep and narrow rather than shallow and wide.

VGG’s cost was its size — around 138 million parameters, most of them in the final fully connected layers. That is the problem the next two architectures attacked.

LeNet-5, in 1998, established the template that everything since has followed: alternating convolution and pooling layers, then fully connected layers at the end. Roughly sixty thousand parameters, and it read handwritten digits on bank cheques — a real deployed system, in the nineties.

VGG, in 2014, scaled that up and made one sharp simplification. Use only three-by-three filters — but stack a lot of them.

Why does that work? Two stacked three-by-threes see the same receptive field as one five-by-five. But they use fewer parameters, and you get an extra nonlinearity in between them, which makes the combination strictly more expressive.

That is the lesson that carried forward: depth built from small filters beats width built from large ones. Cheaper and more expressive. It’s why nearly every modern architecture is deep and narrow rather than shallow and wide.

VGG’s cost was its size — around a hundred thirty-eight million parameters, and most of those were sitting in the final fully connected layers, not in the convolutions at all. That is exactly the problem the next two architectures went after.

That detail is worth pausing on, because it is a good lesson about where cost actually lives. The convolutions — the part everyone talks about — were relatively cheap, precisely because of weight sharing. The expensive part was the ordinary dense layers bolted on the end, which have no sharing at all.

When you profile a model and find the bottleneck, it is very often in the least interesting component. That’s true of forecasting pipelines too: the modeling is rarely what costs you, the data preparation is.

8.4.2 Inception and ResNet: Depth That Trains

Inception [@Szegedy2015] asked why a layer should commit to one filter size. Its module runs \(1\times1\), \(3\times3\) and \(5\times5\) convolutions in parallel and concatenates the results, letting the network choose the scale. \(1\times1\) convolutions reduce channel count first, keeping the cost down.

ResNet [@He2016] solved a different problem. Past a certain depth, deeper networks got worse — not from overfitting, but because gradients degraded on their way back through many layers. The fix is a residual or skip connection:

\[\mathbf{y} = \mathcal{F}(\mathbf{x}) + \mathbf{x}\]

Each block learns the difference from its input rather than a whole new representation, and the \(+\,\mathbf{x}\) gives the gradient a direct path backward. That single change made networks of 100+ layers trainable, and skip connections now appear in nearly every deep architecture — including the Transformers in Lecture 9.

Two different attacks on that problem.

Inception asked a good question: why should a layer commit to one filter size at all? Its module runs one-by-one, three-by-three and five-by-five convolutions in parallel and concatenates the results, letting the network decide which scale matters. The one-by-one convolutions do something clever — they cut the channel count first, so the expensive convolutions operate on a smaller input. ResNet solved a different problem, and it’s the one that transfers to us.

Past a certain depth, deeper networks were getting worse. Not from overfitting — the training error was worse too. The gradients were degrading on their way back through many layers. The fix is one line: y equals F of x, plus x. A residual, or skip, connection.

Two ways to read that. Each block now learns the difference from its input rather than a whole new representation — usually an easier job. And the plus-x gives the gradient a direct path backward, unimpeded by the layers in between.

That single change made networks of a hundred layers and more trainable. Skip connections now appear in essentially every deep architecture — including the Transformers we’ll see in Lecture 9, and, as it turns out, inside the LSTM, which predates ResNet by eighteen years.

I want to draw out why the addition matters, because we will use this exact argument again next week and it’s easier to absorb twice.

When a gradient travels backward through a chain of layers, each layer multiplies it by something. Multiply enough numbers below one together and you get essentially zero — the signal dies before it reaches the early layers, so they never learn.

Addition doesn’t do that. The plus-x term hands the gradient a route home that skips the multiplication entirely. Repeated multiplication shrinks; repeated addition doesn’t.

Remember that sentence. In Lecture 9 you’ll see the LSTM’s cell state doing precisely the same thing, for precisely the same reason, invented eighteen years earlier for time rather than for depth.

8.4.3 What Carries Over to Time Series

These architectures were built for images, on problems that look nothing like weekly demand forecasting. Three of their ideas transfer anyway.

Small stacked filters	A width-3 filter twice sees six weeks of context with far fewer weights than one width-6 filter.
Skip connections	Same fix for the same problem — gradients that fade through depth.
Learned features	You stop hand-engineering lag windows and let filters find the local shapes that matter.

One idea does not transfer, and it is the one we flagged: pooling for translation invariance. An image classifier should not care where the cat is. A forecaster usually should care where the spike was.

These architectures were built for images, on problems that look nothing like weekly demand forecasting. Three of their ideas transfer anyway.

Small stacked filters. A width-three filter applied twice sees six weeks of context, with far fewer weights than one width-six filter would need. Skip connections. Same fix, same problem — gradients that fade as they travel back through depth. Learned features. You stop hand-engineering lag windows and let the filters find whichever local shapes actually matter.

One idea does not transfer, and it’s the one we flagged twenty minutes ago: pooling for translation invariance.

An image classifier should not care where the cat is. A forecaster usually should care very much where the spike was. So we’re carrying two assumptions into the next section, and they are not equally good.

Locality — that nearby time steps are more related than distant ones — is largely true of demand data. Last week matters more than the week before. Good assumption.

Translation invariance — that where a pattern occurred doesn’t matter — is largely false for forecasting. Good assumption for images, bad one here.

A CNN gives you both, bundled together, because they came from the same field. Part of using one well is knowing which half you actually wanted.

8.5 1D CNNs for Time Series

The same machinery, along a time axis — and what it costs.

8.5.1 Why Use a 1D CNN for Forecasting?

A 1D CNN takes a window of the recent past — say 26 weeks — across several channels, and predicts the next value. Each channel is one series: units, SNAP days, events, price.

The appeal is that you stop choosing lags. Rather than deciding that lag 1, lag 4 and lag 52 matter, you hand the model the whole window and let the filters learn which local shapes predict what comes next.

Two genuine advantages. It is far cheaper than a fully connected network on the same window — weight sharing again. And it handles many parallel series naturally, since the same filters apply to every store-category at once.

The catch is the receptive field. Stacked width-5 and width-3 filters see about seven weeks of context. Annual seasonality is 52 weeks away, and no reasonable stack of small filters reaches that far. Either you add dilation, or you feed the annual lag in as its own channel.

Here’s the setup. A one-dimensional CNN takes a window of the recent past — say twenty-six weeks — across several channels, and predicts the next value. Each channel is one series: units, SNAP days, event flags, price.

The appeal is that you stop choosing lags. Instead of deciding that lag one, lag four and lag fifty-two are the ones that matter, you hand the model the whole window and let the filters work out which local shapes predict what comes next.

Two advantages that are genuinely real. It’s far cheaper than a fully connected network on the same window — that’s weight sharing again. And it handles many parallel series naturally, because the same filters apply to every store-category at once. But the catch is the receptive field, and it’s serious.

Stack a width-five filter and a width-three filter and you see about seven weeks of context. That’s it. Annual seasonality sits fifty-two weeks away, and no reasonable stack of small filters is going to reach that far.

So you have two options. Add dilation — filters with gaps in them, which reach further for the same parameter count. Or feed the annual lag in as its own channel, which is a confession that you’re doing feature engineering after all.

And notice what that second option costs you rhetorically. The entire pitch for a CNN was “stop choosing lags, let the filters find the structure.” The moment you hand it lag_52 as a channel because the receptive field can’t reach, you have given that pitch back. You’re now doing feature engineering and paying for a neural network.

That is not a reason never to use one. It is a reason to be clear-eyed about what the architecture is actually buying you on a given problem. Remember that number — seven weeks. It explains the results two slides from now.

8.5.2 1D CNN Forecasting Pipeline in PyTorch

Window in, one forecast out

import torch.nn as nn

model = nn.Sequential(
    nn.Conv1d(n_channels, 32, kernel_size=5), nn.ReLU(),
    nn.Conv1d(32, 64, kernel_size=3),         nn.ReLU(),
    nn.AdaptiveAvgPool1d(1),   # collapses the time axis
    nn.Flatten(),
    nn.Linear(64, 1))

Input shape is (batch, channels, time) — note that channels come before time, which is the transpose of how the data naturally sits in a dataframe.

Everything from Lecture 7 still applies: Adam with weight decay, zero_grad() every batch, shuffle=False, and a split by time rather than at random.

AdaptiveAvgPool1d(1) averages across the whole time axis. Watch what that does on the next slide.

Six lines. Two convolutions with ReLUs, a pooling step, a flatten, and a linear layer down to one number.

One thing that trips everyone up: the input shape is (batch, channels, time). Channels come before time. That is the transpose of how the data naturally sits in a dataframe, and it’s also the opposite of what nn.LSTM wants next week. You will get this wrong at least once; the error message is unhelpful.

Everything from Lecture 7 still applies. Adam with weight decay. zero_grad() every batch. shuffle=False. Split by time, not at random.

And note what AdaptiveAvgPool1d(1) does: it averages across the entire time axis, collapsing it to a single number per channel.

Watch what that costs on the next slide.

8.5.3 Does It Actually Work?

Fitted on our weekly panel — 26-week windows, four channels, 30 store-category series, the same test block as Homework 4.

Model	Test RMSE	Parameters
LASSO (Homework 4)	744	46
XGBoost (Homework 4)	781	—
Random forest (Homework 3)	899	—
1D CNN, flatten head	1,032	8,161
1D CNN, global max pool	1,418	6,945
1D CNN, global average pool	1,534	6,945
Seasonal naive	2,152	0

Every CNN variant loses to a 46-coefficient linear model — a seven-week receptive field cannot see the annual lag LASSO was handed directly.

And swapping average pooling for Flatten, which keeps where each pattern occurred, is worth about 500 RMSE — three seeds each, against a spread of about 80.

Same weekly panel. Twenty-six-week windows, four channels, thirty store-category series, the same test block as Homework 4 — so these numbers are directly comparable to work you’ve already done. LASSO, with forty-six engineered features: seven forty-four. XGBoost: seven eighty-one. Random forest: eight ninety-nine.

Then the CNNs. Flatten head: one thousand thirty-two. Global max pool: fourteen eighteen. Global average pool: fifteen thirty-four. Seasonal naive, at the bottom: twenty-one fifty-two.

Two things to take from this.

First — every CNN variant loses to a forty-six-coefficient linear model. Not narrowly. And now you know why: a seven-week receptive field cannot see the annual lag that LASSO was simply handed as a column. The architecture’s assumption about locality is wrong for this data. Second, and this is the one I promised you.

Swapping average pooling for Flatten — which keeps where each pattern occurred, instead of averaging it away — is worth about five hundred RMSE. Three seeds each, against a run-to-run spread of roughly eighty. That gap is real. That’s the objection from slide 7, measured. Pooling threw away position, and position was carrying signal.

And I want to be careful about how far to push that conclusion, because there’s a tempting overreach here.

This does not mean pooling is bad. It means pooling encodes an assumption — that position is irrelevant — and on this dataset that assumption is false. Run the same experiment on image classification and the sign would flip, decisively.

That’s the shape of nearly every result in this course. There is no ranking of methods that holds across problems. There’s a match, or a mismatch, between what the method assumes and what the data is. Your job is to know what your method assumes — which is much harder than knowing how to call it, and much more valuable.

8.6 Key Takeaways

8.6.1 CNN Architecture Summary

Architecture	Year	The idea that survived
LeNet-5	1998	Conv–pool–conv–pool–dense: the template
VGG	2014	Only \(3\times3\) filters, stacked deep
Inception	2014	Parallel filter sizes; \(1\times1\) to cut channels
ResNet	2015	Skip connections make real depth trainable
1D CNN	—	The same machinery along a time axis

Reach for a 1D CNN when local shape matters more than exact position, when you have many parallel series, or as a cheap feature extractor feeding something else.

Do not reach for one when the dominant signal is at a long lag, when you have well engineered tabular features already, or when a stakeholder needs to read the model. On the evidence above, that covers most of what we forecast in this course.

The lineage in one table. LeNet gave us the template. VGG: small filters, stacked deep. Inception: parallel filter sizes, and one-by-ones to control cost. ResNet: skip connections, which made real depth trainable. And the 1D CNN, which is that same machinery run along a time axis.

Reach for a 1D CNN when local shape matters more than exact position, when you have many parallel series, or as a cheap feature extractor that feeds something else.

That third use is underrated, incidentally. You don’t have to use a CNN as the whole model. A common production pattern is to let convolutions compress a long raw window into a handful of learned summary features, then feed those into gradient boosting alongside your engineered columns. You get the pattern discovery without betting the forecast on it.

Do not reach for one when the dominant signal is at a long lag, when you already have well engineered tabular features, or when a stakeholder needs to read the model and understand it.

And be honest about what that second list covers. On the evidence we just saw, it describes most of what we forecast in this course. Which raises the obvious question: why teach it, then?

Because the receptive field problem is not a fact about CNNs, it’s a fact about this data — a strong annual cycle and a short window. Change either one and the answer changes. And because convolution is a component, not just a model: it shows up inside architectures you’ll meet later, in production forecasting systems, and in every paper on the subject. Knowing what it assumes is what lets you predict, before fitting anything, whether it will work on a problem you haven’t seen yet.

That prediction — made in advance, from structure rather than from trying it — is the skill this course is actually training.

8.6.2 Lecture 8: Key Takeaways

1. Convolution is local connectivity plus weight sharing. A pattern is learned once and detected everywhere, at a fraction of the parameters.

2. Pooling buys translation invariance — which an image classifier wants and a forecaster usually does not. Measured cost here: about 500 RMSE.

3. The architecture history reduces to three transferable ideas: small filters stacked deep, skip connections for trainable depth, and learned rather than hand-engineered features.

4. A 1D CNN takes a window across channels and predicts the next step. Its weakness is the receptive field: seven weeks of context cannot see an annual lag.

5. On our panel every CNN variant lost to a 46-coefficient LASSO. Architecture has to fit the problem; it is not a ladder you climb.

Next: RNNs and LSTMs — architectures built to carry state through time, which is what the receptive field could not do.

One. Convolution is local connectivity plus weight sharing. A pattern is learned once and detected everywhere, at a fraction of the parameters.

Two. Pooling buys translation invariance — which an image classifier wants and a forecaster usually does not. Measured cost on our data: about five hundred RMSE.

Three. The architecture history reduces to three transferable ideas: small filters stacked deep, skip connections for trainable depth, and learned rather than hand-engineered features.

Four. A 1D CNN takes a window across channels and predicts the next step. Its weakness is the receptive field — seven weeks of context cannot see an annual lag.

And five. On our panel, every CNN variant lost to a forty-six-coefficient LASSO. Architecture has to fit the problem. It is not a ladder you climb.

Next time: RNNs and LSTMs. Architectures built to carry state through time — which is precisely what the receptive field could not do.

8.6.3 References