9 RNNs and LSTMs

Lecture 9, Part 1 of ECON 8310.

9.1 Lecture Outline

Where the Last Two Lectures Ran Out
The Vanilla RNN
LSTM: Long Short-Term Memory
In PyTorch, and Does It Work
Key Takeaways

Five parts. First, precisely where Lectures 7 and 8 ran out of road — because the limitation is specific and it’s worth naming. Then the vanilla RNN, which is one equation applied repeatedly. Then why that equation fails on long dependencies, which turns out to be a mathematical fact rather than a tuning problem. Then the LSTM, built to fix exactly that. And finally PyTorch, and an honest scoreboard.

9.2 Where the Last Two Lectures Ran Out

Two architectures, two different ways of failing to see time.

9.2.1 Two Architectures That Cannot See Time

Lecture 7 gave you a feedforward network. It treats each row independently — everything it knows about when something happened, you encoded by hand as a lag column. Shuffle the features and the model is unchanged.

Lecture 8 gave you a 1D CNN, which does see local shape. But its receptive field is fixed by architecture: stacked width-5 and width-3 filters reach about seven weeks back, and no reasonable stack of small filters reaches an annual lag. Worse, pooling deliberately discarded where in the window each pattern occurred — and we measured that costing about 500 RMSE.

Both architectures share one assumption: the input is a fixed-size block to be processed all at once. Neither has any notion of reading a sequence in order and carrying something forward.

That is the gap today closes.

Let’s be precise about what’s missing, because “doesn’t understand time” is too vague to act on.

Lecture 7’s feedforward network treats each row independently. Everything it knows about when something happened, you encoded by hand as a lag column. Shuffle the feature columns and the model is unchanged. Time is not in the architecture; it’s in your feature engineering.

Lecture 8’s CNN does better — it genuinely sees local shape. But its receptive field is fixed by the architecture. Stacked width-five and width-three filters reach about seven weeks back, and no reasonable stack of small filters reaches an annual lag.

And worse, pooling deliberately discarded where in the window each pattern occurred. We measured that costing about five hundred RMSE.

Here’s the assumption both of them share, and it’s the one we’re breaking today.

Both treat the input as a fixed-size block, to be processed all at once. Neither has any notion of reading a sequence in order and carrying something forward from one step to the next. That’s the gap. Everything today closes it.

It’s worth noticing that we have been compensating for this gap all semester, quite successfully. Lag columns are a workaround. Rolling means are a workaround. The Fourier terms in Homework 4 are a workaround. Every one of them is you, the analyst, telling a memoryless model something about time that it cannot work out for itself.

And they work well — LASSO on forty-six of those workarounds is still leading our scoreboard. So this isn’t a story where the old approach was broken. It’s a story about whether the model can discover the temporal structure instead of being handed it. That’s the question today tests.

9.2.2 The Recurrent Idea: Carry State Forward

A recurrent network reads the sequence one step at a time and maintains a hidden state $\mathbf{h}_t$ — a running summary of everything seen so far.

At each step it does two things: update the summary using the new input and the previous summary, then optionally emit an output. The same weights are used at every step, which is weight sharing again — along time rather than along space.

The consequence worth sitting with. There is no fixed window. In principle $\mathbf{h}_t$ encodes the entire history $x_1, \ldots, x_t$, so the architecture imposes no limit on how far back the model can look.

In principle. Most of this lecture is about the gap between that promise and what a recurrent network actually delivers — and about the architecture built to close it.

The recurrent idea is simple enough to state in one sentence, and I’d like you to have it before any equations.

A recurrent network reads the sequence one step at a time, and maintains a hidden state — call it h-t — which is a running summary of everything it has seen so far.

At each step it does two things. It updates that summary, using the new observation and the previous summary. And optionally it emits an output.

Critically, the same weights are used at every step. That’s weight sharing again — but along time rather than along space, which is where the CNN shared them.

Now the consequence, and it’s a big claim.

There is no fixed window. In principle, h-t encodes the entire history — every observation from step one to step t. The architecture imposes no limit whatsoever on how far back the model can look. Compare that with the CNN’s seven weeks, which was baked in by the filter sizes. In principle.

Most of this lecture is about the gap between that promise and what a recurrent network actually delivers. And then about the architecture that was invented specifically to close it.

9.3 The Vanilla RNN

One equation, applied over and over — and the reason that is both the whole idea and the whole problem.

9.3.1 Vanilla RNN: One Equation, Repeated

The recurrence

\[\mathbf{h}_t = \tanh\!\left(\mathbf{W}_h \mathbf{h}_{t-1} + \mathbf{W}_x \mathbf{x}_t + \mathbf{b}\right), \qquad \hat{y}_t = \mathbf{W}_y \mathbf{h}_t + \mathbf{b}_y\]

$\mathbf{W}_h$, $\mathbf{W}_x$, $\mathbf{W}_y$ are the same matrices at every step.

Read the first equation in words: the new summary is a squashed mix of the old summary and the new observation. That is the entire model.

Because the weights are shared across steps, a 26-week window costs no more parameters than a 5-week one — and the same network handles sequences of different lengths without modification. Our RNN below holds about 4,500 parameters, against 22,657 for the small feedforward network in Lecture 7.

For forecasting we usually want one number, not a sequence, so we take the final hidden state $\mathbf{h}_T$ — the summary after reading the whole window — and put one linear layer on top of it.

Here’s the recurrence. The new hidden state h-t is tanh of: W-h times the old hidden state, plus W-x times the new input, plus a bias. And the output is a linear layer on top of the hidden state.

Read the first equation in words: the new summary is a squashed mix of the old summary and the new observation. That is the entire model. Everything else today is consequences of that line.

The three weight matrices are the same at every time step. Not one per step — one, reused. Two things follow from that sharing, and both are genuinely nice.

First, a twenty-six-week window costs no more parameters than a five-week one. Compare the CNN, where a longer window meant more filters or more layers. Here, window length is free.

Second, the same trained network handles sequences of different lengths without modification. You can train on twenty-six weeks and run it on forty. Nothing in the architecture cares.

Our RNN below holds about four and a half thousand parameters. Lecture 7’s small feedforward network held twenty-two thousand six hundred fifty-seven. This is a much smaller model.

One practical note. For forecasting we usually want a single number, not a sequence of outputs. So we take the final hidden state — the summary after reading the whole window — and put one linear layer on top of it. That’s the pattern you’ll write in Homework 5.

A word on the hidden size, since it’s the main thing you’ll tune. It’s how wide that running summary is — how much the model is allowed to carry forward. Sixty-four is a sensible default for our data. Too small and the summary can’t hold enough to be useful; too large and you’re back in the overfitting regime from Lecture 7, with far more parameters than observations.

And notice the tanh in the recurrence isn’t decorative. It squashes the state to between minus one and one at every step, which keeps the running summary from growing without bound as the sequence gets longer. Remove it and the state can drift off to enormous values. It’s also, as we’re about to see, part of why the gradient dies.

9.3.2 Training It: Backpropagation Through Time

Training an RNN is ordinary backpropagation applied to an unusual graph. Unroll the recurrence across the window and it becomes a 26-layer feedforward network — one layer per time step — in which every layer shares the same weights.

Gradients then flow backward from the loss at step $T$ all the way to step 1. Because $\mathbf{W}_h$ appears at every step, its gradient is a sum of contributions from all 26 positions.

This is where it breaks. Propagating a gradient back through $k$ steps multiplies by roughly $\mathbf{W}_h$ $k$ times. If the relevant factor is below 1, the gradient shrinks exponentially; above 1, it explodes. Either way, long-range signal does not survive the trip.

Exploding gradients have an easy fix — clip_grad_norm_ caps the size of the update, and we use it. Vanishing gradients do not, which is what the next section is for.

Training an RNN is ordinary backpropagation applied to an unusual graph, and the trick is to see the graph correctly.

Unroll the recurrence across the window. Twenty-six time steps becomes a twenty-six-layer feedforward network — one layer per step — in which every layer shares the same weights.

Once you see it that way, there’s nothing new. Gradients flow backward from the loss at the final step all the way to step one. Because W-h appears at every step, its gradient is a sum of contributions from all twenty-six positions. And this is where it breaks.

Propagating a gradient back through k steps multiplies it by roughly W-h, k times over. If the relevant factor is below one, the gradient shrinks exponentially as it travels. If it’s above one, it explodes. Either way, long-range signal does not survive the trip.

The exploding case has an easy fix — clip_grad_norm_ caps the size of the update, and we use it in every recurrent model in this course. One line, and the problem is gone.

The vanishing case does not have an easy fix. That’s what the next slide quantifies, and what the whole second half of the lecture is about.

9.3.3 Why a Vanilla RNN Forgets

Put numbers on it. Suppose each backward step multiplies the gradient by a factor of $0.7$ — an unremarkable value for a $\tanh$ network [@Bengio1994].

Steps back	Gradient factor	Meaning
1	$0.70$	full signal
5	$0.17$	weakened but usable
10	$0.028$	barely present
26	$0.000\,002$	gone
52	$\approx 10^{-9}$	the annual lag, invisible

The promise from two slides ago fails in practice. A vanilla RNN can in principle encode arbitrarily long history, but the gradient that would teach it to do so has vanished long before it reaches back that far. In practice these networks learn dependencies of roughly ten steps.

Bengio and colleagues proved this is not an optimization bug to be tuned away — it is inherent to the architecture. Which is why the fix had to be a different architecture.

Let’s put numbers on it, because “the gradient shrinks” is not vivid enough to change anyone’s behavior.

Suppose each backward step multiplies the gradient by a factor of zero-point-seven. That’s an unremarkable value for a tanh network — nothing pathological. One step back: zero-point-seven. Full signal. Five steps: zero-point-one-seven. Weakened, but usable. Ten steps: zero-point-zero-two-eight. Barely present. Twenty-six steps — our window length — two millionths. Gone.

Fifty-two steps, which is the annual lag: about ten to the minus nine. Invisible. There is no optimizer, no learning rate, no amount of training that recovers a signal that small.

So the promise from slide five fails in practice. A vanilla RNN can, in principle, encode arbitrarily long history. But the gradient that would teach it to do so has vanished long before it reaches back that far. In practice these networks learn dependencies of roughly ten steps. Not fifty-two.

And I want to stress one thing about the Bengio result. This is not an optimization bug that better tuning fixes. It was proved to be inherent to the architecture. Which is why the fix had to be a different architecture, and why the next section exists.

9.4 LSTM: Long Short-Term Memory

Add a memory that information passes through rather than being recomputed at every step.

Add a memory that information passes through, rather than being recomputed at every step.

9.4.1 LSTM: A Cell State and Three Gates

An LSTM carries two things forward: a cell state $\mathbf{c}_t$, which is the long-term memory, and a hidden state $\mathbf{h}_t$, which is what the rest of the network sees. Three gates control the traffic between them [@Hochreiter1997].

Forget gate	$\mathbf{f}_t = \sigma(\mathbf{W}_f[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_f)$ — what fraction of the old memory to keep
Input gate	$\mathbf{i}_t = \sigma(\mathbf{W}_i[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_i)$ — how much of the new candidate to write
Output gate	$\mathbf{o}_t = \sigma(\mathbf{W}_o[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_o)$ — how much of memory to expose

The memory update itself is the line that matters:

\[\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} \;+\; \mathbf{i}_t \odot \tilde{\mathbf{c}}_t, \qquad \mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)\]

Each gate is a sigmoid, so it outputs between 0 and 1 — a soft switch the network learns.

An LSTM carries two things forward instead of one.

A cell state, c-t — that’s the long-term memory. And a hidden state, h-t — that’s what the rest of the network actually sees. Three gates control the traffic between them.

The forget gate decides what fraction of the old memory to keep. The input gate decides how much of the new candidate value to write in. The output gate decides how much of the memory to expose to the outside world.

Each gate is a sigmoid, so it outputs a number between zero and one. Think of it as a soft switch — not open or closed, but a dial, and the network learns where to set it.

Now this line is the one that matters. Everything else is bookkeeping.

The new cell state equals the forget gate times the old cell state, plus the input gate times the new candidate.

Note the operations. There’s an element-wise multiply by a number between zero and one — that’s the forgetting. And then there’s an addition.

Keep your eye on that plus sign. Two slides from now it will turn out to be the whole reason the architecture works.

One more thing before we move on: notice what the gates are functions of. Each one takes the previous hidden state and the current input, runs them through a weight matrix, and squashes with a sigmoid.

So the gates are learned and input-dependent. The network is not applying a fixed forgetting rate. It decides, at every single time step, based on what it’s currently looking at, how much to forget and how much to write.

That’s a genuinely different kind of model from anything we’ve built. A LASSO coefficient is the same number for every observation. An LSTM gate is recomputed at every step of every sequence. That flexibility is the appeal — and, as the scoreboard will show, it’s also capacity you have to pay for whether or not you need it.

9.4.2 What the Gates Are Doing, in Plain Terms

The equations are compact but the behavior is intuitive. Take a store forecasting demand week by week.

Forget	“The promotion that ran three months ago no longer explains anything — drop it.” The gate closes toward 0 and that part of memory decays away.
Input	“This week is Thanksgiving and the spike is enormous — record it.” The gate opens toward 1 and the event is written into the cell.
Output	“I am holding a memory of last December, but this week is April — do not act on it yet.” Information stays in $\mathbf{c}_t$ without contaminating the current prediction.

The output gate is the one people skip, and it is the subtle one. It separates what the model remembers from what the model uses right now. A vanilla RNN cannot make that distinction — its single hidden state has to serve both jobs at once, so anything it stores is immediately in play.

The equations are compact, but the behavior is genuinely intuitive. Let me put it in the language of a store forecasting demand week by week.

The forget gate says: “the promotion that ran three months ago no longer explains anything — drop it.” The gate closes toward zero, and that part of memory decays away.

The input gate says: “this week is Thanksgiving and the spike is enormous — record it.” The gate opens toward one, and the event is written into the cell.

The output gate says: “I’m holding a memory of last December, but this week is April — don’t act on it yet.” The information stays in the cell state without contaminating this week’s prediction. The output gate is the one people skip when they explain this, and it’s the subtle one.

It separates what the model remembers from what the model uses right now. Those are different things, and a vanilla RNN cannot make the distinction — it has one hidden state, which has to serve both jobs at once. Anything an RNN stores is immediately in play, affecting the very next prediction.

The LSTM can hold something in reserve. That’s a genuine capability, and it’s why these models work well on language, where a fact mentioned in sentence one might not matter again until sentence forty.

9.4.3 Why the Cell State Fixes the Gradient

Look again at the memory update, and specifically at the operation in the middle:

\[\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t\]

The old memory enters additively, and it is not passed through a weight matrix or a $\tanh$ on the way. When the forget gate is near 1, $\mathbf{c}_t \approx \mathbf{c}_{t-1}$ plus whatever is new — and the gradient flows backward through that addition essentially undiminished.

Compare with the vanilla RNN, where every backward step multiplies by $\mathbf{W}_h$ and the $\tanh$ derivative. Repeated multiplication shrinks; repeated addition does not.

You have seen this fix before. It is the ResNet skip connection from Lecture 8 — $\mathbf{y} = \mathcal{F}(\mathbf{x}) + \mathbf{x}$ — solving the same problem in the same way. Give the gradient an additive path home and depth stops destroying it.

LSTM (1997) predates ResNet (2015) by eighteen years — the idea was rediscovered for depth after being invented for time.

Back to that plus sign.

Look at how the old memory enters. It’s additive. And notice what is not happening to it: it is not passed through a weight matrix, and it is not passed through a tanh, on its way from c-t minus one to c-t.

When the forget gate sits near one, c-t is approximately c-t minus one, plus whatever’s new. And a gradient flowing backward through that addition passes essentially undiminished.

Compare the vanilla RNN, where every single backward step multiplies by W-h and by the derivative of tanh. Repeated multiplication shrinks. Repeated addition does not. And you have seen this fix before — last week.

It is the ResNet skip connection. y equals F of x, plus x. Same problem: gradients dying as they travel back through depth. Same solution: give the gradient an additive path home.

Here’s the detail I enjoy. The LSTM is from 1997. ResNet is from 2015. The idea was invented for time, forgotten by the computer vision community, and then rediscovered eighteen years later for depth.

Same mathematics, two fields, two decades apart. Which is a decent argument for understanding mechanisms rather than memorizing architectures — the mechanism transferred, the architecture didn’t.

9.5 In PyTorch, and Does It Work

Four lines of model code, and an honest measurement against everything else this course has fitted.

9.5.1 LSTM for Forecasting in PyTorch

Window in, one forecast out

class LSTMForecaster(nn.Module):
    def __init__(self, n_features, hidden=64, layers=1):
        super().__init__()
        self.lstm = nn.LSTM(n_features, hidden, num_layers=layers,
                            batch_first=True)
        self.fc = nn.Linear(hidden, 1)

    def forward(self, x):        # x: (batch, seq_len, n_features)
        out, (h, c) = self.lstm(x)
        return self.fc(out[:, -1, :]).squeeze(-1)

batch_first=True gives shape (batch, seq_len, features) — the opposite of Conv1d. out[:, -1, :] is the state after the last step. Lecture 7’s rules still hold, plus clip_grad_norm_(..., 1.0).

The good news: after all that machinery, the code is short. nn.LSTM gives you the entire thing — all three gates, the cell state, the whole update.

You define the LSTM layer, you define a linear layer on top, and in forward you run the sequence through and take the last time step. Two things that will cost you time if you don’t know them.

batch_first=True gives you shape (batch, seq_len, features). That is the opposite of what Conv1d wanted last week, which was (batch, channels, time). If you’re moving code between the two — which Homework 5 asks you to do — this is where you’ll get a shape error.

And out[:, -1, :] is the hidden state after the last step. That’s the summary of the whole window, which is what we put the linear layer on.

Everything from Lecture 7 still applies: Adam, weight decay, zero_grad(), shuffle=False, split by time. Plus one addition — clip_grad_norm_ with a max norm of one, to handle the exploding gradient case from slide 8.

I want to flag how little of the theory shows up in the code, because it cuts both ways.

Everything from the last three slides — the cell state, the three gates, the additive update, the whole reason this architecture exists — is inside nn.LSTM. One line. You could use this model having understood none of it.

That’s the appeal of the tooling, and it’s also the risk. If you don’t know that gating solves long-range forgetting, you have no way to predict in advance that it will do nothing on a twenty-six-week window. You’d just fit it, see a worse number, and have no account of why. The measurement on the next slide is only interpretable because we spent twenty minutes on the mechanism.

9.5.2 Does It Work? The Scoreboard So Far

Same weekly panel, same 26-week windows, same test block as Homework 4.

Model	Test RMSE	Parameters
LASSO, 46 engineered features	744	46
XGBoost, 46 engineered features	781	—
Vanilla RNN, 4 raw channels	842	4,545
Random forest	899	—
LSTM, 4 raw channels	987	17,985
1D CNN, best variant	1,032	8,161
Seasonal naive	2,152	0

Recurrence beats convolution decisively — and it did so on four raw channels with no lag engineering at all, against 46 hand-built features for LASSO. That is the case for reading a sequence in order, in one line.

And the LSTM lost to the vanilla RNN — at a fixed 30-epoch budget. That is not a typo, and the next three slides are about what it does and does not mean.

Same weekly panel, same twenty-six-week windows, same test block as Homework 4. Everything here is directly comparable.

LASSO with forty-six engineered features: seven forty-four. XGBoost: seven eighty-one. Then the vanilla RNN at eight forty-two, on four thousand five hundred parameters. Random forest: eight ninety-nine. The LSTM at nine eighty-seven, on eighteen thousand parameters. The best CNN variant at one thousand thirty-two. Seasonal naive at the bottom, twenty-one fifty-two. Two readings, and the first is genuinely good news for recurrence.

Recurrence beats convolution decisively — eight forty-two against one thousand thirty-two. And it did that on four raw channels, with no lag engineering at all, against forty-six hand-built features for LASSO. That is a real argument for reading a sequence in order, and it’s the strongest result deep learning gets on our data. And the second reading. The LSTM lost to the vanilla RNN — at a fixed thirty-epoch budget.

That is not a typo. Hold onto the qualifier, though, because the next three slides are about what that line does and does not mean.

9.5.3 Why the Gated Model Lost at 30 Epochs

The result holds at all three window lengths. That rules out a fluke of one window — though all three rows share one panel and one split, so it is one finding measured three ways, not three independent tests:

Window	Vanilla RNN	LSTM
13 weeks	878	990	RNN wins
26 weeks	842	987	RNN wins
52 weeks	805	998	RNN wins

An LSTM solves a problem this data does not have. Gating preserves information across long dependency chains, and weekly demand is dominated by the recent past.

The gates are not free: 17,985 parameters against 4,545. But before concluding that they bought nothing, a table like this one owes two checks — was the comparison fair, and what in the data explains it. The next two slides run both.

A margin measured at one training budget is a claim about that budget. We chose 30 epochs because it is the default, not because it was shown to be enough for both models.

Thirteen weeks: RNN eight seventy-eight, LSTM nine ninety. Twenty-six weeks: eight forty-two against nine eighty-seven. Fifty-two weeks: eight-oh-five against nine ninety-eight. The RNN wins at all three window lengths.

Now, be careful about how much that buys us. Three windows rules out a fluke of one window. It does not rule out much else — all three rows are the same thirty series, the same split, the same recency structure. It is one finding measured three ways, not three independent tests. Say that out loud to the room; it is the kind of distinction that separates a result from a slogan.

The natural reading is: an LSTM solves a problem this data does not have. Gating preserves information across long dependency chains — that is what it was invented for — and weekly demand is dominated by the recent past. And the gates are not free: eighteen thousand parameters against four and a half thousand.

But before we sign off on that, notice what we have not checked. Both models trained for thirty epochs, because thirty is what the training function defaults to. Nobody showed that thirty epochs is enough for both of them — and one of them has four times the parameters. A margin measured at one training budget is a claim about that budget.

So the next two slides do the two things this table owes us: was the comparison fair, and what in the data explains it.

9.5.4 Check 1: Was It a Fair Fight?

An LSTM carries four times the parameters. Bigger models converge slower. So vary the training budget and watch the margin, at the 26-week window:

Epochs	10	30	60	100
Vanilla RNN	1,099	842	828	830
LSTM	1,002	987	926	821
Gap (LSTM − RNN)	−97	+145	+98	−9

The gap is largest at exactly the budget we used, and it closes at every checkpoint after it. The RNN is flat from about epoch 45; the LSTM is still improving at 100 — we ran out of patience, not the model.

At a fixed 30-epoch budget the RNN wins by a wide margin. At convergence the two tie. The RNN’s genuine advantage is that it gets there roughly three times faster on a quarter of the parameters — an optimization advantage, not a verdict on gating.

Same two models, twenty-six-week window. This time we vary the training budget and watch the margin.

At ten epochs, the LSTM is ahead by ninety-seven. At thirty — our setting — the RNN is ahead by a hundred and forty-five. At sixty, ninety-eight. At a hundred, the gap is minus nine.

Read the shape of it. The gap is largest at exactly the budget we chose. Not near the largest: the largest anywhere in the range we looked at.

And look at the curves separately, because they are different shapes. The RNN is finished by about epoch forty-five — eight twenty-eight, eight thirty-three, eight thirty. Flat. The LSTM is still coming down at a hundred: nine eighty-seven, nine twenty-six, eight twenty-one. It has not turned over. We ran out of patience, not the model. So the honest verdict is not “the LSTM loses.” It is three things.

At a fixed thirty-epoch budget, the RNN wins by a wide margin. Given enough epochs, the two tie. And the RNN gets to its best answer roughly three times faster on a quarter of the parameters — which is a real advantage, and worth having, and is an optimization advantage rather than a verdict about gating.

I want to be direct about what happened here. The first version of this slide deck did not have this check, and the previous slide’s table was presented as an architecture result. It is not one. It took varying a parameter nobody thinks of as part of the experiment to find that out — which is the same lesson as the seeds in Lecture 7, one level up.

9.5.5 Check 2: What in the Data Explains It

Gating pays when a value must survive many steps that would otherwise overwrite it. Three measurements say this panel has no such structure.

1. Most variance is not sequential. 91% of the variation in weekly units is between series — which store, which category — and 9% is within a series over time. The main task is reading the level, and the last few weeks give you that.

2. The series are smooth. Within-series autocorrelation is 0.83 at lag 1 and still 0.43 at lag 52. That is why long memory is unnecessary rather than why it is needed: the recent past is a sufficient statistic for the distant past. Nothing must be carried across a gap, because the gap was filled in continuously.

3. Nothing in the inputs arrives early. snap_days correlates +0.14 with units in the same week and ≈ 0 at every lead; event_days is flat everywhere. No announcement structure — and that is precisely what a forget gate is for.

Ask these three of your own series before reaching for a gated model: how much of my variance is cross-sectional? Is my series smooth enough that recent values already summarize old ones? Does anything in my inputs lead the target?

Second check. Gating pays off when a value has to survive many steps that would otherwise overwrite it. Does this panel have anything like that? Three measurements say no.

One. Most of the variance is not sequential at all. Ninety-one percent of the variation in weekly units is between series — which store, which category — and nine percent is within a series over time. The main thing a model has to get right is the level of the series in front of it, and the last few weeks tell you that. No memory mechanism required.

Two. The series are smooth — and this one is counter-intuitive, so take it slowly. Within-series autocorrelation is point eight three at lag one, and still point four three at lag fifty-two. That looks like long memory, and it is actually the reason long memory is unnecessary. Because neighboring weeks are nearly the same number, the recent past is a sufficient statistic for the distant past. Nothing has to be carried across the gap, because the gap was filled in continuously along the way.

Gating earns its keep when something must survive steps that would overwrite it. Here nothing overwrites anything.

Three. Nothing in the inputs arrives early. SNAP days correlate about point one four with units in the same week, and essentially zero at every lead. Event days are flat everywhere, including at zero. There is no announcement structure in this data — nothing shows up at time t that has to be stored until t plus k. That is the textbook use for a forget gate and an input gate, and this panel does not contain one instance of it.

And this is the part to carry out of the lecture — not the answer, the three questions. Before you reach for a gated model, or an attention model, on your own series: how much of my variance is cross-sectional rather than sequential? Is my series smooth enough that recent values already summarize the old ones? Does anything in my inputs lead the target? If the answers look like this panel’s, the extra machinery has nothing to do.

9.6 Key Takeaways

What recurrence buys, what it costs, and what is still missing.

9.6.1 RNN and LSTM in Context

Architecture	Sees order?	Effective memory	Cost
FFN on lags	no	whatever you engineered	cheap
1D CNN	local only	receptive field ($$7 steps)	cheap
Vanilla RNN	yes	$$10–30 steps in practice \| cheap \| \| LSTM \| yes \| hundreds of steps \| 4$$ an RNN

An LSTM holds roughly four times the parameters of a vanilla RNN at the same hidden size — each of the three gates plus the candidate needs its own weight matrix. That is the price of the memory, and on a short window you pay it for nothing.

What is still missing. An LSTM reads strictly left to right, one step at a time, so it cannot be parallelized across the sequence, and everything it knows about step 3 must survive being squeezed through every state between 3 and $T$. Part 2 removes both constraints.

Four architectures, three questions. Does it see order? How much can it actually remember? What does it cost?

The FFN on lags: no order, and it remembers whatever you engineered. The CNN: local order only, about seven steps. The vanilla RNN: yes to order, roughly ten to thirty steps in practice — note that’s practice, not the theoretical unlimited. The LSTM: yes, and hundreds of steps, at four times the cost.

An LSTM holds roughly four times the parameters of a vanilla RNN at the same hidden size, because each of the three gates plus the candidate needs its own weight matrix. That’s the price of the memory. On a short window you pay it for nothing.

If you go looking, you’ll also meet the GRU — the gated recurrent unit. It merges the forget and input gates into one and drops the separate cell state, so it sits between the two models on this table: more gating than a vanilla RNN, fewer parameters than an LSTM.

On data like ours, where the gates aren’t earning their keep anyway, a GRU often performs about like an LSTM at lower cost. It’s a reasonable thing to try in your final project if you go down this road — and a reasonable thing to skip, given what the scoreboard says about this whole family on this dataset. And here’s what’s still missing, because it motivates next week.

An LSTM reads strictly left to right, one step at a time. Two consequences. It cannot be parallelized across the sequence — step t needs step t minus one finished, so a thousand-step sequence takes a thousand sequential operations no matter how much hardware you own.

And everything it knows about step three has to survive being squeezed through every hidden state between three and the end. The gates make that survival more likely, but the information is still passing through a bottleneck at every step.

Part 2 removes both constraints, with a mechanism that lets every position look directly at every other position in a single operation.

9.6.2 Lecture 9 Part 1: Key Takeaways

A recurrent network reads a sequence in order and carries a hidden state. Weights are shared across time, so window length costs no parameters.
Training is backpropagation through time: unroll the recurrence and it is a deep network whose layers share weights. Repeated multiplication shrinks the gradient exponentially — at 52 steps, around $10^{-9}$.
An LSTM adds a cell state updated additively. Addition preserves gradients where multiplication destroys them — the ResNet fix, eighteen years earlier.
Three gates: forget what is stale, input what is new, output what is relevant now.
On our panel both recurrent models beat every CNN variant on four raw channels — and both still lose to a 46-coefficient LASSO.
The LSTM lost to the vanilla RNN at a 30-epoch budget, at all three window lengths — but the two tie once both are trained to convergence, and 91% of this panel’s variance is cross-sectional anyway. Gating solves long-range forgetting; this data does not suffer from it. Fit the architecture to the problem — and check that a margin survives the budget you measured it at.

Next week: attention and Transformers — reading the whole sequence at once.

One. A recurrent network reads a sequence in order and carries a hidden state. Weights are shared across time, so window length costs no parameters.

Two. Training is backpropagation through time — unroll the recurrence and it’s a deep network whose layers share weights. Repeated multiplication shrinks the gradient exponentially. At fifty-two steps, around ten to the minus nine.

Three. An LSTM adds a cell state updated additively. Addition preserves gradients where multiplication destroys them — the ResNet fix, eighteen years earlier. Four. Three gates: forget what’s stale, input what’s new, output what’s relevant now.

Five. On our panel, both recurrent models beat every CNN variant, on four raw channels. And both still lose to a forty-six-coefficient LASSO.

And six. The LSTM lost to the vanilla RNN at a thirty-epoch budget, at all three window lengths — and tied once both reached convergence. Gating solves long-range forgetting; a twenty-six-week window doesn’t suffer from it. So the gates bought four times the parameters and nothing else. Fit the architecture to the problem.

Next week: attention and Transformers. Reading the whole sequence at once.