Two Architectures That Cannot See Time

Lecture 7’s feedforward network treats each row independently: everything it knows about when, you encoded by hand as lag columns.

Lecture 8’s 1D CNN sees local shape, but each filter’s receptive field is about seven weeks (our two small kernels; depth or dilation would widen it), and pooling threw away where each pattern occurred — a measured 500 RMSE.

The Recurrent Idea: Carry State Forward

A recurrent network reads the sequence one step at a time, keeping a hidden state \(\mathbf{h}_t\): a running summary of everything so far. Each step updates the summary from the new input and the old summary, with the same weights every step — weight sharing again, along time instead of space.

The promise: no fixed window. In principle \(\mathbf{h}_t\) carries the whole history.

In principle. Most of this lecture is about the gap between that promise and what a recurrent network delivers — and the architecture built to close it.

Vanilla RNN: One Equation, Repeated

In words: the new summary is a squashed mix of the old summary and the new week. Shared weights mean a 26-week window costs no more parameters than a 5-week one — about 4,500 here, against 22,657 for Lecture 7’s network.

Training It: Backpropagation Through Time

Training is ordinary backpropagation on an unusual graph. Unroll the recurrence and it is a 26-layer network, one layer per week, every layer sharing the same weights. Gradients flow back from step \(T\) to step 1, and \(\mathbf{W}_h\)’s gradient sums contributions from all 26.

Exploding is fixed by clip_grad_norm_; vanishing is not — the next section.

Why a Vanilla RNN Forgets (Bengio et al. 1994)

Suppose each backward step multiplies the gradient by \(0.7\) — an unremarkable value for a \(\tanh\) network — and compare our own RNN, trained on 52-week windows, measured:

The same factor every step is a straight line on a log scale. Past the first few steps, ours keeps about \(0.79\) a step; measured, 0.4% of the gradient reaches 26 weeks back, 0.001% the same week last year. Bengio and colleagues proved this is inherent to the architecture.

LSTM: A Cell State and Three Gates

An LSTM carries two things forward: a cell state \(\mathbf{c}_t\), which is the long-term memory, and a hidden state \(\mathbf{h}_t\), which is what the rest of the network sees. Three gates control the traffic between them (Hochreiter and Schmidhuber 1997; Gers et al. 2000).

The memory update itself is the line that matters:

\[\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} \;+\; \mathbf{i}_t \odot \tilde{\mathbf{c}}_t, \qquad \mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)\]

Here \(\sigma\) is the sigmoid, so each gate outputs between 0 and 1 — a soft switch the network learns. \([\mathbf{h}_{t-1}, \mathbf{x}_t]\) stacks the two vectors; \(\odot\) multiplies element by element.

What the Gates Are Doing, in Plain Terms

The equations are compact but the behavior is intuitive. Take a store forecasting demand week by week.

The output gate is the one people skip, and it is the subtle one. It separates what the model remembers from what the model uses right now. A vanilla RNN cannot make that distinction — its single hidden state has to serve both jobs at once, so anything it stores is immediately in play.

Why the Cell State Fixes the Gradient

Follow a gradient backward through \(\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t\). Repeated multiplication shrinks; repeated addition does not.

On our panel the median forget gate is 0.52, so 7% survives four weeks. Check 2 says why.

LSTM for Forecasting in PyTorch

class LSTMForecaster(nn.Module):
    def __init__(self, n_features, hidden=64, layers=1):
        super().__init__()
        self.lstm = nn.LSTM(n_features, hidden, num_layers=layers,
                            batch_first=True)
        self.fc = nn.Linear(hidden, 1)

    def forward(self, x):        # x: (batch, seq_len, n_features)
        out, (h, c) = self.lstm(x)   # nn.RNN instead: out, h = self.rnn(x)
        return self.fc(out[:, -1, :]).squeeze(-1)

batch_first=True gives shape (batch, seq_len, features) — the opposite of Conv1d. out[:, -1, :] is the state after the last step. Lecture 7’s rules still hold, plus clip_grad_norm_(..., 1.0).

Does It Work? The Scoreboard So Far

Same weekly panel, same 26-week windows, same test block as Homework 4.

The vanilla RNN beats convolution decisively (842 against 1,032; the LSTM only edges it) — on four raw channels with no lag engineering at all, against 46 hand-built features for LASSO. That is the case for reading a sequence in order, in one line.

And the LSTM lost to the vanilla RNN — at a fixed 30-epoch budget. That is not a typo, and the next four slides are about what it does and does not mean.

Why the Gated Model Lost at 30 Epochs

All three windows share one panel and one split: one finding measured three ways, not three tests.

An LSTM solves a problem this data does not have: gating preserves information across long chains, and weekly demand is dominated by the recent past. The gates cost 17,985 parameters against 4,545. Two checks first: was the comparison fair, and what in the data explains it?

Check 1: Was It a Fair Fight?

Bigger models converge slower, and the LSTM has 4× the parameters. Vary the budget:

Check 2: What in the Data Explains It

Gating pays when a value must survive many steps that would otherwise overwrite it. Does this panel have that?

1. Most variance is not sequential. 91% is between series: the task is reading the level, and the last few weeks give you that.

2. History helps, but it arrives smoothly. A longer window still improves the RNN (878 → 805), but no value has to survive many steps untouched.

Check 2, Continued: Nothing Arrives Early

3. Nothing arrives early. snap_days correlates +0.14 with units in the same week. Its −0.13 and +0.12 two and four weeks ahead only echo that through the monthly SNAP calendar, and event_days is flat everywhere. No announcement structure — exactly what a forget gate is for.

RNN and LSTM in Context

An LSTM holds about four times an RNN’s parameters at the same hidden size: three gates and the candidate each need a weight matrix. On a short window you pay that for nothing.

Still missing: an LSTM reads strictly left to right, so it cannot be parallelized, and everything about step 3 must squeeze through every state between 3 and \(T\). Part 2 removes both.

Lecture 9, Part 1: Key Takeaways