Two Problems Built Into Recurrence

Part 1’s forecaster reads strictly left to right. Follow one week, \(t-52\), to the forecast:

The information bottleneck: week \(t-52\) must survive 52 squeezes through one 64-number state.
The sequential bottleneck: each update waits for the last, however many GPUs you own.

Scaled Dot-Product Attention (Vaswani et al. 2017)

\[\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V\]

Each week’s query is scored against every week’s key, and the scores weight the values; \(Q\), \(K\), \(V\) are learned projections of the same weeks, \(d_k\) the key length.

The score is the query’s length times the key’s shadow on it. Softmax turns the scores into positive weights that sum to one, so the output stays inside the values. Attention averages; it cannot extrapolate. The \(\sqrt{d_k}\) keeps the softmax from saturating.

What That Buys a Forecaster

Attention can weight any week in the window in one step; an RNN needs 52 updates to reach last year. Where our trained model actually looked, over the test year:

Mostly the last two to seven weeks; in one of two training runs, one head found last year’s week. The weights are learned, not specified — and they show where the model looked, not what mattered.

Multi-Head Attention

One attention operation gives one set of weights — one notion of what is relevant. Multi-head attention runs \(h\) of them in parallel on different learned projections, then concatenates:

\[\text{MultiHead}(Q,K,V) = \text{concat}(\text{head}_1, \ldots, \text{head}_h)\, W^O\]

Each head can specialize. Ours did: most watch the last few weeks, one spreads its weight evenly, and in one run one found the annual lag.

The cost is a surprise. Each head works in \(d_{\text{model}}/h\) dimensions, so eight heads cost about the same as one full-width head.

Our model: nhead=4, d_model=64 — four heads of 16 dimensions.

The Encoder Block

A Transformer encoder layer is four components in a fixed arrangement:

Stack \(N\) of those and you have the encoder. Nothing in the block is new to you except attention itself — the skip connections are ResNet’s, LayerNorm rescales each week’s vector, and the feed-forward sublayer is the Lecture 7 network applied to every week.

One thing is conspicuously missing, and it is not an oversight. Nothing above depends on the order of the input.

Positional Encoding: Attention Cannot Tell Time

Attention ignores order: shuffle the window and each week’s output moves with it; the rest is an unordered bag. A forecaster that cannot tell two weeks ago from thirty is not forecasting.

The fix adds 32 pairs \((\sin \omega_i t,\ \cos \omega_i t)\), \(\omega_i = 10000^{-2i/d}\), to each week — each pair a clock hand turning at its own speed:

Shifting \(\Delta\) weeks turns each hand by a fixed angle, so similarity depends only on distance. We measure what removing it costs at the end.

Why Transformers Took Over

Each cell is one query–key score: \(O(T^2)\) time and memory, fine at \(T = 52\), painful at \(T = 10{,}000\).

Transformer Encoder in PyTorch

layer = nn.TransformerEncoderLayer(d_model=64, nhead=4,
          dim_feedforward=128, dropout=0.1, batch_first=True)
self.enc = nn.TransformerEncoder(layer, num_layers=2)
self.inp = nn.Linear(n_features, 64)
self.fc  = nn.Linear(64, 1)

def forward(self, x):
    h = self.inp(x) + self.pe      # position must be added
    return self.fc(self.enc(h)[:, -1, :]).squeeze(-1)

nn.TransformerEncoderLayer gives you the whole block — but not the positional encoding. You add that yourself, and forgetting is a silent failure.

Multi-step forecasting also needs a causal mask.

Does It Work?

Same weekly panel and test block as everything since Homework 3; every sequence model reads 52-week windows (Part 1’s headline 842 and 987 were at 26 weeks).

The Transformer beats the CNN and ties the LSTM — 990 against 998, a difference far smaller than the run-to-run spread. It loses to a vanilla RNN holding one-fifteenth as many parameters, and to a 46-coefficient linear model.

Reading That Result Honestly

Every advantage on the “why Transformers took over” slide is an advantage at scale, and this problem has none of the scale.

And positional encoding is doing real work, which we can measure. Remove it and the model still knows the last week, where it reads out, but sees the other 51 as an unordered bag:

292 RMSE — dropping it below even the 1D CNN. That gap is what the order of those 51 weeks is worth, and the reason the sinusoids exist.

Sequence Architecture Comparison

Choose by the structure of your problem, not by recency of the architecture. Short window and recent-lag signal: a plain RNN, or honestly a linear model. Long sequences, abundant data, dependencies at arbitrary distance: a Transformer, and nothing else comes close.

That second case is language, which is why this architecture reshaped that field and has been slower to displace gradient boosting in tabular forecasting — where the M5 competition was won by gradient-boosted trees, Lecture 5’s method.

Lecture 9, Part 2: Key Takeaways

Next: Bayesian Statistics I — a different answer to what a forecast should tell you.