10 Transformers

Lecture 9, Part 2 of ECON 8310.

10.1 Lecture Outline

What Recurrence Could Not Do
The Attention Mechanism
The Transformer Encoder
Transformers for Time Series
Key Takeaways

Five parts. What recurrence could not do — two specific constraints. Then attention itself, which is a weighted lookup and much less mysterious than the formula makes it look. Then the encoder block, which is attention plus three things you already know. Then the implementation and an honest measurement. And finally where all six sequence architectures sit relative to each other.

10.2 What Recurrence Could Not Do

Two constraints that are built into reading a sequence one step at a time.

10.2.1 Two Problems Built Into Recurrence

Part 1 ended with a working recurrent forecaster. Both of its remaining problems come from the same source — it reads strictly left to right.

The information bottleneck. Everything the model knows about step 3 must survive being squeezed through every hidden state between 3 and $T$. An LSTM’s gates make that survival more likely, but the whole history is still compressed into one fixed-size vector.

The sequential bottleneck. Step $t$ cannot be computed until step $t-1$ is finished. That is a hard constraint: a 1,000-step sequence takes 1,000 sequential operations no matter how many GPUs you own. Convolutions parallelize; recurrence does not.

Attention removes both at once. Let every position look directly at every other position, in one operation. No compression through intermediate states, and no ordering constraint on the computation.

Part 1 left us with a working recurrent forecaster. Both of its remaining problems come from the same source: it reads strictly left to right.

The information bottleneck. Everything the model knows about step three has to survive being squeezed through every hidden state between three and the end of the window. An LSTM’s gates make that survival more likely — that’s what they’re for — but the entire history is still being compressed into one fixed-size vector, over and over.

The sequential bottleneck. Step t cannot be computed until step t minus one is finished. That is a hard constraint, not an implementation detail. A thousand-step sequence takes a thousand sequential operations no matter how many GPUs you own. Convolutions parallelize beautifully; recurrence cannot.

Attention removes both at once, and the idea is almost aggressively simple: let every position look directly at every other position, in one operation.

No compression through intermediate states — position twenty-six reads position three directly. And no ordering constraint on the computation — all positions are computed simultaneously.

10.3 The Attention Mechanism

A learned, weighted lookup — which is a more useful way to read it than the formula suggests.

10.3.1 Scaled Dot-Product Attention

The whole mechanism

\[\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V\]

$Q$ = queries, $K$ = keys, $V$ = values, $d_k$ = key dimension.

Read it as a lookup table with soft matching [@Vaswani2017]. Each position emits a query: “what am I looking for?” Every position also offers a key: “here is what I have” — and a value: “here is what I would contribute.”

$QK^\top$ scores every query against every key. The softmax turns those scores into weights that sum to one. The output is a weighted average of the values, with more weight where query and key matched.

Why divide by $\sqrt{d_k}$? Dot products grow with dimension, and large inputs push the softmax toward one-hot, where its gradient nearly vanishes. The scaling keeps it trainable.

Here’s the formula. Attention of Q, K, V equals softmax of Q K-transpose over root d-k, times V. That’s the whole mechanism. Every Transformer in the world is that line, repeated.

Now let me translate it, because the formula makes it look more exotic than it is. Read it as a lookup table with soft matching.

Each position emits a query — that’s the “what am I looking for?” vector. Every position also offers a key, which says “here’s what I have.” And a value, which says “here’s what I would contribute if you pick me.”

So: Q times K-transpose scores every query against every key. That’s a big matrix of similarity scores — every position against every position. The softmax turns each row of those scores into weights that sum to one. And then you multiply by V, which gives you a weighted average of the values, with more weight wherever query and key matched well. That’s it. Score, normalize, average.

And it’s worth noticing you have built something very close to this already, by hand. In Homework 4, a rolling mean is a weighted average over recent positions — with weights you fixed in advance, equal across the window. A Fourier term is a weighted combination too, with weights set by a formula.

Attention is the same shape of operation with the weights learned, and recomputed for every input. That’s the whole upgrade. Not a new kind of mathematics — a new source for the numbers.

Why divide by root d-k? Because dot products grow with dimension. In high dimensions the raw scores get large, and large inputs push the softmax toward a one-hot distribution — all the weight on one position — where its gradient is nearly zero and nothing trains. Dividing by root d-k keeps the scores in a range where the softmax stays soft. It’s a numerical fix, not a conceptual one, but leave it out and the model doesn’t learn.

10.3.2 What That Buys a Forecaster

Concretely: predicting units for the first week of December, the model can place high weight on last December and on the surrounding weeks, and near-zero weight on an unremarkable Tuesday in March.

Two properties make that different from anything so far.

The weights are learned, not specified. You do not tell the model that lag 52 matters. Compare that with LASSO in Lecture 6, where you built lag52 by hand and the penalty decided whether to keep it.

The path is direct. Attention reaches week $t-52$ in one operation, where a recurrent model needs 52 successive state updates and a CNN needs a receptive field that wide. Distance costs attention nothing.

The price is quadratic: scoring every position against every other is $O(T^2)$ in time and memory — fine at $T = 26$, painful at $T = 10{,}000$.

Let me make that concrete for our problem.

You’re predicting units for the first week of December. With attention, the model can place high weight on last December, and on the weeks around it, and near-zero weight on an unremarkable Tuesday in March. It looks where the information is.

Two properties make that genuinely different from anything we’ve built.

The weights are learned, not specified. You do not tell the model that lag fifty-two matters. Contrast that with Lecture 6, where you built the lag52 column by hand and the LASSO penalty merely decided whether to keep it. Here the model discovers which positions to consult.

And the path is direct. Attention reaches week t minus fifty-two in one operation. A recurrent model needs fifty-two successive state updates to get there — and we saw last week what survives fifty-two multiplications. A CNN needs a receptive field that wide, which it doesn’t have. Distance costs attention nothing. That’s the headline.

There’s a third property worth naming, because it matters for how you’d sell this to a stakeholder. Attention weights are inspectable. You can pull out the matrix and look at which weeks the model consulted when it made a given forecast.

That is not the same as an explanation — a high attention weight tells you where the model looked, not why it decided what it decided, and there’s a real literature arguing about how much to read into them. But compared with an LSTM’s hidden state, which is sixty-four numbers with no interpretation whatsoever, it’s a meaningful step toward a model whose reasoning you can interrogate.

And here’s the price, which is real. Scoring every position against every other position is quadratic — order T-squared in both time and memory. At T equals twenty-six, that’s irrelevant. At T equals ten thousand, it is the central engineering constraint of the entire field, and there’s a whole literature on approximating it.

10.3.3 Multi-Head Attention

One attention operation produces one set of weights — one notion of what is relevant. That is limiting, because different kinds of relevance coexist.

Multi-head attention runs $h$ attention operations in parallel on different learned projections of the same input, then concatenates:

\[\text{MultiHead}(Q,K,V) = \text{concat}(\text{head}_1, \ldots, \text{head}_h)\, W^O\]

Each head can specialize. In a forecasting model one head might track weekly position, another the annual cycle, another the effect of promotional flags — all attending over the same window at once.

The cost is not what you would guess. Each head works in a lower-dimensional subspace of size $d_{\text{model}}/h$, so eight heads cost about the same as one head at full width. You buy the diversity nearly for free.

Our model below uses nhead=4 with d_model=64 — four heads of 16 dimensions each.

One attention operation produces one set of weights — one notion of what is relevant. That’s limiting, because different kinds of relevance coexist in the same sequence.

Multi-head attention runs h attention operations in parallel, on different learned projections of the same input, then concatenates the results.

Each head can specialize. In a forecasting model, one head might track position within the month, another the annual cycle, another the effect of promotional flags — all attending over the same window simultaneously. And the cost is not what you’d guess.

You might expect eight heads to cost eight times one head. They don’t. Each head works in a lower-dimensional subspace — d-model divided by h. So eight heads of sixteen dimensions each cost about the same as one head of one hundred twenty-eight.

You’re not adding capacity. You’re partitioning the capacity you already had, and letting different partitions specialize. The diversity is close to free, which is why nobody uses a single head. Our model below uses four heads with d-model of sixty-four — four heads of sixteen dimensions.

10.4 The Transformer Encoder

Attention, plus the three pieces that make it trainable.

10.4.1 The Encoder Block

A Transformer encoder layer is four components in a fixed arrangement.

Multi-head attention	Every position attends to every position.
Residual + LayerNorm	$\mathbf{z} = \text{LayerNorm}(\mathbf{x} + \text{MultiHead}(\mathbf{x}))$. The skip connection from Lecture 8, doing the same job.
Feed-forward sublayer	A small MLP applied to each position independently — the Lecture 7 network, at every step.
Residual + LayerNorm	Again, around the feed-forward block.

Stack $N$ of those and you have the encoder. Nothing in the block is new to you except attention itself — the residual connections are ResNet’s, and the feed-forward sublayer is the network from Lecture 7.

One thing is conspicuously missing, and it is not an oversight. Nothing above depends on the order of the input.

A Transformer encoder layer is four components in a fixed arrangement, and the good news is that you already know three of them. Multi-head attention — every position attends to every position. That’s the new part.

Residual plus LayerNorm — z equals LayerNorm of x plus MultiHead of x. That’s the ResNet skip connection from Lecture 8, doing exactly the same job: giving the gradient an additive path home.

A feed-forward sublayer — a small MLP applied to each position independently. That is literally the network from Lecture 7, run at every time step. Residual and LayerNorm again, this time around the feed-forward block.

Stack N of those and you have the encoder. Nothing in the block is new to you except attention itself.

That is genuinely worth pausing on. The architecture that reorganized machine learning is one new mechanism, wrapped in a residual connection from 2015 and a two-layer network from the 1980s.

Progress in this field often looks like that from the inside — not a wholesale reinvention, but one new component slotted into scaffolding that already worked. It’s a reason to learn components rather than model names.

And one thing is conspicuously missing. It is not an oversight, and I want you to spot it before I say it.

Look at every operation in that block. Attention: a weighted average over positions. Feed-forward: applied to each position independently. LayerNorm: per position. Residual: per position. Nothing in there depends on the order of the input.

10.4.2 Positional Encoding: Attention Cannot Tell Time

Attention computes a weighted average over positions. Averaging does not care about order — shuffle the input sequence and every attention output is unchanged.

Attention is permutation-invariant. For language that is fatal, and for forecasting it is worse: a model that cannot distinguish last week from thirty weeks ago is not forecasting at all.

The fix is to add position information to the input itself, before attention sees it. The original paper uses fixed sinusoids of different frequencies:

\[PE_{(t, 2i)} = \sin\!\left(\frac{t}{10000^{2i/d}}\right), \qquad PE_{(t, 2i+1)} = \cos\!\left(\frac{t}{10000^{2i/d}}\right)\]

Each position gets a distinct pattern across dimensions, and nearby positions get similar patterns — so the model can learn relative distance, not just absolute index.

We measure what removing it costs on the next slide but one.

Let’s make that precise, because it’s the most important slide in the lecture.

Attention computes a weighted average over positions. Averaging does not care about order. Shuffle the input sequence, and every attention output is unchanged. The technical term is that attention is permutation-invariant.

For language that’s fatal — “dog bites man” and “man bites dog” would be identical inputs. For forecasting it’s arguably worse. A model that cannot distinguish last week from thirty weeks ago is not forecasting at all; it’s computing a weighted average of an unordered bag of numbers. So the fix is to add position information to the input itself, before attention ever sees it.

The original paper uses fixed sinusoids at different frequencies — sines on the even dimensions, cosines on the odd ones, with wavelengths that grow geometrically.

Two properties make that work. Each position gets a distinct pattern across dimensions, so the model can tell positions apart. And nearby positions get similar patterns, so the model can learn relative distance rather than just absolute index. You can also learn the position vectors instead of fixing them; both are used in practice. We measure exactly what removing this costs, on the slide after next. Hold the question.

10.4.3 Why Transformers Took Over

Three properties, in the order they mattered commercially.

Parallelism. Every position is computed at once, so training scales across hardware in a way recurrence never could. This, more than accuracy, is why the architecture won — it could absorb far more data and compute per unit of wall-clock time.

Direct long-range paths. Any position reaches any other in one step, so there is no distance at which information degrades.

Scale. The architecture kept improving as parameters and data grew, well past the point where earlier architectures plateaued. BERT [@Devlin2019] is an encoder stack used for understanding; GPT is a decoder stack used for generation. Both are this block, repeated.

Every one of those advantages is about large data and long sequences. Hold that thought against a 26-week window and 5,940 training examples.

Three properties, in the order they actually mattered commercially.

Parallelism. Every position is computed at once. Training scales across hardware in a way recurrence never could. And I want to be clear that this — more than accuracy — is why the architecture won. It could absorb far more data and far more compute per unit of wall-clock time than an RNN. When your competitive advantage is training on more data, the architecture that trains faster wins by default.

Direct long-range paths. Any position reaches any other in one step. There is no distance at which information degrades. Compare the ten-to-the-minus-nine we computed last week.

Scale. The architecture kept getting better as parameters and data grew, well past the point where earlier architectures plateaued. That property is the entire foundation of the last five years of AI. BERT is an encoder stack used for understanding; GPT is a decoder stack used for generation. Both are this block, repeated.

Read that list again with a forecaster’s eye, though. Every single one of those advantages is about large data and long sequences.

Hold that against what we actually have: a twenty-six-week window, and five thousand nine hundred forty training examples.

10.5 Transformers for Time Series

The implementation, the measurement, and an honest reading of it.

10.5.1 Transformer Encoder in PyTorch

Minimal forecasting encoder

layer = nn.TransformerEncoderLayer(d_model=64, nhead=4,
          dim_feedforward=128, dropout=0.1, batch_first=True)
self.enc = nn.TransformerEncoder(layer, num_layers=2)
self.inp = nn.Linear(n_features, 64)
self.fc  = nn.Linear(64, 1)

def forward(self, x):
    h = self.inp(x) + self.pe      # position must be added
    return self.fc(self.enc(h)[:, -1, :]).squeeze(-1)

nn.TransformerEncoderLayer gives you the whole block — but not the positional encoding. You add that yourself, and forgetting is a silent failure.

Multi-step forecasting also needs a causal mask.

nn.TransformerEncoderLayer gives you the entire block — multi-head attention, both residuals, both LayerNorms, the feed-forward sublayer. nn.TransformerEncoder stacks N of them. You add a linear layer to project your features up to d-model, and another to project down to one output. Note batch_first=True again, giving (batch, seq_len, features) — same as the LSTM, opposite of Conv1d. And here is the one that will cost someone in this room a weekend.

nn.TransformerEncoderLayer gives you the whole block — but not the positional encoding.

You add that yourself. It’s the + self.pe on the forward pass. And if you forget it, PyTorch does not complain. There’s no error, no warning, no shape mismatch. The model trains perfectly happily on an unordered bag of weeks and returns a forecast.

It’s a silent failure, which by now you should recognize as the most dangerous kind. We’ll quantify it in two slides.

Add it to the catalogue we’ve been building all semester. KFold on a time series. Scaling outside the fold. A rolling feature you forgot to shift. shuffle=True on a DataLoader whose rows are ordered in time. And now a Transformer with no positional encoding.

Not one of those raises an exception. Every one of them returns a plausible number. The only defense is procedural — you check them because they’re on a list, not because something broke.

One aside: for multi-step forecasting you also need a causal mask, so position t can’t attend to positions after it. We’re doing one-step-ahead, so we don’t need it here — but if you extend this in your project, that’s the thing to look up.

10.5.2 Does It Work?

Same weekly panel, same 26-week windows, same test block as everything since Homework 3.

Model	Test RMSE	Parameters
LASSO, 46 engineered features	744	46
XGBoost	781	—
Vanilla RNN	842	4,545
Random forest	899	—
LSTM	987	17,985
Transformer encoder, 2 layers	990	67,329
1D CNN, best variant	1,032	8,161
Seasonal naive	2,152	0

The Transformer beats the CNN and ties the LSTM — 990 against 987, a difference far smaller than the run-to-run spread. It loses to a vanilla RNN holding one-fifteenth as many parameters, and to a 46-coefficient linear model.

LASSO with forty-six engineered features: seven forty-four. XGBoost: seven eighty-one. Vanilla RNN: eight forty-two, on four and a half thousand parameters. Random forest: eight ninety-nine. LSTM: nine eighty-seven. Transformer encoder, two layers: nine ninety — on sixty-seven thousand parameters. Best CNN: one thousand thirty-two. Seasonal naive: twenty-one fifty-two.

So: the Transformer beats the CNN, and it ties the LSTM — nine ninety against nine eighty-seven, a difference far smaller than the run-to-run spread. Calling that a win in either direction would be reading noise.

And it loses to a vanilla RNN holding one-fifteenth as many parameters. And to a forty-six-coefficient linear model.

That is the state-of-the-art architecture of the last decade, finishing sixth of eight on our data. Next slide is why, and the reason is not that we implemented it badly.

Look at the parameter column once before we move on, because the trend across the whole table is the real story of these three lectures.

Four and a half thousand parameters bought eight forty-two. Eighteen thousand bought nine eighty-seven. Sixty-seven thousand bought nine ninety. And forty-six coefficients bought seven forty-four.

Model size and accuracy are running in opposite directions on this dataset. That is not a universal law — on a large enough corpus the ordering flips entirely, which is precisely why these architectures exist. But on the data in front of you, more capacity has bought worse forecasts every single time.

10.5.3 Reading That Result Honestly

Every advantage on the “why Transformers took over” slide is an advantage at scale, and this problem has none of the scale.

Parallel computation	Real, but irrelevant — the RNN trains in seconds either way.
Direct long-range paths	Worth little when the signal is dominated by recent weeks.
Scales with data	5,940 training windows is not the regime where that pays.

And positional encoding is doing real work, which we can measure. Remove it and the same model must forecast from an unordered bag of 26 weeks:

Transformer with positional encoding	990
Transformer without positional encoding	1,282

292 RMSE — dropping it below even the 1D CNN. That gap is the price of permutation invariance, and the reason the sinusoids exist.

Go back to slide twelve — the three advantages — and take them one at a time against this problem.

Parallel computation. Completely real. And completely irrelevant here: the RNN trains in seconds either way. Parallelism matters when training time is your binding constraint. Ours is about ninety seconds.

Direct long-range paths. Real, and worth very little when the signal is dominated by the recent past. Attention’s superpower is reaching week t minus fifty-two cheaply. Our data mostly wants week t minus one.

Scales with data. True, and five thousand nine hundred forty training windows is simply not the regime where that pays. Transformers overtake other architectures somewhere, but that somewhere is orders of magnitude from here.

But I don’t want you to leave thinking the mechanism did nothing, because one part of it is doing very visible work — and we can measure it.

Take the same model and remove the positional encoding. Now it has to forecast from an unordered bag of twenty-six weeks. With positional encoding: nine ninety. Without: one thousand two hundred eighty-two. That’s a penalty of two hundred ninety-two RMSE — which drops it below even the 1D CNN.

That gap is the price of permutation invariance, and it’s the reason the sinusoids exist. It also tells you the model genuinely is using temporal position when you give it to it.

And remember: forgetting that one line produces no error message. It produces a model that is twenty-nine percent worse and looks completely normal.

10.6 Key Takeaways

Six architectures, one series, and what the ranking actually taught us.

10.6.1 Sequence Architecture Comparison

Architecture	Long range	Parallel	Needs
1D CNN	receptive field only	yes	modest data
Vanilla RNN	$$10–30 steps	no	modest data
LSTM	hundreds of steps	no	more data
Transformer	unlimited, $O(T^2)$	yes	much more data

Choose by the structure of your problem, not by recency of the architecture. Short window and recent-lag signal: a plain RNN, or honestly a linear model. Long sequences, abundant data, dependencies at arbitrary distance: a Transformer, and nothing else comes close.

That second case is language, which is why this architecture reshaped that field and has been slower to displace gradient boosting in tabular forecasting — where the M5 competition was won by ensembles of exactly the methods in Lectures 5 and 6.

Four architectures on three axes. Long range: the CNN sees its receptive field only, the RNN about ten to thirty steps, the LSTM hundreds, the Transformer unlimited — at quadratic cost. Parallel: CNN yes, RNN no, LSTM no, Transformer yes. Data needed: modest, modest, more, and much more. Choose by the structure of your problem, not by the recency of the architecture.

Short window, signal in recent lags? A plain RNN — or honestly, a linear model with good features, which is what won here. Long sequences, abundant data, dependencies at arbitrary distance? A Transformer, and nothing else comes close.

That second case is language. Which is exactly why this architecture reshaped that field, and why it has been much slower to displace gradient boosting in tabular forecasting — where, notably, the M5 competition itself was won by ensembles of the methods you learned in Lectures 5 and 6. Not by a Transformer. By gradient boosting. Let me anticipate a reasonable objection: if this architecture loses here, why spend an hour on it?

Two answers. The first is that you now have a reason — you can say, before fitting anything, that a twenty-six-week window with recent-lag signal and six thousand training examples is not the regime where attention pays. That prediction-in-advance is worth far more than the fitted number.

The second is that most of you will encounter this architecture outside forecasting, in tools you use daily. Understanding that it is a soft lookup with learned weights, that it is permutation-invariant unless told otherwise, and that its advantages are advantages at scale — that understanding transfers well beyond this course.

10.6.2 Lecture 9 Part 2: Key Takeaways

Attention is a learned soft lookup: score every query against every key, softmax the scores, average the values. Distance in the sequence costs nothing.
The price is quadratic in sequence length — fine at 26 steps, the central constraint at 10,000.
Multi-head attention runs several notions of relevance in parallel, at roughly the cost of one, by splitting the dimension.
An encoder block is attention $+$ residual $+$ LayerNorm $+$ a feed-forward sublayer. Only attention is new — the residuals are ResNet, the sublayer is Lecture 7.
Attention is permutation-invariant, so position must be added explicitly. PyTorch does not do this for you, and omitting it fails silently.
On our panel the Transformer tied the LSTM, beat the CNN, and lost to a vanilla RNN with one-fifteenth the parameters. Its advantages are advantages at scale.

Next: Bayesian Statistics I — a different answer to the question of what a forecast should tell you.

One. Attention is a learned soft lookup: score every query against every key, softmax the scores, average the values. Distance in the sequence costs nothing.

Two. The price is quadratic in sequence length — irrelevant at twenty-six steps, the central constraint at ten thousand.

Three. Multi-head attention runs several notions of relevance in parallel, at roughly the cost of one, by splitting the dimension rather than adding to it.

Four. An encoder block is attention plus residual plus LayerNorm plus a feed-forward sublayer. Only attention is new — the residuals are ResNet, the sublayer is Lecture 7.

Five. Attention is permutation-invariant, so position has to be added explicitly. PyTorch does not do it for you, and omitting it fails silently — two hundred ninety-two RMSE, no error message.

And six. On our panel the Transformer tied the LSTM, beat the CNN, and lost to a vanilla RNN with one-fifteenth the parameters. Its advantages are advantages at scale, and this problem has none of the scale.

Next time we change the question entirely. For ten weeks every model has handed you one number per period. Bayesian methods hand you a distribution — and that turns out to answer questions a point forecast simply cannot.