15 Synthesis — What the Semester Measured
Lecture 13 of ECON 8310.
15.1 Lecture Outline
- What We Built
- What the Results Taught
- Choosing a Method
- Doing It Honestly
- Where to Go Next
Five parts. What we built, and how the four parts of the course relate. What the results actually taught — three lessons, each paid for with a measurement. How to choose a method. How to do it honestly, which is the longest section. And where to go next.
15.2 What We Built
Four parts, twelve lectures, one dataset — and a scoreboard nobody would have predicted in Week 1.
15.2.1 The Arc of the Course
| Part | Lectures | The central idea |
|---|---|---|
| Time series | 1–3 | Specify a structure — level, trend, season — and estimate it. |
| Trees | 4–6 | Stop specifying. Hand an algorithm features and let it find the structure. |
| Deep learning | 7–9 | Let the architecture encode what kind of structure to expect. |
| Bayesian | 10–12 | Change the deliverable: return a distribution, not a number. |
Read down that last column and the course is not really four toolkits. It is four answers to a single question: where does the knowledge in a forecast come from?
From you, in the model structure. From the data, via a search procedure. From the architecture’s assumptions. Or from a prior, stated openly and updated. That framing is what survives when the libraries change.
Four parts. Classical time series, Lectures 1 to 3: specify a structure — level, trend, season — and estimate it. Trees, 4 to 6: stop specifying, hand an algorithm features and let it find the structure. Deep learning, 7 to 9: let the architecture encode what kind of structure to expect. Bayesian, 10 to 12: change the deliverable — return a distribution, not a number.
Now read down that last column again, because there’s something there.
This course is not really four toolkits. It’s four answers to a single question: where does the knowledge in a forecast come from?
From you, encoded in the model structure. From the data, via a search procedure. From the architecture’s built-in assumptions. Or from a prior, stated openly and then updated.
That framing is what survives when the libraries change. Every method you meet for the rest of your career sits somewhere on that spectrum, and knowing where tells you what it will be good at and what it will fail at.
15.2.2 The Scoreboard
Every model in this course, on the M5 weekly panel — same 30 series, same held-out block.
| Model | Test RMSE | Where |
|---|---|---|
| LASSO, 46 engineered features | 744 | Homework 4 |
| XGBoost, tuned | 781 | Homework 4 |
| Vanilla RNN, 4 raw channels | 842 | Lecture 9 Part 1 |
| Random forest | 899 | Homework 3 |
| LSTM | 987 | Lecture 9 Part 1 |
| Transformer encoder | 990 | Lecture 9 Part 2 |
| 1D CNN, flatten head | 1,032 | Lecture 8 |
| Seasonal naive | 2,152 | every week |
The ordering is not the one anyone predicts in Week 1. A 46-coefficient linear model is at the top. The two architectures with the most attention behind them — the LSTM and the Transformer — finished fifth and sixth of eight, behind a plain RNN with a fraction of their parameters — at the training budget each was measured at, which Lecture 9 Part 1 shows is part of the claim.
Every model in this course, on the same M5 weekly panel. Same thirty series, same held-out block.
LASSO with forty-six engineered features: seven forty-four. XGBoost, tuned: seven eighty-one. Vanilla RNN on four raw channels: eight forty-two. Random forest: eight ninety-nine. LSTM: nine eighty-seven. Transformer encoder: nine ninety. 1D CNN: one thousand thirty-two. And seasonal naive, at the bottom: twenty-one fifty-two. That ordering is not the one anyone predicts in Week 1. A forty-six-coefficient linear model is at the top. Above gradient boosting, above every neural architecture we built.
And the two architectures with the most attention and money behind them — the LSTM and the Transformer — finished fifth and sixth of eight, behind a plain RNN with a fraction of their parameters.
I want to be careful about what that does and doesn’t mean, because the wrong lesson is easy to take. It does not mean Transformers are overrated in general — they reorganized an entire field. It means they are mismatched to this problem, and the rest of this lecture is about learning to see that mismatch in advance rather than after nine hours of fitting.
15.3 What the Results Taught
Three lessons, each paid for with a measurement.
15.3.1 Lesson 1: The Benchmark Is a Real Competitor
In Homework 1 the seasonal-naive benchmark beat every exponential smoothing and ARIMA model you fitted — and the reason is countable. Holt-Winters must estimate 52 seasonal indices from 4.3 annual cycles of training data: about four observations per parameter. Seasonal naive estimates nothing, and the level it carries moves +3.7%, +1.3%, −4.4%, +2.9% year over year — stable inside ±5%.
Across the whole course the benchmark was beaten, but rarely by the margin the effort implied. The Bayesian structural model beat it by 4%. The GAMs in Homework 2 beat it by 12%. The best model on the weekly panel beat it by 65% — after 46 engineered features and a cross-validated penalty.
Compute the benchmark first, every time. It costs one line, it is often close, and it is the only number that tells you whether the rest of the work was worth doing.
This is also the cheapest professional habit you can carry out of this course. A forecast presented without a benchmark is a number no one can evaluate — including you.
Start with the one that should be most uncomfortable.
In Homework 1, the seasonal-naive benchmark — same week last year, one line of code — beat every exponential smoothing and ARIMA model you fitted.
That was not a trick question or a rigged setup. It is simply what the data supports at that horizon. And across the whole course, the benchmark was beaten, but rarely by the margin the effort implied.
The Bayesian structural model beat it by four percent. The GAMs in Homework 2 beat it by twelve percent. The best model on the weekly panel beat it by sixty-five percent — after forty-six engineered features and a cross-validated penalty. So: compute the benchmark first, every time.
It costs one line. It is often close. And it is the only number that tells you whether the rest of your work was worth doing.
This is also the cheapest professional habit you can carry out of this course, and the one most likely to distinguish you early in a job. A forecast presented without a benchmark is a number nobody can evaluate — including you, including the person who built it. If you take one thing from twelve weeks, take that.
15.3.2 Lesson 2: More Sophisticated Did Not Mean Better
Four times this course, the more advanced method failed to win — and each time for a reason we could name.
| LSTM lost to a plain RNN | At three window lengths, at a fixed 30-epoch budget — they tie by 100. Gating fixes long-range forgetting; 91% of this panel’s variance is between-series, so there is little to forget. |
| Transformer lost to an RNN | 67,329 parameters against roughly 230 effectively independent 26-week blocks — ours overlap 96%. Attention’s long-range paths had nothing here to reach. |
| CNNs lost to LASSO | A seven-week receptive field cannot see an annual lag. |
| Tuning bought nothing measurable | Nine fits and a 3-unit change, against a seed spread of 20–90 at one fixed config. CV improved; test did not. |
None of these means the losing method is bad — each solves a problem this data does not have. LSTMs earn their gates on long dependency chains; Transformers earn their parameters on large corpora.
Match the method to the structure of the problem, not to the recency of the paper.
Four times this semester the more advanced method lost. And each time, we could name the reason — which is the part that matters.
The LSTM lost to a plain RNN, at three window lengths. Gating fixes long-range forgetting; a twenty-six-week window does not suffer from long-range forgetting. Four times the parameters, no gain.
The Transformer lost to an RNN, with fifteen times the parameters. Its advantages are advantages at scale, and five thousand nine hundred forty windows is not scale. CNNs lost to LASSO, because a seven-week receptive field cannot see an annual lag. And tuning bought nothing measurable — nine fits, and a three-unit change.
Both numbers came from a stochastic fit: the assignment specifies subsample=0.8, colsample_bytree=0.8. Fixing random_state=42 makes each number reproducible, but it does not make their difference meaningful — two different hyperparameter settings at the same seed draw entirely different subsamples. Vary only the seed at one fixed config on this panel and the spread is twenty to ninety RMSE. A three-unit gap is inside that many times over.
So the lesson is not that tuning hurt. It is that nine fits bought nothing you could measure — XGBoost’s defaults are well chosen for tabular data, and a grid search that improves cross-validation need not improve test.
None of these means the losing method is bad. Every one of them solves a problem this data does not have. LSTMs earn their gates on long dependency chains. Transformers earn their parameters on large corpora. Those are real capabilities that simply were not needed here. Match the method to the structure of the problem, not to the recency of the paper.
And notice the standard I’m holding these to. It’s not “the fancy model lost, therefore fancy models are bad.” It’s “the fancy model lost, and we can say precisely which of its assumptions didn’t apply.” That second statement is worth something. The first is just cynicism.
15.3.3 Lesson 3: One Idea Appeared Six Times
| Lecture | Name it went by | What it penalized |
|---|---|---|
| 3 | GAM smoothing penalty | Wiggliness of a spline |
| 4 | Cost-complexity pruning | Number of leaves |
| 5 | reg_lambda, gamma |
Leaf weights, leaf count |
| 6 | Ridge, LASSO, Elastic Net | Coefficient size |
| 7 | Weight decay, dropout | Network weights |
| 11 | Hierarchical prior | Distance from the group mean |
Six model families that look nothing alike, one bargain underneath: fit against complexity, with a single parameter setting the exchange rate.
Five of those six tune that parameter by cross-validation. The sixth — the hierarchical prior — estimates it, which is the cleanest version of the idea in the course.
This is what to remember when the tools change. The libraries in this course will be replaced. The trade between fit and complexity will not.
Here’s the one I most want you to leave with.
Lecture 3: the GAM smoothing penalty, penalizing the wiggliness of a spline. Lecture 4: cost-complexity pruning, penalizing the number of leaves. Lecture 5: reg_lambda and gamma, penalizing leaf weights and leaf count. Lecture 6: Ridge, LASSO, Elastic Net, penalizing coefficient size. Lecture 7: weight decay and dropout, penalizing network weights. Lecture 11: the hierarchical prior, penalizing distance from the group mean.
Six model families that look nothing like each other. Splines, trees, boosted trees, linear models, neural networks, and a Bayesian hierarchy. One bargain underneath all of them: fit against complexity, with a single parameter setting the exchange rate.
And five of those six tune that parameter by cross-validation — you try values and pick one. The sixth, the hierarchical prior, estimates it, with a posterior and its own uncertainty. That makes it the cleanest version of the idea in the course.
This is what to remember when the tools change. And they will change — most of the specific libraries in this course will look dated within a decade.
The trade between fit and complexity will not. When you meet a method none of us have heard of yet, one of the first useful questions is: where is its complexity penalty, and what sets it?
15.4 Choosing a Method
15.4.1 Which Method, When
| If your situation is… | start here |
|---|---|
| One series, clear seasonality, short history | ETS or ARIMA — and check the benchmark |
| Several seasonal cycles at once | GAM or Prophet |
| Many engineered features, tabular | Regularized regression first, then XGBoost |
| Many parallel series, shared drivers | Gradient boosting with series identifiers |
| Genuinely long dependencies, lots of data | Recurrent or attention architectures |
| Few observations per group | Hierarchical Bayesian — partial pooling |
| The decision needs a probability | Bayesian, for the predictive distribution |
“Regularized regression first” is not a concession — it won our largest comparison and its coefficients can be read aloud. And the last row is the only one where the question changes rather than the accuracy.
One series, clear seasonality, short history? ETS or ARIMA — and check the benchmark. Several seasonal cycles at once? GAM or Prophet. Many engineered features, tabular data? Regularized regression first, then XGBoost. Many parallel series with shared drivers? Gradient boosting with series identifiers. Genuinely long dependencies and lots of data? Recurrent or attention architectures. Few observations per group? Hierarchical Bayesian — partial pooling. The decision needs a probability? Bayesian, for the predictive distribution.
Two things about that list.
“Regularized regression first” is not a concession or a beginner’s option. It won our largest comparison, and its coefficients can be read aloud in a meeting. Start there and make the complicated model prove it deserves to replace it.
And the last row is the only one where the question changes rather than the accuracy. Everything above it is a horse race on RMSE. That row is about needing a different kind of answer — which is the thing Lectures 10 through 12 were really for.
15.5 Doing It Honestly
The errors that do not announce themselves.
15.5.1 Errors That Fail Silently I: the Pipeline
Every one of these produces a confident, plausible, wrong answer with no error message — and every one is a bug in the code.
KFold on a time series |
Random folds put the future in training. CV error looks excellent; the forecast fails. |
| Scaling outside the fold | Validation statistics leak into training. Smaller effect, same class of error. |
| Rolling features unshifted | A window including the current week leaks the target into its own predictor. |
| Reading MDI importance | Pure noise ranked 17th of 29 in Homework 3 — above a real feature — purely for having many distinct values. |
The common thread: the failure is not a crash, it is a number that looks fine — which is why the discipline is procedural. Split by time, scale inside the fold, shift the window, plant a control.
Every error on this slide produces a confident, plausible, wrong answer with no error message. And every one of them is a bug in the code.
KFold on a time series. Random folds put the future into training. Your CV error looks excellent, and your forecast fails. Scaling outside the fold. Validation statistics leak into training. Smaller effect, same class of error. Rolling features unshifted. A window that includes the current week leaks the target into its own predictor.
Reading MDI importance. In Homework 3, pure noise ranked seventeenth of twenty-nine — above a real feature — purely because it had many distinct values. The common thread is that the failure is not a crash. It’s a number that looks fine.
Which is exactly why the discipline has to be procedural. Split by time. Scale inside the fold. Shift the window. Plant a control.
You do those things because they’re on a list, not because something broke — because nothing is ever going to break.
15.5.2 Errors That Fail Silently II: the Code Was Fine
Harder, and more common in professional work. Here nothing is wrong with the program: the model fits, the diagnostics pass, and the answer is still wrong.
| Pooling groups that are not exchangeable | Lecture 11 Part 2 pooled ten FOODS series with twenty that ignore SNAP. Textbook shrinkage, clean diagnostics, and it reported \(+0.011\) where held-out weeks said \(+0.089\). |
| Standardizing a coefficient you then interpret | Lecture 12: divide log price by its standard deviation and b_price stops being an elasticity. Identical fit, identical diagnostics, and the 5% price scenario moves by 22\(\times\). |
| Omitting a confounder | The naive elasticity was six times too large with an interval half the width. Precision is not accuracy. |
| Ignoring \(\hat{R}\) and divergences | Our first two Bayesian fits returned reasonable-looking forecasts while failing convergence outright. |
Only the last row has a diagnostic. The first three were caught by held-out data, by a DAG drawn before the model, and by asking what one unit of a variable means — none of which any library will run for you.
Now the harder category, and the one that is far more common in professional work.
Here nothing is wrong with your program. The model fits. The sampler converges. Every diagnostic passes. And the answer is still wrong.
Pooling groups that are not exchangeable. Lecture 11 Part 2’s lab pooled ten FOODS series with twenty that ignore SNAP. Textbook shrinkage, clean diagnostics — and it reported an effect of plus point-oh-one-one where the held-out weeks said plus point-oh-eight-nine.
Standardizing a coefficient you then interpret. Lecture 12: divide log price by its standard deviation, and b_price stops being an elasticity. Identical fit, identical diagnostics, and the five percent price scenario moves by a factor of twenty-two.
Omitting a confounder. The naive elasticity was six times too large — with an interval half the width of the correct one. Precision is not accuracy. Ignoring R-hat and divergences. Our first two Bayesian fits returned reasonable-looking forecasts while failing convergence outright.
Look at that list and ask which of them a diagnostic would have caught. Only the last one.
The first three were caught by held-out data, by a DAG drawn before the model, and by asking what one unit of a variable actually means. None of those is something a library will run for you. There is no check_exchangeability(). There is no warning when your coefficient stops being an elasticity.
That’s the difference between knowing how to call a method and knowing what it assumes. Twelve weeks of this course have been aimed at the second one.
15.5.3 The Day Every Model in This Course Fails
Every 25 December the stores close and recorded sales are exactly zero — five times in our sample, against a daily average near 2,800.
| 25 Dec | 2012 | 2013 | 2014 | 2015 |
|---|---|---|---|---|
| Actual | 0 | 0 | 0 | 0 |
| Seasonal naive | 2,230 | 2,045 | 2,485 | 2,365 |
Four days — 0.25% of the evaluation window — carry 4.6% of its squared error, eighteen times their share. Drop them and the benchmark’s RMSE falls from 537 to 525 without a single modeling change.
The information was never missing. is_event fires on 158 days, of which 5 are closures: the feature cannot tell “the doors are locked” from an ordinary Super Bowl, so no model learned the difference.
Knowable years ahead, and every method still failed — because the encoding discarded the distinction, not because the model was misspecified. The harder case is a shock nobody could have known at all: the Favorita earthquake of 16 April 2016 is one, and that dataset is available for final projects.
One more failure, and it is not on the previous slide’s list because it is not a mistake anybody made. It is a limit.
Every twenty-fifth of December, the stores close. Recorded sales are exactly zero — five times in our sample, against a daily average near twenty-eight hundred.
And the seasonal-naive benchmark, on those four evaluable days, predicts twenty-two thirty, twenty forty-five, twenty-four eighty-five, twenty-three sixty-five. Wrong by a hundred percent of the true value, every time, on a date known years in advance.
Now the number that makes this more than an anecdote. Those four days are one quarter of one percent of the evaluation window, and they carry four point six percent of its squared error. Eighteen times their share. Drop four days out of fifteen hundred and the benchmark’s RMSE falls from 537 to 525 — without changing a single thing about the model. So why did nobody catch it? Not for want of information.
is_event fires on 158 days in this data. Five of them are closures. Every model in this course — the trees that split on it, the networks that took it as a channel, the Bayesian regressions that gave it a coefficient — saw a flag that lumps a locked front door together with a hundred and fifty-three Super Bowls and Easters. The feature could not express the difference, so no model could learn it.
And that is the honest shape of this failure. It was knowable years ahead. Nothing was missing from the world; something was missing from the encoding. The model was not misspecified — it was asked a question in a vocabulary that had no word for “closed.”
The harder case is a shock nobody could have known about at all. On the sixteenth of April 2016 a magnitude 7.8 earthquake struck Ecuador, and sales at a large grocery chain there spiked for weeks as people bought water and relief supplies. A forecast made on the fifteenth — by any method in this course — is catastrophically wrong for the following month. Not because the model was bad, but because the future stopped resembling the past, which is the one assumption every method here shares.
That dataset is available if your project wants a real structural break rather than a simulated one. Knowing where the whole toolkit stops working is part of knowing how to use it.
15.5.4 What a Defensible Forecast Report Contains
| A benchmark | Computed, not cited. Every model measured against it. |
| A time-aware split | Walk-forward or a held-out final block, never random. |
| More than one evaluation | A single test split can flatter. Homework 4 ranked models the same way twice, which is why that ranking is believable. |
| The result that went against you | Including tuning that did not help, and the model you expected to win. |
| An interpretation | In the units of the business, not of the loss function. |
The fourth row is the one that distinguishes professional work. A report where every result confirms the author’s expectation is not a report anyone should act on.
You have practised this all semester: the assignments were built so the obvious answer is frequently the wrong one, and the credit was for noticing.
So what does good work actually look like when you hand it to someone? A benchmark — computed, not cited. Every model measured against it. A time-aware split — walk-forward or a held-out final block. Never random.
More than one evaluation. A single test split can flatter. Homework 4 ranked models the same way twice, and that’s why that ranking is believable.
The result that went against you — including tuning that didn’t help, and the model you expected to win. An interpretation, in the units of the business rather than of the loss function.
That fourth row is the one that distinguishes professional work, and it’s the one that feels counterintuitive when you’re junior.
A report where every result confirms the author’s expectation is not a report anyone should act on. Either the author got lucky, or they stopped looking, or they’re not telling you everything. Reporting what went against you is what makes the rest of it credible.
And you have practised this all semester, whether or not it felt like it. The assignments were built so that the obvious answer is frequently the wrong one — and the credit was for noticing, not for getting the flattering number.
15.5.5 Where You Do This Yourself
The final project asks for exactly the report on the previous slide, on data of your choosing.
| Three methods, from at least two parts of the course | Classical, trees, deep learning, Bayesian. Three variations on one family does not count — the comparison must be between approaches differing in kind. |
| Plus a benchmark | Which does not count toward the three, and is not optional. Lesson 1 is why. |
| Walk-forward validation | For every comparison. Not a random split, not once. |
| A recommendation a non-specialist can act on | In the units of the business, with the uncertainty attached. |
| A methods reflection | Which model won, and what about your data made it win. This is where the course is actually assessed. |
The last row is not a formality. Every scoreboard in this course came with a reason, and a project that reports a ranking without one has done the computation but not the work.
Which brings us to the final project, because it asks for exactly the report on the previous slide, on data of your choosing.
Three methods, from at least two parts of the course. Classical, trees, deep learning, Bayesian. Three variations on one family does not satisfy this — the comparison has to be between approaches that differ in kind.
Plus a benchmark, which does not count toward the three and is not optional. Lesson 1 is why. Walk-forward validation for every comparison. Not a random split, and not once. A recommendation a non-specialist can act on, in business units, with the uncertainty attached. A methods reflection — which model won, and what about your data made it win. That last row is not a formality, and it’s where the grade actually lives.
Every scoreboard in this course came with a reason. The LSTM lost because gating solves a problem this data doesn’t have. The CNN lost because seven weeks can’t see an annual lag. The Transformer lost because its advantages are advantages at scale. A project that reports a ranking without a reason has done the computation but not the work.
15.6 Where to Go Next
15.6.1 What This Course Did Not Cover
| Hierarchical reconciliation | Making store forecasts add up to regional totals. Different from Lecture 11’s hierarchy. FPP Ch. 11. |
| Intermittent demand | Series that are mostly zeros — Croston’s method and its relatives. Common in spare parts and slow-moving SKUs. |
| Structural breaks | When the future stops resembling the past. Every method here fails together, and detecting it early is its own field. |
| Causal inference | DAGs told us confounders exist. Estimating causal effects credibly is a separate course. |
| Foundation models | Pretrained forecasters applied zero-shot. Moving quickly — FPP-Py Ch. 15 is a starting point. |
The third row is the one most likely to matter in your career. Every model in this course assumes the future resembles the past. When that stops being true, the model does not tell you — and the benchmark fails at the same moment.
Some honest gaps, so you know what you don’t know.
Hierarchical reconciliation — making store forecasts add up to regional totals. Genuinely different from Lecture 11’s hierarchy, and FPP Chapter 11 covers it.
Intermittent demand — series that are mostly zeros. Croston’s method and its relatives. Very common in spare parts and slow-moving SKUs, and none of our methods handle it well. Structural breaks — when the future stops resembling the past. Causal inference — DAGs told us confounders exist. Estimating causal effects credibly is a separate course. Foundation models — pretrained forecasters applied zero-shot. Moving fast; FPP-Py Chapter 15 is a starting point.
The third row is the one most likely to matter in your career, so let me be explicit about why.
Every model in this course assumes the future resembles the past. Every single one. And when that stops being true — a pandemic, a competitor entering, a supply shock, a pricing change — the model does not tell you.
Worse: the benchmark fails at exactly the same moment, so your usual safety net goes with it. Your forecast is wrong, your comparison is wrong, and everything still looks internally consistent. Detecting that early is its own field, and it’s worth knowing that field exists.
15.6.2 Closing
Compute the benchmark first. It is often close, and it is the only number that says whether your work was worth doing.
Match the method to the problem, not to the paper’s publication date. The most advanced model on our data finished sixth.
One idea recurs: fit against complexity, with a parameter setting the rate. You have now seen it six times under six names.
Split by time, always. The leakage errors in this field do not crash — they return a better-looking number.
Diagnostics check the model you wrote, not whether it was the right one. Convergence, \(\hat{R}\) and ESS all passed on the two worst answers this course produced. Held-out data caught them.
Report what went against you. It is the part that makes the rest credible.
Know what your forecast is for. A point estimate and a predictive distribution answer different questions, and only one of them supports a decision under uncertainty.
The libraries will change. The habits are the transferable part.
One. Compute the benchmark first. It’s often close, and it’s the only number that says whether your work was worth doing.
Two. Match the method to the problem, not to the paper’s publication date. The most advanced model on our data finished sixth.
Three. One idea recurs: fit against complexity, with a parameter setting the rate. Six times, under six names.
Four. Split by time, always. The leakage errors in this field don’t crash — they return a better-looking number.
Five. Diagnostics check the model you wrote, not whether it was the right one. Convergence, R-hat and ESS all passed on the two worst answers this course produced. Held-out data caught them. Six. Report what went against you. It’s the part that makes the rest credible.
Seven. Know what your forecast is for. A point estimate and a predictive distribution answer different questions, and only one of them supports a decision under uncertainty. The libraries will change. The habits are the transferable part. Thank you — and good luck with the project.