6  Regularization & Model Selection

Lecture 6 of ECON 8310.

6.1 Lecture Outline

1. The Problem: Too Many Predictors
2. Shrinkage Methods
3. Choosing the Penalty
4. In Practice
5. Key Takeaways

Four parts. The problem — why too many predictors breaks ordinary least squares. Shrinkage — Ridge, LASSO, and Elastic Net. Choosing the penalty, which is where most people go wrong. And then practice, plus a map of which method to reach for when.

6.2 The Problem: Too Many Predictors

Feature engineering gives us more predictors than we can safely estimate. Something has to decide which ones survive.

Feature engineering gives us more predictors than we can safely estimate. Something has to decide which ones survive.

6.2.1 Why Regularize?

OLS minimizes \(\sum_{i}(y_i - \mathbf{x}_i'\boldsymbol{\beta})^2\). A weekly model with lags, calendar dummies, promotions, and macro series can easily carry \(k = 60\) predictors on \(n = 150\) observations — OLS will fit that sample beautifully and forecast badly.

Three things go wrong at once. The variance of \(\hat{\boldsymbol{\beta}}\) grows with \(k\). Correlated predictors produce unstable, wildly large coefficients with opposing signs. And once \(k > n\) there is no unique solution at all.

Regularization adds a penalty on coefficient size:

\[\underbrace{\sum_i (y_i - \hat{y}_i)^2}_{\text{fit}} + \underbrace{\lambda \cdot \text{Penalty}(\boldsymbol{\beta})}_{\text{complexity}}\]

You accept a little bias to buy a large reduction in variance — and for a forecast, that trade is almost always worth making.

Start with what OLS actually does. It minimizes the sum of squared residuals. Nothing else. It has no opinion about how many predictors you handed it or how large the coefficients get.

Now put a realistic number on it. A weekly model with lags, calendar dummies, promotions, and a couple of macro series carries sixty predictors easily. And three years of weekly history is a hundred and fifty-six observations.

Sixty predictors, a hundred and fifty observations. OLS will fit that sample beautifully. It will also forecast badly. Three things go wrong, and they’re worth separating because they have different fixes.

First, variance grows with k. Every predictor you add gives the model another dimension in which to chase noise. The coefficients get less precisely estimated, all of them, including the ones you cared about.

Second, correlated predictors produce unstable coefficients. This is the one you’ll actually see in your output. Put lag one and lag two in the same model — and they move together, almost by construction — and OLS can’t tell which deserves the credit. So you get a coefficient of plus eight hundred on one and minus seven hundred and forty on the other. They nearly cancel. The fit is fine. The individual numbers are nonsense, and if you report them to a stakeholder you are reporting noise with a decimal point on it.

Third, once k exceeds n there is no unique solution at all. Not a bad estimate — no estimate. Infinitely many coefficient vectors fit the training data perfectly. Regularization’s answer is to stop minimizing fit alone, and minimize fit plus a penalty on coefficient size.

Look at the shape of that expression on the slide, because you have seen it before. Fit on the left. A complexity penalty on the right. Lambda setting the exchange rate between them.

That is the GAM smoothing penalty from Lecture 3. That is cost-complexity pruning from Lecture 4. Same skeleton, different penalty — and by the end of today I’ll show you it’s been hiding in one more place you didn’t expect.

And the bargain is always the same: you accept a little bias to buy a large reduction in variance. For a forecast — where all you care about is the error on data you haven’t seen — that trade is almost always worth making.

6.2.2 Subset Selection: The Classical Answer

Before shrinkage, the standard approach was to count predictors rather than shrink them [@Guyon2003].

Best subset fits all \(2^k\) models and picks the winner — exact, and computationally hopeless past \(k \approx 30\). Stepwise adds the strongest predictor or drops the weakest, one at a time; it is greedy, and can miss the best combination entirely. Either way the choice is made by adjusted \(R^2\), AIC, AICc, BIC, or CV.

Subset selection is discrete — a predictor is in or out. Small changes in the data flip predictors in and out, so the selected model is itself high-variance. You have not removed the instability; you have moved it somewhere harder to see.

This is the approach in FPP §7.5. Shrinkage replaces the in-or-out decision with a continuous dial — more stable, and easier to tune.

Before shrinkage, the standard approach was to count predictors rather than shrink them. It’s worth understanding properly, both because you’ll meet it and because its failure mode is instructive.

Best subset fits every possible model — all two-to-the-k of them — and picks the winner. It is exact. It is also computationally hopeless past about thirty predictors. Two to the thirty is over a billion models.

Stepwise is the practical cousin. Add the strongest predictor, or drop the weakest, one at a time. It’s tractable, and you’ll notice it’s the same greedy logic as CART from Lecture 4 — and it has the same weakness. It can miss the best combination entirely, because it never reconsiders an earlier decision.

Either way you need a criterion to compare models of different sizes: adjusted R-squared, AIC, AICc, BIC, or cross-validation.

Now the warning box, because this is the real objection and it’s subtle. Subset selection is discrete. A predictor is in, or it is out. There is no middle.

And that means small changes in the data flip predictors in and out of the model. Add three weeks of data, rerun, and you get a different set of variables — sometimes a substantially different set. So the selected model is itself a high-variance object.

Sit with that, because it’s easy to miss. You adopted subset selection to reduce variance. And you did reduce the variance of the coefficients, conditional on the model. But you introduced variance in which model you chose, and that variance is invisible — it doesn’t show up in any standard error your software prints. You haven’t removed the instability. You’ve moved it somewhere harder to see.

This is the approach in FPP section seven point five, which is your reading this week alongside ISLP chapter six. It’s still taught, and it’s still reasonable when you have a handful of candidate predictors and you need a story you can defend. Shrinkage replaces the in-or-out switch with a continuous dial. More stable, and easier to tune.

6.3 Shrinkage Methods

Keep every predictor, but pull its coefficient toward zero.

The core idea in one line: keep every predictor, but pull its coefficient toward zero.

6.3.1 Ridge Regression (\(\ell_2\))

Ridge [@HoerlKennard1970]

\[\hat{\boldsymbol{\beta}}^{\text{ridge}} = \arg\min_{\boldsymbol{\beta}} \sum_{i=1}^{n}(y_i - \mathbf{x}_i'\boldsymbol{\beta})^2 + \lambda \sum_{j=1}^{k} \beta_j^2\]

Ridge shrinks every coefficient toward zero but sets none exactly to zero. At \(\lambda = 0\) you have OLS; as \(\lambda \to \infty\) all \(\beta_j \to 0\). Its particular strength is correlated predictors: rather than letting two collinear variables fight for the same coefficient, it splits the effect between them.

Use it when you believe most predictors carry a little signal and you want to keep them all.

Standardize first. The penalty is not scale-invariant — a predictor measured in dollars is penalized differently from the same predictor in thousands of dollars. Get this wrong and the units decide which variables survive.

Ridge adds lambda times the sum of squared coefficients.

Read what that does. Squaring means large coefficients are punished disproportionately — a coefficient of ten contributes a hundred to the penalty, while ten coefficients of one contribute ten total. So Ridge has a strong preference for spreading effect around rather than concentrating it.

At lambda equals zero you have OLS exactly. As lambda goes to infinity every coefficient goes to zero. Everything useful happens in between.

Ridge shrinks every coefficient toward zero but sets none of them exactly to zero. Hold onto that, because it’s the whole contrast with the next slide.

Its particular strength is exactly the failure I described two slides ago. Remember plus eight hundred and minus seven hundred and forty? Ridge won’t do that. Rather than letting two collinear variables fight over the same coefficient, it splits the effect between them — you get something like plus thirty and plus twenty-eight instead. Far more stable, and far more honest about what the data can actually distinguish.

Use Ridge when you believe most of your predictors carry a little signal and you want to keep them all.

Now the practical warning, and I want to be emphatic because it is the single most common implementation error with any of these methods. Standardize first.

The penalty is not scale-invariant. Lambda times beta-squared depends entirely on what units beta is in. Measure a predictor in dollars and its coefficient is small, so it’s barely penalized. Measure the same predictor in thousands of dollars and its coefficient is a thousand times larger, so it’s penalized a million times harder. Same variable. Same information. Completely different treatment.

Get this wrong and your units decide which variables survive. Not your data. We’ll see the right way to do it in the code slide, and it is not what most people do first.

6.3.2 LASSO (\(\ell_1\))

LASSO [@Tibshirani1996]

\[\hat{\boldsymbol{\beta}}^{\text{lasso}} = \arg\min_{\boldsymbol{\beta}} \sum_{i=1}^{n}(y_i - \mathbf{x}_i'\boldsymbol{\beta})^2 + \lambda \sum_{j=1}^{k} |\beta_j|\]

Swap the squared penalty for an absolute one and the behavior changes qualitatively. LASSO sets some coefficients exactly to zero, performing selection and estimation in a single step, and leaving a sparse model you can actually read aloud to a stakeholder.

Use it when you believe only a handful of predictors genuinely matter and you want the model to tell you which.

But be careful what you conclude. Among a group of highly correlated predictors, LASSO keeps one more or less arbitrarily and zeros the rest. The survivor is not “the important one” — it is the one that happened to win a coin toss between near-identical variables.

Now swap the squared penalty for an absolute one. Sum of absolute values instead of sum of squares. That looks like a small change. It is not — the behavior changes qualitatively.

LASSO sets some coefficients exactly to zero. Not small. Zero. Which means it performs selection and estimation in a single step, and it leaves you with a sparse model you can actually read aloud to a stakeholder.

That last part matters more than it sounds. A model with eight nonzero coefficients is something you can put on a slide and defend in a meeting. A model with sixty small ones is not, even if its RMSE is a hair better.

Use LASSO when you believe only a handful of predictors genuinely matter and you want the model to tell you which. But now be very careful about what you conclude, because this is where LASSO gets misused.

Among a group of highly correlated predictors, LASSO keeps one — more or less arbitrarily — and zeros the rest.

So suppose lag one, lag two, and lag four are all correlated, all genuinely informative, and LASSO keeps lag four. It is tempting — and I’ve seen this in professional work — to report that lag four is “the important lag.”

It isn’t. It’s the one that happened to win a coin toss between near-identical variables. Rerun on slightly different data and lag one wins instead.

The variable that survived is not the important one. It’s the survivor. Those are different claims, and only one of them is supported.

6.3.3 Why LASSO Zeros Out and Ridge Does Not

Both solve “minimize squared error subject to a budget on coefficient size.” The shape of the budget is the whole difference.

Ridge: \(\sum_j \beta_j^2 \le t\) — a circle

• Smooth boundary, no corners
• Error contours almost never touch it on an axis
• So coefficients get small, never zero

LASSO: \(\sum_j |\beta_j| \le t\) — a diamond

• Corners sit exactly on the axes
• Contours are likely to hit a corner first
• A corner means \(\beta_j = 0\) exactly

So the sparsity of LASSO is a geometric accident of the \(\ell_1\) ball having corners — not a statistical test of significance. A zeroed coefficient means “not worth its budget here,” not “proven irrelevant.” See @James2021 §6.2.

So why does one zero out and the other doesn’t? The geometry answers it, and once you see it you won’t forget it.

Both methods can be written the same way: minimize squared error subject to a budget on coefficient size. Spend the budget however you like, but don’t exceed it. The shape of that budget is the entire difference.

Ridge. Sum of squared betas below some budget t. In two dimensions that constraint region is a circle. Smooth boundary, no corners anywhere.

The solution sits where the error contours — the ellipses radiating out from the OLS estimate — first touch that circle. And on a smooth round boundary, the touch point almost never lands exactly on an axis. Landing on an axis would mean one coefficient is exactly zero, and that’s a measure-zero coincidence. So Ridge coefficients get small. They don’t get to zero.

LASSO. Sum of absolute betas below t. That region is a diamond — and a diamond has corners, sitting exactly on the axes.

Now the contours expanding outward are quite likely to hit a corner first. Corners stick out. And a corner is precisely the point where one coefficient equals zero. That’s it. That’s the whole mechanism. Which leads to the interpretive point I most want you to take from this slide.

The sparsity of LASSO is a geometric accident of the ℓ₁ ball having corners. It is not a statistical test of significance.

When LASSO zeros a coefficient it is saying “this predictor isn’t worth its share of the budget at this lambda.” It is not saying “this predictor has been proven irrelevant.” There’s no null hypothesis anywhere in what we just did, and no p-value.

I labor this because LASSO output gets read as if it were a significance table, in papers and in business reports, and it isn’t one. James and co-authors work through the geometry carefully in section six point two if you want the picture drawn out.

6.3.4 Elastic Net: Both Penalties at Once

Elastic Net [@ZouHastie2005]

\[\sum_{i=1}^{n}(y_i - \mathbf{x}_i'\boldsymbol{\beta})^2 + \lambda \left[ \alpha \sum_j |\beta_j| + (1-\alpha) \sum_j \beta_j^2 \right]\]

\(\lambda\) sets the total penalty; \(\alpha \in [0,1]\) sets the mix (\(\alpha=1\) is LASSO, \(\alpha=0\) is Ridge).

The mix keeps LASSO’s ability to zero out predictors while adding Ridge’s grouping effect: correlated predictors now enter or leave together instead of one surviving arbitrarily.

This is usually the right default for forecasting. Lagged predictors are correlated almost by construction — \(y_{t-1}\) and \(y_{t-2}\) move together — which is precisely the situation where pure LASSO picks a winner at random and pure Ridge refuses to simplify.

Elastic Net says: why choose? Use both penalties.

Two knobs now, and keep them straight. Lambda sets the total amount of penalty. Alpha sets the mix between the two types — alpha equals one is pure LASSO, alpha equals zero is pure Ridge, and anything between is a blend.

What you get is LASSO’s ability to zero predictors out, plus Ridge’s grouping effect: correlated predictors now enter or leave the model together, instead of one surviving arbitrarily. Which is the direct fix for the coin-toss problem two slides back.

For forecasting, this is usually the right default, and the reason is structural rather than a matter of taste.

Our predictors are lagged values of the same series. Lag one and lag two are correlated almost by construction — they’re the same series shifted by one week.

That is precisely the situation where pure LASSO picks a winner at random, and pure Ridge refuses to simplify anything. Elastic Net is built for it.

So if you want a default for the homework: start with Elastic Net, tune the mix, and let the data tell you how far toward LASSO or Ridge it wants to sit.

6.4 Choosing the Penalty

\(\lambda\) is not estimated from the model. It is tuned — and how you tune it decides whether the model generalizes.

And here’s the part people rush. Lambda is not estimated from the model. It is tuned — and how you tune it decides whether the model generalizes.

6.4.1 The Regularization Path

Fit the model across a grid of \(\lambda\) values and trace each coefficient as it moves.

Read the path left to right. At small \(\lambda\) the coefficients sit near their OLS values and the model is dense and unstable. As \(\lambda\) grows they shrink, and under LASSO they drop to zero one at a time. At large \(\lambda\) only the most robust predictors survive, and in the limit you are left with the intercept — the mean forecast.

The order in which predictors leave is the useful output. The last survivors are the ones that hold up under the most pressure. Report that ordering rather than a single model’s coefficient table — far more stable across resamples, and far more honest about what the data can support.

The regularization path is the best diagnostic these methods give you, and it’s underused.

The idea: fit the model across a whole grid of lambda values, and trace each coefficient as it moves.

Read it left to right. At small lambda the coefficients sit near their OLS values — the model is dense, and unstable in the way we discussed. As lambda grows they shrink, and under LASSO they drop to zero one at a time. At large lambda only the most robust predictors are left. And in the limit, everything is gone and you’re left with the intercept, which is just the mean forecast — the naive benchmark from Lecture 1.

Now here’s the part I actually want you to use. The order in which predictors leave is the useful output.

Think about what that ordering means. A predictor that survives until very high lambda is one that keeps earning its place as the penalty gets punishing. The last survivors are the ones that hold up under the most pressure.

That ordering is far more stable across resamples than any single model’s coefficient table. Move your training window a few weeks and the exact coefficients at your chosen lambda will shift. The dropout order mostly won’t.

So when you report — in the homework and in your final project — report the ordering. “Under increasing penalty, the last three predictors standing were the four-week lag, the fifty-two-week lag, and SNAP days” is a much stronger claim than “the coefficient on lag four was two thousand one hundred.” It’s more honest about what the data supports, and it’s the version that survives someone rerunning your analysis.

6.4.2 Choosing \(\lambda\) by Cross-Validation

\(\lambda\) is not estimated from the model — it is tuned, and how you tune it decides whether the model generalizes.

1. Define a grid of \(\lambda\), usually log-spaced
2. For each \(\lambda\), run walk-forward CV and record the mean validation RMSE
3. Pick the \(\lambda\) minimizing CV error, then refit on all training data

The one-standard-error rule is worth knowing: rather than the outright minimum, take the largest \(\lambda\) whose CV error is within one standard error of it. You give up almost no accuracy and get a simpler, more stable model.

Never use KFold here. Random folds put future observations in the training set and past ones in validation. The CV error will look excellent and the forecast will fail [@Bergmeir2018]. Use TimeSeriesSplit, exactly as in Lecture 1’s walk-forward evaluation.

So how do you pick lambda? You tune it. Three steps.

One. Define a grid, usually log-spaced — lambda matters multiplicatively, so you want to try point zero one, point one, one, ten, not one through ten. Two. For each lambda, run walk-forward cross-validation and record the mean validation RMSE. Three. Pick the lambda that minimizes CV error, then refit on all the training data.

Now, the one-standard-error rule, which is worth knowing and takes ten seconds to apply.

Instead of taking the outright minimum, take the largest lambda whose CV error is within one standard error of that minimum.

The reasoning: the CV curve is usually flat near its bottom, and the exact location of the minimum is itself noisy — rerun with a different split and it moves. Anything within one standard error is statistically indistinguishable from the best. So among all those equally-good options, take the one with the most regularization: simpler model, more stable, and essentially no accuracy given up.

And now the warning box, which is the same warning I have given you in every lecture, and I am going to keep giving it. Never use KFold here.

Random folds put future observations in the training set and past observations in validation. You are letting the model see the future to predict the past.

The consequence is nasty specifically because it’s silent. There’s no crash. No warning. Your CV error just looks excellent — often dramatically better than what you’d get honestly — and then the forecast fails in production and you have no idea why. Bergmeir and co-authors have the careful treatment of this if you want it. Use TimeSeriesSplit, exactly as in Lecture 1’s walk-forward evaluation. Every time.

6.5 In Practice

Standardize, tune on a time-aware split, and read the path.

Three things: standardize, tune on a time-aware split, and read the path.

6.5.1 Regularization in Python

Code Template

from sklearn.linear_model import ElasticNetCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import TimeSeriesSplit

model = make_pipeline(
    StandardScaler(),
    ElasticNetCV(l1_ratio=[.1, .5, .7, .9, .95, 1],
                 cv=TimeSeriesSplit(5), random_state=42)
).fit(X_train, y_train)

make_pipeline is the part that matters: it keeps standardization inside each CV fold. Scale on the full sample and you have leaked the validation set into training, and your CV error is optimistic.

l1_ratio is \(\alpha\), the Ridge/LASSO mix; the ...CV variants tune \(\lambda\) for you.

sklearn calls the penalty alpha, not lambda.

The code is short, and one line of it is doing something important that’s easy to miss.

We import ElasticNetCV, make_pipeline, StandardScaler, and TimeSeriesSplit. Then we build a pipeline: scaler, then Elastic Net with a list of candidate l1_ratio values and TimeSeriesSplit for the CV. Fit it. make_pipeline is the part that matters, and I want to explain why rather than just assert it.

The obvious thing to do — and what almost everyone does the first time — is standardize your whole dataset once, up front, then run cross-validation on the scaled data. That is leakage.

Because StandardScaler computes a mean and a standard deviation. If you compute those on the full sample, they incorporate the validation folds. So every training fold has been scaled using information from data it isn’t supposed to have seen.

It’s a small leak — it won’t wreck you the way KFold will — but it’s real, it biases your CV error optimistically, and it is completely avoidable.

Wrapping the scaler in a pipeline fixes it. The pipeline refits the scaler inside each fold, using only that fold’s training data. One function call, and the leak is gone.

Two smaller notes.

l1_ratio is alpha — the Ridge-LASSO mix. Give it a list and the CV searches over the mix and the penalty strength together.

And a naming trap that will bite you: sklearn calls the penalty strength alpha, not lambda. Which is unfortunate, because in the textbook notation — and on the slide behind me — alpha is the mix. So sklearn’s alpha is the textbook’s lambda, and sklearn’s l1_ratio is the textbook’s alpha.

Read the docstring, not your memory.

6.6 Key Takeaways and Roadmap

Shrinkage is the linear-model counterpart to what trees do by pruning.

And here’s the line that this whole lecture has been building toward: shrinkage is the linear-model counterpart to what trees do by pruning.

6.6.1 Linear Methods: Which One When

Method	Penalty	Zeros out?	Best when
OLS	none	no	\(k \ll n\), predictors uncorrelated
Stepwise	discrete	yes	Few candidate predictors, need a story
Ridge	\(\ell_2\)	no	Many predictors, all mildly useful
LASSO	\(\ell_1\)	yes	Few predictors truly matter
Elastic Net	both	yes	Many correlated predictors

Connection to Lecture 5. Trees and shrinkage solve the same problem in different geometries. XGBoost’s reg_lambda is literally a Ridge penalty on leaf weights, and its gamma penalizes leaf count the way LASSO penalizes nonzero coefficients. Regularization is not a linear-model topic — it is how every flexible model is kept honest.

The table is your decision guide, and it’s the thing to photograph.

OLS when k is much smaller than n and predictors are uncorrelated. That’s a real situation, just not a common one in forecasting. Stepwise when you have few candidates and need a story you can tell. Ridge when you have many predictors that are all mildly useful. LASSO when you believe few predictors truly matter. Elastic Net when you have many correlated predictors — which, as we said, is most forecasting problems.

Now the connection to Lecture 5, which is the point of the whole lecture. Trees and shrinkage look like completely different technologies. They solve the same problem in different geometries.

Look at XGBoost’s hyperparameters — the ones on the regularization slide last week. reg_lambda is literally a Ridge penalty on the leaf weights. Same ℓ₂ form, applied to leaf values instead of regression coefficients. And gamma penalizes the number of leaves the way LASSO penalizes the number of nonzero coefficients — a fixed charge per unit of complexity, which is what produces the drop-to-zero behavior in both. You were doing regularization last week. Nobody called it that.

So: regularization is not a linear-model topic. It is how every flexible model is kept honest.

And you’ve now met it four times. The GAM smoothing penalty in Lecture 3. Cost-complexity pruning in Lecture 4. Shrinkage on trees in Lecture 5. Ridge and LASSO today.

Next week it appears a fifth time, in neural networks, as weight decay and dropout. Weight decay is Ridge — the same ℓ₂ penalty, applied to network weights, under a different name.

If you take one thing from the first half of this course, take that. The methods multiply. The underlying ideas don’t.

6.6.2 Lecture 6: Key Takeaways

1. With many predictors OLS overfits: coefficient variance grows with \(k\), and with \(k > n\) there is no unique solution at all.

2. Subset selection counts predictors in or out. It is discrete, greedy, and unstable — small data changes flip the selected model.

3. Shrinkage replaces that switch with a dial. Ridge (\(\ell_2\)) shrinks everything smoothly; LASSO (\(\ell_1\)) shrinks and zeros out; Elastic Net does both and keeps correlated predictors together.

4. Standardize first, inside each CV fold — none of these penalties are scale-invariant. \(\lambda\) is tuned, not estimated: use TimeSeriesSplit, never random \(K\)-fold, which leaks the future.

5. The regularization path — the order in which predictors drop out — is a more stable finding than any single coefficient table.

Next: Introduction to Neural Networks (PyTorch) — where these same penalties reappear as weight decay and dropout.

Five things.

One. With many predictors OLS overfits. Coefficient variance grows with k, and once k exceeds n there’s no unique solution at all.

Two. Subset selection counts predictors in or out. It’s discrete, greedy, and unstable — small data changes flip the selected model, and that instability is invisible in your output.

Three. Shrinkage replaces the switch with a dial. Ridge shrinks everything smoothly. LASSO shrinks and zeros out. Elastic Net does both and keeps correlated predictors together.

Four. Standardize first, inside each CV fold — none of these penalties are scale-invariant. And lambda is tuned, not estimated: TimeSeriesSplit, never random K-fold.

Five. The regularization path — the order in which predictors drop out — is a more stable finding than any single coefficient table. Report it.

That closes the first half of the course. Everything so far has been a model you specify and estimate, or an algorithm that partitions your data. Both are things you can, with effort, look inside and explain.

Next week we start neural networks, and that changes. We’ll trade a good deal of interpretability for flexibility — and the same penalties come with us, wearing the names weight decay and dropout.

Assignment 4 is where today’s material and last week’s meet: XGBoost tuning alongside a LASSO regularization path, and Elastic Net against Ridge. Build the path plot — it’s the fastest way to develop intuition for what lambda is actually doing.

And if you want to see the connection I just drew with your own hands, add reg_lambda to your XGBoost grid and watch what it does. It isn’t required, but it’s the cheapest way to convince yourself the two halves of that assignment are the same idea.

6.6.3 References

References are on the slide. Two I’d single out.

Tibshirani nineteen ninety-six is the original LASSO paper, and it’s unusually readable for a foundational statistics paper — worth an hour if you’re curious where this came from.

Zou and Hastie two thousand five introduces Elastic Net, and the motivating section is essentially the correlated-predictor problem we spent this lecture on.

For the homework, ISLP chapter six is the one to read. It covers this material at exactly the level you need, with Python code alongside. See you next week for neural networks.

6.7 Timing guide

Segment	Slides	Target
Opening & the problem	1–5	~7 min
Ridge, LASSO, geometry	6–10	~9 min
Choosing lambda	11–13	~5 min
Python & the synthesis	14–19	~6 min
Total		~27 min

If you must compress, slide 17’s method table survives trimming — it’s a reference students will photograph rather than absorb live. Do not compress slides 9 or 12: the ℓ₁-corner geometry and the regularization path are the two things that make the rest more than a list of estimators.

Slide 17’s XGBoost connection is the payoff for the whole first half of the course. If you are running long, cut elsewhere and protect it.