OLS vs. Bayesian Regression: What Changes?

OLS, which you have fitted since Lecture 2, solves for one best coefficient vector:

\[\hat{\boldsymbol{\beta}} = (\mathbf{X}^{\top}\mathbf{X})^{-1}\mathbf{X}^{\top}\mathbf{y}\]

The Bayesian version uses Lecture 10’s update rule instead:

\[p(\boldsymbol{\beta} \mid \mathbf{y}) \;\propto\; p(\mathbf{y} \mid \boldsymbol{\beta}) \cdot p(\boldsymbol{\beta})\]

And Lecture 6 reappears: regularization is a prior. The posterior mode under a Normal prior is the Ridge estimate; under a Laplace prior, it is the LASSO estimate.

Bayesian Linear Regression in PyMC

with pm.Model() as model:
    a       = pm.Normal("a", y.mean(), 1, shape=J)  # one per series
    b_price = pm.Normal("b_price", 0, 1)      # weakly informative
    b_snap  = pm.Normal("b_snap",  0, 1)
    b_event = pm.Normal("b_event", 0, 1)
    b_trend = pm.Normal("b_trend", 0, 1)
    sigma   = pm.Exponential("sigma", 1.0)
    mu = a[idx] + b_price*lpc + b_snap*snap + b_event*event + b_trend*yr
    pm.Normal("obs", mu=mu, sigma=sigma, observed=y)
    idata = pm.sample(1000, tune=1500, chains=4)

Do not standardize log price — with both in logs b_price is an elasticity, percent change in demand per percent change in price. lpc is log price centered within series, which fixed ESS 184 \(\rightarrow\) 3,097. J is the number of series (30), idx maps each row to its series, and yr counts years since the first week.

Reading a Posterior Coefficient

Fitted on the weekly panel, controlling for series and trend:

The HDI — highest density interval — is the shortest interval holding 94% of the posterior, and it means what everyone wants: a 94% probability the coefficient is inside.

The last row is the useful one. b_event’s interval spans zero, so the data do not establish a direction. Two slides on, we ask whether it is also too small to matter.

Asking the Posterior a Direct Question

b = idata.posterior["b_price"].values.ravel()
(b < 0).mean()       # P(demand falls when price rises)
(b < -1).mean()      # P(elastic: a price rise loses revenue)
(b < -0.5).mean()    # P(a 1% rise costs over 0.5% of volume)

Each answer is the share of the area past a line. Demand is inelastic in every draw; the half-percent question stays open. The usual p-value, against zero, gives only the direction.

Too Small to Matter? A ROPE

b_event’s HDI spans zero: no direction. A ROPE — region of practical equivalence — asks a different question: is the effect too small to act on?

The width is a business choice, not a statistic. We use Homework 7’s \(|\beta| < 0.05\), about \(\pm 5\%\) a week; even at \(\pm 1\%\), 95% is inside.

Directed Acyclic Graphs

A DAG is a picture of what you believe causes what, drawn from domain knowledge before the model. It decides which variables belong in the regression.

So “add every control you have” is bad advice, and no model fit settles it.

Whether a DAG is right is a business question, not a statistical one.

Confounding, Measured

Two confounders move price and units together: the category and time.

Food is cheap and sells in volume; prices and volumes both rose, 2011–2016. Removing the series means alone flips the sign. With both removed: \(-0.493\), HDI \([-0.596, -0.387]\). The naive \(-2.805\), HDI \([-2.856, -2.756]\), is nearly six times too large, at half the width.

Scenario Analysis with Posterior Samples

Push a scenario through the draws to get a distribution of outcomes, not one number.

b = idata.posterior["b_price"].values.ravel()
pct_change = 100 * (np.exp(b * np.log(1.05)) - 1)

pct_change.mean(), np.percentile(pct_change, [3, 97])

With an elasticity near \(-0.493\), a 5% price rise implies roughly a 2.4% fall in units, 94% of the posterior between \(-2.9\%\) and \(-1.9\%\).

The decision follows. Revenue changes by a factor \(1.05^{\,1+b}\), which rises exactly when \(b > -1\). Every draw is above \(-1\), so \(P(\text{revenue rises}) = 1.00\): about \(+2.5\%\) on average.

Valid only to the extent the DAG is — this is a causal claim about intervening on price.

Course Method Map

XGBoost’s 781 is from an earlier xgboost; the pinned 3.4.1 gives 792.

Lecture 12: Key Takeaways