Frequentist vs. Bayesian: What Is Random?

The whole difference is which thing you treat as uncertain.

Everything else follows from that first row. If the parameter is fixed, you can only describe how your estimate would vary across hypothetical repeated samples. If the parameter is uncertain, you can describe your uncertainty about it directly — which is almost always the question someone actually asked.

What This Buys You in a Meeting

A 95% confidence interval of \([0.02,\ 0.08]\) does not mean a 95% chance the parameter is in it. It means the procedure catches the true value in 95% of repeated samples.

A 95% credible interval does mean that, given your model and data.

A full distribution also answers a planner’s question: what is the chance we stock out if we order 400 units?

The cost: you must state a prior, and defend it.

Bayes’ Theorem (Gelman et al. 2013)

Read it as a sentence. What I believe after seeing the data equals how well each value explains the data, times what I believed beforehand, rescaled to sum to one.

The denominator is only that rescaling — it does not change the shape. So the working form is:

\[\text{posterior} \;\propto\; \text{likelihood} \times \text{prior}\]

You will not compute this by hand, and this course never asks you to. You are responsible for choosing the prior, checking the sampler worked, and reading the posterior.

Bayes’ Theorem, Drawn

CA_1 FOODS, 277 weeks. \(\theta\): a SNAP week. \(\mathcal{D}\): a strong week, above its year’s median.

The gray 0.40 is \(p(\text{strong} \mid \text{other weeks})\), 82 of 205; that it equals the posterior is a coincidence.

The Update, in Pictures

A conversion rate \(\theta\): a Beta(5, 45) prior, like 5 successes in 50 trials, then 30 in 200.

Multiply the curves point by point and rescale: the posterior sits between them. Its mean balances the prior’s 0.10, worth 50 observations, against the data’s 0.15, worth 200, at 0.14. With little data the prior does most of the work: a feature when it is defensible, a problem when it is not.

Priors You Will Actually Use

A prior is a distribution over a parameter, chosen to match what that parameter can be.

The support is the first thing to get right. A Normal prior on a standard deviation puts half its mass on negative values, which are impossible: the model is not wrong so much as incoherent. Fold that half onto \(\sigma \ge 0\) and you have the HalfNormal.

Choosing a Prior You Can Defend

The standard objection to Bayesian work is that the prior is arbitrary. The standard answer is that you should not be choosing arbitrary priors.

The honest framing: a prior is an assumption stated in public, where it can be argued with. Frequentist analyses contain assumptions too: functional form, variable selection, sampling scheme. They are just not written as distributions.

MCMC: The Big Idea

Beyond textbook examples, the denominator \(p(\mathcal{D})\) cannot be computed. So we do not compute the posterior: we draw samples from it.

Each step compares two heights of likelihood \(\times\) prior: higher, move; lower, move with probability equal to their ratio. Both heights carry the same \(p(\mathcal{D})\), so it cancels.

This is random-walk Metropolis. PyMC’s default, NUTS, takes smarter steps; the intuition is unchanged.

The Samples Are the Answer

The record of where you walked is the answer: a histogram of the kept draws has the shape of the posterior, and nobody ever computed it.

The planner’s stock-out question becomes one line: the share of simulated demand draws above 400 units.

PyMC: A First Model

import pymc as pm

with pm.Model() as model:
    theta = pm.Beta("theta", alpha=5, beta=45)        # prior
    obs   = pm.Binomial("obs", n=200, p=theta,        # likelihood
                        observed=30)
    idata = pm.sample(2000, tune=1000, chains=4, random_seed=42)

import arviz as az
az.plot_posterior(idata)

A PyMC model describes how the data could have been generated: what I believe about the rate, and how observations arise given it. You write the story; the sampler infers.

tune=1000 discards the first 1,000 steps, while the sampler is still finding the high ground. chains=4 runs four independent walks — which is what makes the next slide possible.

Did the Sampler Actually Work? (Martin et al. 2021)

MCMC can fail quietly: it always returns samples, from the posterior or not.

az.summary(idata, ci_kind="hdi", ci_prob=0.94) prints the interval, \(\hat{R}\) and ESS. Pass both arguments — ArviZ now defaults to an 89% equal-tailed interval, not the HDI. HDI: the narrowest interval holding 94% of the draws. 94 or 95 is a choice; say which.

On the model above: \(\hat{R} = 1.00\), ESS 3,290, zero divergences. Report all three.

Prior Predictive Checks (McElreath 2020)

Before touching the data, simulate from the prior alone.

with model:
    prior_pred = pm.sample_prior_predictive(500)

az.plot_ppc(prior_pred, group="prior")

Ask whether the simulated data could plausibly have come from your business. Impossible values — negative sales, probabilities above one — mean the prior is misspecified, not merely wide.

Three Priors, Three Predictions

The conversion-rate model before any data: successes out of 200 under each prior.

Flat is too vague: 190 successes is as likely as 20. Beta(500, 4500) is too tight: worth 5,000 observations, it has fixed \(\theta\) before the data arrive, and the spread left is binomial noise. Beta(5, 45) rules out the absurd and leaves room.

Lecture 10: Key Takeaways

Next: Bayesian Statistics II — Bayesian time series models, trend and seasonality, hierarchical models.