11 Bayesian Statistics I — Foundations

Lecture 10 of ECON 8310.

11.1 Lecture Outline

Why Bayesian Thinking?
Bayes’ Theorem
Prior Distributions
Posterior Inference with PyMC
Key Takeaways

Five parts. Why you’d think this way at all — and I’ll make that argument in business terms, not philosophical ones. Then Bayes’ theorem, which is one equation and simpler than its reputation. Then priors, which is the part people object to. Then PyMC and, critically, how to tell whether the answer you got is trustworthy. And takeaways.

11.2 Why Bayesian Thinking?

A different answer to the question of what a forecast should tell you.

11.2.1 Frequentist vs. Bayesian: What Is Random?

The whole difference is which thing you treat as uncertain.

	Frequentist	Bayesian
The parameter	A fixed, unknown constant	A quantity you are uncertain about
The data	Random — one draw of many possible samples	Fixed — it is what you observed
The answer	A point estimate and a standard error	A full distribution over values
Prior belief	No formal place for it	Stated explicitly, and updated

Everything else follows from that first row. If the parameter is fixed, you can only describe how your estimate would vary across hypothetical repeated samples. If the parameter is uncertain, you can describe your uncertainty about it directly — which is almost always the question someone actually asked.

The whole difference between the two frameworks is which thing you treat as uncertain. Everything else follows.

The parameter. For a frequentist, it’s a fixed unknown constant — there is one true value, you just don’t know it. For a Bayesian, it’s a quantity you are uncertain about, and you describe that uncertainty with a distribution.

The data. For a frequentist, the data is random — your sample is one draw from many possible samples. For a Bayesian, the data is fixed. It’s what you observed. It’s sitting on your disk. The answer. A point estimate and a standard error, versus a full distribution over values. Prior belief. No formal place for it, versus stated explicitly and updated. Everything else follows from that first row, so let me make it concrete.

If the parameter is fixed and the data is random, then all you can describe is how your estimate would vary across hypothetical repeated samples that you never actually took. If the parameter is uncertain, you can describe your uncertainty about it, directly. And that second thing is almost always what someone actually asked you. I want to head off a misconception, because it costs people a lot of confusion later.

These are not rival theories of probability where one is correct and the other is a mistake. They are different questions, and each framework answers its own question well. Frequentist methods give you guarantees about procedures repeated over the long run, which is exactly right for quality control on a production line.

Bayesian methods give you a statement about this parameter given this data, which is what you need when you have one dataset and one decision to make on Thursday.

Most forecasting sits in that second category. That’s the reason for these three lectures — not that the last nine weeks were wrong.

11.2.2 What This Buys You in a Meeting

A 95% confidence interval of \([0.02,\ 0.08]\) does not mean there is a 95% chance the parameter lies in that range. It means that intervals built this way would contain the true value 95% of the time across repeated samples. That is a statement about the procedure.

A 95% credible interval of \([0.02,\ 0.08]\) does mean there is a 95% probability the parameter lies in that range, given your model and data.

That is not a technicality. It is the difference between a sentence you can say out loud to a decision-maker and one you have to keep correcting.

And because the answer is a full distribution, you can ask questions of it directly: What is the probability the promotion lifts demand by more than 5%? What is the chance we stock out if we order 400 units? Those are the questions a planner has, and a point estimate with a standard error does not answer them.

The cost is that you must state a prior, and defend it. That is the rest of this lecture.

Here’s the sharpest version of the difference.

A ninety-five percent confidence interval of point-oh-two to point-oh-eight does not mean there’s a ninety-five percent chance the parameter lies in that range. It means intervals constructed this way would contain the true value ninety-five percent of the time across repeated samples. That is a statement about the procedure, not about your parameter.

A ninety-five percent credible interval of point-oh-two to point-oh-eight does mean there is a ninety-five percent probability the parameter lies in that range, given your model and your data.

Now — I know that sounds like a technicality that only statisticians care about. It isn’t.

It is the difference between a sentence you can say out loud to a decision-maker and a sentence you have to keep correcting. Everyone in every meeting you have ever been in interprets a confidence interval as a credible interval. The Bayesian version is the one that means what everybody already thinks it means.

And there’s a second, larger payoff. Because the answer is a full distribution, you can interrogate it directly.

What is the probability the promotion lifts demand by more than five percent? Count the posterior samples above five percent. What’s the chance we stock out if we order four hundred units? Count the samples above four hundred.

Those are the questions a planner actually has. A point estimate with a standard error does not answer them — not because it’s inaccurate, but because it isn’t that kind of object.

The cost is that you must state a prior, and defend it. That’s the rest of the lecture.

11.3 Bayes’ Theorem

One equation, and it is a bookkeeping rule rather than a philosophy.

Here’s the equation. Posterior equals likelihood times prior, over the evidence.

Read it as a sentence, and it stops being intimidating: what I believe after seeing the data equals how well each parameter value explains the data, times what I believed beforehand, rescaled so it sums to one. That’s it. It’s a bookkeeping rule for updating beliefs in light of evidence.

The denominator — p of D, the evidence — is only that rescaling. It doesn’t change the shape of the distribution, only its normalization. So in practice people write the working form: Posterior is proportional to likelihood times prior.

That proportionality is what we’ll actually use, and it’s why the denominator being uncomputable turns out not to matter — which is the subject of slide 13. One reassurance, since some of you are looking at that equation with concern.

You will not compute this by hand. This course never asks you to. There is no integral on any homework in this course.

What you are responsible for is three things: choosing a prior you can defend, checking that the sampler actually worked, and reading the posterior correctly. Those are judgment skills, and they are the ones that matter in practice.

And it’s worth registering that the equation itself is not controversial. It’s a theorem — it follows from the definition of conditional probability in two lines, and no statistician of any persuasion disputes it.

What was historically contested is whether you’re entitled to put a probability distribution on a parameter at all, as opposed to on data. That’s a philosophical argument, it ran for most of the twentieth century, and it is not one we’re going to settle in this room.

What settled it in practice was computation. These methods were mathematically fine and practically impossible until MCMC and cheap computers arrived in the 1990s. The philosophy didn’t change; the feasibility did.

11.3.1 Bayes’ Theorem

The update rule

\[\underbrace{p(\theta \mid \mathcal{D})}_{\text{posterior}} = \frac{\overbrace{p(\mathcal{D} \mid \theta)}^{\text{likelihood}} \cdot \overbrace{p(\theta)}^{\text{prior}}}{\underbrace{p(\mathcal{D})}_{\text{evidence}}}\]

Read it as a sentence [@Gelman2013]. What I believe after seeing the data equals how well each value explains the data, times what I believed beforehand, rescaled to sum to one.

The denominator is only that rescaling — it does not change the shape. So the working form is:

\[\text{posterior} \;\propto\; \text{likelihood} \times \text{prior}\]

You will not compute this by hand, and this course never asks you to. You are responsible for choosing the prior, checking the sampler worked, and reading the posterior.

Here’s the equation. Posterior equals likelihood times prior, over the evidence.

You will not compute this by hand. This course never asks you to. There is no integral on any homework in this course.

11.3.2 The Update, in Pictures

Take a concrete case: estimating a conversion rate \(\theta\) from 200 trials.

Prior	A curve over \(\theta\) from 0 to 1, showing what you thought before. Broad if you know little; peaked if you have history.
Likelihood	A second curve, from the data alone: 30 successes out of 200 peaks near 0.15.
Posterior	Multiply the two curves point by point and rescale. It sits between them, closer to whichever was more confident.

Our numbers: the prior is centered on 0.10, the data say 0.15, and the posterior mean lands at 0.14 — between them, nearer the data, because 200 observations outweigh a prior worth about 50.

More data sharpens the likelihood, so the posterior moves toward it and the prior matters less. With little data the prior does most of the work — a feature when it is defensible, a problem when it is not.

Let’s make it concrete with a case you could actually meet. Estimating a conversion rate — theta — from two hundred trials.

The prior is a curve over theta, from zero to one, showing what you thought before you looked. Broad if you know little; peaked if you have history.

The likelihood is a second curve, coming from the data alone. Thirty successes out of two hundred peaks near point-one-five.

The posterior is what you get when you multiply those two curves point by point and rescale. And it lands between them — closer to whichever one was more confident.

Our numbers. The prior is centered on point-one-zero. The data say point-one-five. And the posterior mean comes out at point-one-four.

Between the two, much nearer the data — because two hundred observations outweigh a prior worth about fifty.

That’s the whole mechanism, and it behaves the way you’d want. More data sharpens the likelihood, so the posterior moves toward it and the prior matters less. With very little data the prior does most of the work — which is a feature when the prior is defensible, and a problem when it isn’t.

11.4 Prior Distributions

The part you have to justify, and the part people worry about.

11.4.1 Priors You Will Actually Use

A prior is a distribution over a parameter, chosen to match what that parameter can be.

Prior	Support	Use it for
\(\theta \sim \text{Beta}(\alpha, \beta)\)	\([0, 1]\)	Rates and proportions — conversion, click-through, defect rate
\(\theta \sim \mathcal{N}(\mu_0, \sigma_0^2)\)	all reals	Regression coefficients, trends, anything that can be negative
\(\theta \sim \text{Exp}(\lambda)\)	\([0, \infty)\)	Standard deviations, waiting times, anything that must be positive

The support is the first thing to get right. Putting a Normal prior on a standard deviation allows negative values, which are impossible — the sampler will struggle and the model may not be wrong so much as incoherent.

\(\text{Beta}(5, 45)\) has mean \(5/50 = 0.1\) — a prior centered on a 10% rate, worth about 50 observations of prior evidence.

A prior is just a distribution over a parameter, chosen to match what that parameter can be. Beta, on the interval zero to one — for rates and proportions. Conversion rates, click-through, defect rates. Normal, on all the reals — for regression coefficients, trends, anything that can legitimately be negative. Exponential, on zero to infinity — for standard deviations, waiting times, anything that must be positive.

And the first thing to get right is the support — the set of values the distribution allows.

Put a Normal prior on a standard deviation and you have allowed negative standard deviations, which are not a modeling approximation, they’re impossible. The sampler will struggle, and the model isn’t so much wrong as incoherent.

That’s the most common beginner error in Bayesian modeling, and it’s entirely avoidable: match the support to the parameter.

One arithmetic note worth internalizing. Beta of five and forty-five has a mean of five over fifty — point-one. And it carries about as much weight as fifty prior observations. That’s a useful way to think about prior strength: how many observations is this worth?

That question — how many observations is my prior worth? — is the most useful one to ask when you’re choosing, and it turns an abstract decision into a concrete one.

If you have two hundred data points and your prior is worth fifty, the data will dominate but the prior will still pull. If your prior is worth five thousand, you have effectively decided the answer in advance and the data is decoration.

And notice the asymmetry that creates. With plenty of data, the prior barely matters and people argue about it anyway. With very little data, the prior matters enormously — and that is exactly the situation where Bayesian methods are most useful, and where you owe the most careful justification.

11.4.2 Choosing a Prior You Can Defend

The standard objection to Bayesian work is that the prior is arbitrary. The standard answer is that you should not be choosing arbitrary priors.

Weakly informative	Rules out the absurd, stays agnostic about the plausible. A Normal(0, 1) on a standardized coefficient says “probably not larger than about 2,” which is nearly always true and nearly never controversial. This is the default.
Informative	Encodes real prior evidence — last year’s conversion rate, a published elasticity. Defensible when you can name the source.
Flat / “uninformative”	Sounds neutral, often is not, and can be improper. Avoid it as a reflex choice.

The honest framing: a prior is an assumption stated in public, where it can be argued with. Frequentist analyses contain assumptions too — functional form, variable selection, sampling scheme — they are just not written as distributions.

Now the objection. The standard criticism of Bayesian work is that the prior is arbitrary — that you can get any answer you like by choosing the right one. The standard answer is that you should not be choosing arbitrary priors.

Weakly informative is your default, and it’s the one you’ll use in this course. It rules out the absurd while staying agnostic about the plausible. A Normal zero-one on a standardized coefficient says “probably not larger than about two,” which is nearly always true and nearly never controversial.

Informative priors encode real evidence — last year’s conversion rate, a published elasticity. Perfectly defensible when you can name the source.

Flat, or so-called uninformative, sounds neutral. It often isn’t, it can be improper, and it is a bad reflex choice. Avoid it. And here’s the honest framing of the whole objection.

A prior is an assumption stated in public, in a form where it can be argued with. Somebody can read your model, disagree with your prior, change it, and re-run.

Frequentist analyses contain assumptions too — functional form, which variables you included, which you dropped, the sampling scheme, what you did with outliers. They are just not written down as distributions, so they’re harder to see and harder to contest. The Bayesian prior isn’t an extra assumption. It’s an assumption made visible.

11.5 Posterior Inference with PyMC

Sampling, the code, and how to tell whether the answer is trustworthy.

11.5.1 MCMC: The Big Idea

For any model more complicated than a textbook example, the rescaling denominator in Bayes’ theorem cannot be computed. So we do not compute the posterior — we draw samples from it, and work with the samples.

The picture to hold. You are exploring a hilly landscape in the dark, where height is posterior probability. You take a step, and if the ground is higher you move; if it is lower you usually move back. Over thousands of steps you spend most of your time on the high ground.

The record of where you walked is the answer. A histogram of your visited positions has the shape of the posterior, without ever having computed it.

The samples are the answer. Posterior mean, credible interval, and the probability of any event you care about are all just summaries of that list of visited values — counted, not derived.

PyMC’s default sampler is NUTS, which takes smarter steps than the random walk above. The intuition is unchanged.

Back to that denominator. For any model more complicated than a textbook example, it cannot be computed — it’s an integral over every possible parameter combination.

So we don’t compute the posterior. We draw samples from it, and then work entirely with the samples.

Here’s the picture I want you to hold. You are exploring a hilly landscape in the dark, where height corresponds to posterior probability. You take a step. If the ground is higher, you move there. If it’s lower, you usually move back — usually, not always.

Do that thousands of times and you spend most of your time on the high ground, proportionally to how high it is.

And here is the trick that makes it all work: the record of where you walked is the answer.

A histogram of the positions you visited has the shape of the posterior. You never computed the posterior. You just wandered around in it and wrote down where you’d been.

So everything downstream is counting. Posterior mean is the average of your visited values. Credible interval is a percentile of them. The probability that a parameter exceeds a threshold is the fraction of samples above it.

Counted, not derived. That’s why Bayesian output can answer arbitrary questions — you’re just querying a list of numbers.

PyMC’s actual sampler is NUTS, which takes far smarter steps than the random walk I described. The intuition is unchanged.

One consequence of the sampling approach is worth flagging now, because it will shape how you plan your work.

This is slow. Not slow like XGBoost is slow — slow in a different category. A random forest on our panel fits in seconds. A Bayesian model on the same data can take minutes, and a poorly specified one can take much longer.

You are not solving an optimization problem and stopping. You are wandering around a distribution, thousands of steps, four times over.

That has a practical implication for the remaining assignments: start them early, and test your model on a small subset before you run the full thing. Discovering a specification error forty minutes into sampling is a genuinely miserable experience, and one you can avoid.

11.5.2 PyMC: A First Model

Estimating a conversion rate — 30 successes in 200 trials

import pymc as pm

with pm.Model() as model:
    theta = pm.Beta("theta", alpha=5, beta=45)        # prior
    obs   = pm.Binomial("obs", n=200, p=theta,        # likelihood
                        observed=30)
    idata = pm.sample(2000, tune=1000, chains=4)      # posterior

import arviz as az
az.plot_posterior(idata)

A PyMC model describes how the data could have been generated: what I believe about the rate, and how observations arise given it. You write the story; the sampler infers.

tune=1000 discards the first 1,000 steps, while the sampler is still finding the high ground. chains=4 runs four independent walks — which is what makes the next slide possible.

Here’s the conversion-rate problem, in six lines.

A Beta prior with alpha five and beta forty-five — that’s our point-one prior worth fifty observations. A Binomial likelihood: two hundred trials, thirty observed successes. Then pm.sample, and you’re done.

Notice what you actually wrote. You did not specify an estimation procedure. You did not choose an optimizer or a loss function.

You described how the data could have been generated: here’s what I believe about the rate, and here’s how observations arise given a rate. That’s a story about the world.

You write the story; the sampler does the inference. That is a genuinely different way of working from everything else in this course, where you chose a fitting algorithm.

Two arguments to understand. tune=1000 discards the first thousand steps, while the sampler is still finding the high ground — those steps aren’t from the posterior yet. And chains=4 runs four independent walks from different starting points. That second one seems wasteful. It’s what makes the next slide possible.

Think about why four independent walks are worth the compute. If a single chain wanders into some corner of the parameter space and gets stuck there, it will happily report a confident, tight, completely wrong posterior — and from inside that one chain, there is no way to tell.

Run four from different starting points and ask whether they agree. If they do, that’s evidence they all found the same distribution. If they don’t, you know something is wrong even though no individual chain looks troubled.

It’s the same logic as the ensemble in Lecture 5, applied to diagnosis rather than prediction: independent attempts agreeing is evidence; a single confident attempt is not.

11.5.3 Did the Sampler Actually Work?

MCMC can fail quietly. It always returns samples; whether they came from the posterior is a separate question [@MartinKumarLao2021].

Check	Want	What it means
\(\hat{R}\)	\(< 1.01\)	Did the chains land in the same place? If not, nothing else matters.
ESS	\(> 400\)	Effective sample size — 8,000 correlated draws may carry as much information as 200 independent ones.
Divergences	\(0\)	The sampler hit terrain it could not navigate. Even a few means the posterior is explored wrongly.

az.summary(idata, ci_kind="hdi", ci_prob=0.94) prints the interval, \(\hat{R}\) and ESS. Pass both arguments — ArviZ now defaults to an 89% equal-tailed interval, not the HDI.

On the model above: \(\hat{R} = 1.00\), ESS 3,686, zero divergences. Report all three.

MCMC can fail quietly. It always returns samples. Whether those samples came from your posterior is an entirely separate question, and nothing raises an exception if they didn’t. So there are three checks, and this course requires all three on every model from here on.

R-hat, and you want it below one-point-oh-one. It compares the four chains: did independent walks, started in different places, end up describing the same distribution? If they didn’t, nothing else on the output matters.

ESS — effective sample size — and you want it above four hundred. MCMC draws are correlated with each other, so eight thousand draws may carry only as much information as two hundred independent ones. ESS tells you how many you effectively have.

Divergences, and you want exactly zero. A divergence means the sampler hit terrain it could not navigate — typically a sharp funnel. Even a handful means parts of the posterior were explored wrongly. We’ll meet this for real in Lecture 11, twice. One practical note that will save you confusion. Call az.summary with both ci_kind="hdi" and ci_prob=0.94.

ArviZ now defaults to an eighty-nine percent equal-tailed interval, not the HDI. If you don’t pass those arguments you will get a different interval than the one you think you’re reading, and than the one on these slides.

On our simple model: R-hat one-point-oh-oh, ESS three thousand six hundred eighty-six, zero divergences. Report all three, every time. A posterior mean without its diagnostics is not a result.

11.5.4 Prior Predictive Checks

Before touching the data, simulate from the prior alone [@McElreath2020].

Two lines

with model:
    prior_pred = pm.sample_prior_predictive(500)

az.plot_ppc(prior_pred, group="prior")

Ask whether the simulated data could plausibly have come from your business. Impossible values — negative sales, probabilities above one — mean the prior is misspecified, not merely wide. Data spanning a billion units means it is too vague; identical simulations mean it is too tight.

A prior predictive check is model debugging that costs two lines — and the habit most often skipped.

One more habit, and it’s the one most often skipped. Before you touch the data, simulate from the prior alone. Two lines: sample_prior_predictive, then plot it.

What you’re doing is asking your model to hallucinate datasets, using only your priors, and then looking at whether those datasets could plausibly have come from your business.

Three things to look for.

Impossible values — negative sales, probabilities above one. That means the prior is misspecified, not merely wide. Fix the support.

Absurd ranges — simulated weekly demand spanning zero to a billion units. That prior is too vague, and vague is not the same as neutral.

Identical simulations — every simulated dataset looking the same. Now the prior is too tight and the data won’t be able to move it.

This is model debugging that costs two lines, and it catches errors before you’ve spent an hour sampling. It’s required in Homework 6, and I’d encourage you to make it reflexive.

The general principle is one you’ve met in other forms all semester: check the thing that could silently be wrong, before it costs you.

11.6 Key Takeaways

Bayesian inference: beliefs in, beliefs out. Data updates your prior into a posterior.

11.6.1 Lecture 10: Key Takeaways

Bayesian inference treats parameters as random variables with probability distributions. The posterior distribution \(p(\theta \mid \mathcal{D})\) represents updated beliefs after seeing data.
Bayes’ theorem: posterior \(\propto\) likelihood \(\times\) prior. The normalizing constant \(p(\mathcal{D})\) is usually intractable, so we sample instead of computing it.
Priors encode prior knowledge. Use weakly informative priors (Beta, Normal, HalfNormal) to constrain parameters to plausible values without overwhelming the data.
MCMC (specifically NUTS in PyMC) draws samples from the posterior. From samples, compute any summary statistic: mean, median, HDI, or the probability that a parameter exceeds a threshold.
Prior predictive checks verify that your model generates plausible data before fitting. Run them every time.

Next: Bayesian Statistics II — Bayesian time series models, trend and seasonality, hierarchical models.

One. Bayesian inference treats parameters as random variables with probability distributions. The posterior represents your updated beliefs after seeing the data.

Two. Bayes’ theorem: posterior is proportional to likelihood times prior. The normalizing constant is usually intractable, so we sample instead of computing it.

Three. Priors encode knowledge. Use weakly informative ones — Beta, Normal, HalfNormal — to constrain parameters to plausible values without overwhelming the data. And match the support to the parameter.

Four. MCMC, specifically NUTS in PyMC, draws samples from the posterior. From samples you can compute any summary you like: mean, median, HDI, or the probability a parameter exceeds a threshold. Counted, not derived. Five. Prior predictive checks verify your model generates plausible data before you fit. Run them every time.

And the one that isn’t numbered, which is really the point of the hour: this doesn’t make your forecast more accurate. It changes what the forecast is, from a number into a distribution — and that’s what lets you answer a question about probability.

Next time: Bayesian time series. Trend and seasonality with priors attached — and a sampler failure that looks exactly like success.