Probability basics

Alfred Galichon (NYU)

'math+econ+code' masterclass series: econometrics

Conditional expectations, modes of convergence, and the LLN

© 2018–2026 by Alfred Galichon. Past and present support from NSF grant DMS-1716489, ERC grant CoG-866274 are acknowledged, as well as inputs from contributors listed here.

If you reuse material from this masterclass, please cite as:
Alfred Galichon, 'math+econ+code' masterclass series. https://www.math-econ-code.org/

Learning objectives

• Recall the basic vocabulary used throughout econometrics: probability spaces, random variables, conditional expectations, and the Law of Iterated Expectations (LIE).

• Verify LIE and the variance decomposition formula on a real dataset, and connect them to the Conditional Expectation Function (CEF): the central object of regression analysis.

• Distinguish the four standard modes of convergence for sequences of random variables: almost sure, in probability, in $L^p$, and in distribution; remember the hierarchy among them.

• Use the Law of Large Numbers (LLN) to derive the asymptotic behavior of sample averages, and visualize the conclusion through simulation.

The Central Limit Theorem and its companions (CMT, Slutsky), together with a first confidence interval, are taken up in Lecture 2.

References

[H] Hansen, B. E. (2022). Econometrics. Princeton University Press. Chapters 5–6. Online at https://users.ssc.wisc.edu/~bhansen/econometrics/.

[vdV] van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press.

[CB] Casella, G., and Berger, R. L. (2002). Statistical Inference, 2nd edition. Duxbury.

[M] Mroz, T. A. (1987). 'The Sensitivity of an Empirical Model of Married Women's Hours of Work to Economic and Statistical Assumptions'. Econometrica, 55(4), 765–799.

Motivation: from samples to populations

• Almost every estimator in econometrics has the form of a sample average: possibly composed with a smooth function, possibly applied to a transformed dataset. OLS, IV, MLE, and GMM all fall into this template after a suitable reformulation.

• As a consequence, the entire toolkit of asymptotic statistics, LLN, CLT, CMT, Slutsky, translates almost mechanically into statements about the behavior of econometric estimators. Most of the work later in the course will consist of recognizing the right sample-average representation and applying these tools.

• This first lecture is therefore foundational rather than applied. We adopt a simulation-first approach: every theorem will be illustrated by Monte Carlo experiments where the data-generating process is known, so that the asymptotic behavior is a conclusion that one can check directly in a plot.

• For one important computation, the law of iterated expectations, we work directly with a real dataset, namely the Mroz (1987) sample of married women's labor supply. This dataset will reappear throughout the course.

Loading our libraries

We start with the standard scientific Python stack.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy import stats

rng = np.random.default_rng(777)
plt.rcParams['figure.figsize'] = (8, 4.5)

Loading our data

We use Mroz's (1987) cross-section of $I=753$ married women, of whom $428$ report a positive hourly wage. The dataset is hosted as a CSV on Vincent Arel-Bundock's Rdatasets repository, which mirrors the standard R packages used in econometrics teaching. (If the URL is unreachable, an offline alternative is pip install wooldridge followed by wooldridge.data('mroz'), modulo a small column-name change: in that variant the labor-force-participation indicator is inlf rather than lfp.)

url = 'https://vincentarelbundock.github.io/Rdatasets/csv/sampleSelection/Mroz87.csv'
try:
    mroz = pd.read_csv(url)                       # Rdatasets Mroz87 (needs a network)
except Exception:
    # Offline fallback: the identical Mroz (1987) data ships with the `wooldridge` package.
    import wooldridge
    mroz = (wooldridge.data('mroz')
            .rename(columns={'inlf': 'lfp', 'kidslt6': 'kids5', 'kidsge6': 'kids618'})
            .copy())
print(f'Number of observations: {len(mroz)}')
print(f'Working women (lfp=1):  {(mroz["lfp"]==1).sum()}')
mroz.head(3)

Number of observations: 753
Working women (lfp=1):  428

	lfp	hours	kids5	kids618	age	educ	wage	repwage	hushrs	husage	...	faminc	mtr	motheduc	fatheduc	unem	city	exper	nwifeinc	lwage	expersq
0	1	1610	1	0	32	12	3.3540	2.65	2708	34	...	16310.0	0.7215	12	7	5.0	0	14	10.910060	1.210154	196
1	1	1656	0	2	30	12	1.3889	2.65	2310	30	...	21800.0	0.6615	7	7	11.0	1	5	19.499981	0.328512	25
2	1	1980	1	3	35	12	4.5455	4.04	3072	40	...	21040.0	0.6915	12	7	5.0	0	15	12.039910	1.514138	225

3 rows × 22 columns

For the rest of the lecture we restrict attention to working women, for whom the hourly wage wage is positive and observed. We also create the log-wage variable lwage.

mroz_w = mroz.loc[mroz['lfp'] == 1].copy()
mroz_w['lwage'] = np.log(mroz_w['wage'])
print(f'I = {len(mroz_w)}')
print(mroz_w[['wage', 'lwage', 'educ', 'exper']].describe().round(2))

I = 428
         wage   lwage    educ   exper
count  428.00  428.00  428.00  428.00
mean     4.18    1.19   12.66   13.04
std      3.31    0.72    2.29    8.06
min      0.13   -2.05    5.00    0.00
25%      2.26    0.82   12.00    7.00
50%      3.48    1.25   12.00   12.00
75%      4.97    1.60   14.00   18.00
max     25.00    3.22   17.00   38.00

1. Conditional expectations and the Law of Iterated Expectations

1.1 The conditional expectation

Split the population into groups according to the value of a variable $X$, and average $Y$ within each group:

$$ \mathbb{E}[Y \mid X = x] \;=\; \text{the average of } Y \text{ over just those cases with } X = x . $$

Doing this for every value $x$ produces a function of $x$, $$ m(x) \;:=\; \mathbb{E}[Y \mid X = x], $$ called the Conditional Expectation Function (CEF): the central object of regression analysis. Writing $\mathbb{E}[Y\mid X] := m(X)$ turns it into a random variable: its value is whichever group average corresponds to the $X$ you happened to draw.

We shall use the following two properties throughout this section:

• $\mathbb{E}[Y\mid X]$ is constant within each group: it carries only the information contained in $X$;
• it is the best predictor of $Y$ given $X$: among all functions $g$, the choice $g=m$ minimizes the mean squared error $\mathbb{E}\bigl[(Y-g(X))^2\bigr]$.

For a discrete $X$, the group average is literally $$ \mathbb{E}[Y\mid X=x] \;=\; \sum_{y} y\,\mathbb{P}(Y=y \mid X=x), $$ and when $X$ is continuous the sums become integrals. (There is a general measure-theoretic definition that covers both at once, and allows conditioning on richer information than a single variable. We will not need it in this course.)

The toy example below, a single fair die, makes the idea concrete.

# A finite example: one roll of a fair die.  Outcomes omega_i, i in [I] := {1,...,I},
# with I = 6, each with probability 1/I.
I = 6
omega_i = np.arange(1, I + 1)
p_i     = np.full(I, 1 / I)
Y_i     = omega_i ** 2                    # the random variable  Y(omega) = omega^2

# Condition on X = parity.  Two groups: odd {1,3,5} and even {2,4,6}.
is_even_i = (omega_i % 2 == 0)

# E[Y | X] is constant on each group, equal to the average of Y within that group.
condexp_i = np.empty(I)
for group_i in (~is_even_i, is_even_i):
    condexp_i[group_i] = np.average(Y_i[group_i], weights=p_i[group_i])

print("omega          =", omega_i)
print("Y = omega^2    =", Y_i)
print("E[Y | parity]  =", condexp_i, "  <- constant within each group")
print()
print(f"  E[Y | X=odd ] = {condexp_i[~is_even_i][0]:.4f}   (= 35/3)")
print(f"  E[Y | X=even] = {condexp_i[ is_even_i][0]:.4f}   (= 56/3)")
print()
print(f"LIE:  E[ E[Y|X] ] = {np.dot(p_i, condexp_i):.6f}")
print(f"      E[Y]        = {np.dot(p_i, Y_i):.6f}   (= 91/6)")

omega          = [1 2 3 4 5 6]
Y = omega^2    = [ 1  4  9 16 25 36]
E[Y | parity]  = [11.66666667 18.66666667 11.66666667 18.66666667 11.66666667 18.66666667]   <- constant within each group

  E[Y | X=odd ] = 11.6667   (= 35/3)
  E[Y | X=even] = 18.6667   (= 56/3)

LIE:  E[ E[Y|X] ] = 15.166667
      E[Y]        = 15.166667   (= 91/6)

Each value of E[Y | parity] is a group average, and averaging those groups with their probabilities recovers $\mathbb{E}[Y]=91/6$: the Law of Iterated Expectations, which we now state in general.

1.2 Law of Iterated Expectations

---

Theorem (LIE). For random variables $X,Y$ with $\mathbb{E}\lvert Y\rvert<\infty$, $$ \mathbb{E}\bigl[\mathbb{E}[Y\mid X]\bigr] = \mathbb{E}[Y]. $$ In words: the average of the group averages, each weighted by how likely that group is, is the overall average.

---

Why it is true. Sort everybody into groups by the value of $X$. Adding up $Y$ group by group, and then across groups, is the same as adding up over everybody: $$ \mathbb{E}[Y] \;=\; \sum_x \mathbb{P}(X=x)\,\underbrace{\mathbb{E}[Y\mid X=x]}_{m(x)} \;=\; \mathbb{E}\bigl[m(X)\bigr] \;=\; \mathbb{E}\bigl[\mathbb{E}[Y\mid X]\bigr]. \qquad \square $$ For continuous $X$ the sum becomes an integral and the argument is unchanged.

A useful extension is the tower property: conditioning on less information averages the finer group means into the coarser ones: for instance $\mathbb{E}\bigl[\mathbb{E}[Y\mid X_1,X_2]\mid X_1\bigr]=\mathbb{E}[Y\mid X_1]$.

1.3 Variance decomposition

A second consequence of LIE is the variance decomposition (sometimes called the law of total variance): $$ \operatorname{Var}(Y) = \mathbb{E}\bigl[\operatorname{Var}(Y\mid X)\bigr] + \operatorname{Var}\bigl(\mathbb{E}[Y\mid X]\bigr). $$

The first term is the within-group variance, the residual variation around the CEF, and the second is the between-group variance: the variation of the CEF itself. The ratio of the between-group variance to the total variance is the population $R^2$ of $Y$ on $X$.

1.4 Verification on Mroz data

We bin years of education into discrete categories, treat the cell means as estimates of the CEF $\mathbb{E}[\log\text{wage}\mid\text{educ bucket}]$, and verify both formulas numerically. We use sample analogs throughout: the population formulas hold exactly when expectations are taken with respect to the empirical measure.

# Bucket education into 4 categories
edges = [0, 11, 12, 15, 17]
labels = ['<HS', 'HS', 'Some college', 'College+']
mroz_w['educ_bucket'] = pd.cut(mroz_w['educ'], bins=edges, labels=labels, right=True)

# Sample CEF: cell means of log-wage by education bucket.
# We use ddof=0 (i.e. dividing by n_x rather than n_x-1) so that
# the variance decomposition holds exactly with respect to the empirical measure.
aggregated = mroz_w.groupby('educ_bucket', observed=True)['lwage'].agg(
    **{
        'E[lwage|x]':  'mean',
        'Var[lwage|x]': lambda s: s.var(ddof=0),
        'n_x':         'count',
    }
)
print(aggregated.round(4))

              E[lwage|x]  Var[lwage|x]  n_x
educ_bucket                                
<HS               0.8765        0.4400   72
HS                1.1273        0.4153  212
Some college      1.1585        0.6430   66
College+          1.6774        0.4443   78

aggregated.head()

	E[lwage|x]	Var[lwage|x]	n_x
educ_bucket
<HS	0.876487	0.439997	72
HS	1.127295	0.415305	212
Some college	1.158493	0.643023	66
College+	1.677436	0.444298	78

mroz_w.head()

	lfp	hours	kids5	kids618	age	educ	wage	repwage	hushrs	husage	...	mtr	motheduc	fatheduc	unem	city	exper	nwifeinc	lwage	expersq	educ_bucket
0	1	1610	1	0	32	12	3.3540	2.65	2708	34	...	0.7215	12	7	5.0	0	14	10.910060	1.210154	196	HS
1	1	1656	0	2	30	12	1.3889	2.65	2310	30	...	0.6615	7	7	11.0	1	5	19.499981	0.328512	25	HS
2	1	1980	1	3	35	12	4.5455	4.04	3072	40	...	0.6915	12	7	5.0	0	15	12.039910	1.514138	225	HS
3	1	456	0	3	34	12	1.0965	3.25	1920	53	...	0.7815	7	7	5.0	0	6	6.799996	0.092123	36	HS
4	1	1568	1	2	31	14	4.5918	3.60	2000	32	...	0.6215	12	14	9.5	1	7	20.100058	1.524272	49	Some college

5 rows × 23 columns

# Verify LIE: weighted average of conditional means equals overall mean
N = aggregated['n_x'].sum()
w_b = aggregated['n_x'] / N                       # weight of each education bucket b
E_lwage = (w_b * aggregated['E[lwage|x]']).sum()  # E[E[Y|X]]
E_lwage_direct = mroz_w['lwage'].mean()           # E[Y]

print(f'  E[E[lwage|educ]]  =  {E_lwage:.6f}')
print(f'  E[lwage]          =  {E_lwage_direct:.6f}')
print(f'  difference        =  {E_lwage - E_lwage_direct:.2e}')

  E[E[lwage|educ]]  =  1.190173
  E[lwage]          =  1.190173
  difference        =  0.00e+00

# Verify the variance decomposition
within  = (w_b * aggregated['Var[lwage|x]']).sum()              # E[Var(Y|X)]
between = (w_b * (aggregated['E[lwage|x]'] - E_lwage)**2).sum() # Var(E[Y|X])
total   = mroz_w['lwage'].var(ddof=0)                           # Var(Y)

print(f'  E[Var(lwage|educ)]   (within)   =  {within:.6f}')
print(f'  Var(E[lwage|educ])   (between)  =  {between:.6f}')
print(f'  sum                              =  {within + between:.6f}')
print(f'  Var(lwage)                       =  {total:.6f}')
print(f'  population R^2                   =  {between/total:.4f}')

  E[Var(lwage|educ)]   (within)   =  0.459858
  Var(E[lwage|educ])   (between)  =  0.061935
  sum                              =  0.521793
  Var(lwage)                       =  0.521793
  population R^2                   =  0.1187

The numbers match up to numerical precision, as they must when the expectations are computed with respect to the empirical distribution. About $12\%$ of the variance of log-wages is explained by the education bucket: the rest is residual variation around the CEF. A standard regression of lwage on continuous educ would extract a slightly different $R^2$, since the linear projection is a coarser summary of $\mathbb{E}[Y\mid X]$ than the full CEF; we return to this distinction in Lecture 3.

2. Modes of convergence

Let $(X_n)_{n\geq 1}$ and $X$ be random variables on $(\Omega,\mathcal{F},\mathbb{P})$, with distribution functions $F_n$ and $F$.

• Almost sure convergence ($X_n\xrightarrow{a.s.}X$):

$$ \mathbb{P}\bigl(\lim_{n\to\infty} X_n = X\bigr) = 1. $$

• Convergence in probability ($X_n\xrightarrow{p}X$): for every $\varepsilon>0$,

$$ \lim_{n\to\infty} \mathbb{P}\bigl(\lvert X_n - X\rvert > \varepsilon\bigr) = 0. $$

• Convergence in $L^p$ ($X_n\xrightarrow{L^p}X$, $p\geq 1$):

$$ \lim_{n\to\infty} \mathbb{E}\bigl[\lvert X_n - X\rvert^p\bigr] = 0. $$

• Convergence in distribution ($X_n\xrightarrow{d}X$): $F_n(x)\to F(x)$ at every continuity point $x$ of $F$.

The hierarchy between these modes is summarized as follows:

$$ X_n\xrightarrow{a.s.}X \;\Longrightarrow\; X_n\xrightarrow{p}X \;\Longrightarrow\; X_n\xrightarrow{d}X, $$

and $$ X_n\xrightarrow{L^p}X \;\Longrightarrow\; X_n\xrightarrow{p}X. $$

None of the reverse implications hold in general. However, two partial converses are useful:

• if $X_n\xrightarrow{d}c$ for a constant $c$, then $X_n\xrightarrow{p}c$;
• if $X_n\xrightarrow{p}X$ and $(X_n)$ is uniformly integrable, then $X_n\xrightarrow{L^1}X$.

2.1 A.s. is strictly stronger than in probability

We illustrate that convergence in probability does not imply almost sure convergence. The classical typewriter sequence on $[0,1]$ provides the textbook counterexample. Let $U\sim\mathrm{Unif}(0,1)$ and define, for $n=2^k+j$ with $0\leq j<2^k$, $$ X_n = \mathbf{1}\bigl\{U\in[j/2^k,(j+1)/2^k]\bigr\}. $$ Then $\mathbb{P}(X_n=1)=2^{-k}\to 0$, so $X_n\xrightarrow{p}0$. However, for every $\omega\in[0,1)$, $X_n(\omega)=1$ infinitely often, hence $X_n(\omega)\not\to 0$. We visualize this below.

def typewriter(u, n):
    # X_n(u) for n = 2^k + j, 0 <= j < 2^k
    n = np.asarray(n)
    k = np.floor(np.log2(np.maximum(n, 1))).astype(int)
    j = n - 2**k
    return ((u >= j/2**k) & (u < (j+1)/2**k)).astype(float)

M = 255
n_m = np.arange(1, M + 1)          # the sequence of n's, indexed by m

# Left: P(X_n = 1) -> 0 (convergence in probability)
p_m = 2.0 ** (-np.floor(np.log2(np.maximum(n_m, 1))).astype(int))

# Right: a single sample path X_n(omega) — infinitely many 1's
u = 0.37
X_m = typewriter(u, n_m)

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

axes[0].plot(n_m, p_m, lw=1.2)
axes[0].set_xlabel('n')
axes[0].set_ylabel(r'$\mathbb{P}(X_n = 1)$')
axes[0].set_title(r'Convergence in probability: $\mathbb{P}(X_n=1)\to 0$')
axes[0].set_yscale('log')

axes[1].vlines(n_m, 0, X_m, lw=0.7)
axes[1].set_xlabel('n')
axes[1].set_ylabel(r'$X_n(\omega)$')
axes[1].set_title(r'Sample path at $\omega$ with $U(\omega)=0.37$: not a.s. convergent')
plt.tight_layout()
plt.show()

On the left, the probability that $X_n=1$ decays geometrically, convergence in probability holds. On the right, a single realization of the sequence keeps hitting $1$ at arbitrarily large $n$, the sequence does not converge almost surely.

3. The Law of Large Numbers

3.1 Statement

---

Theorem (Khintchine's Weak LLN). If $(X_i)_{i\geq 1}$ are i.i.d. with $\mathbb{E}\lvert X_1\rvert <\infty$ and mean $\mu$, then $$ \bar X_I := \frac{1}{I}\sum_{i\in[I]} X_i \xrightarrow{p} \mu, \qquad [I]:=\{1,\dots,I\}. $$

---

Theorem (Kolmogorov's Strong LLN). Under the same hypotheses, $\bar X_I\xrightarrow{a.s.}\mu$.

---

Proof of the WLLN under finite variance. Suppose in addition $\sigma^2:=\operatorname{Var}(X_1)<\infty$. Then $\operatorname{Var}(\bar X_I)=\sigma^2/I$, and Chebyshev's inequality yields $$ \mathbb{P}\bigl(\lvert \bar X_I-\mu\rvert > \varepsilon\bigr) \leq \frac{\sigma^2}{I\varepsilon^2}\to 0. \qquad\square $$

The full WLLN (without the finite variance assumption) and the SLLN require more work; see [vdV, ch. 1] or [H, ch. 5–6].

3.2 Simulation

We draw i.i.d. samples from a few distributions with finite mean, plot the running sample mean as a function of $I$, and overlay the true mean.

def running_mean(x_i):
    return np.cumsum(x_i) / np.arange(1, len(x_i) + 1)

I = 10_000
R = 5
specs = {
    'Normal(0,1)':       (lambda size: rng.standard_normal(size), 0.0),
    'Exponential(1)':    (lambda size: rng.exponential(1.0, size), 1.0),
    'Bernoulli(0.3)':    (lambda size: (rng.random(size) < 0.3).astype(float), 0.3),
    'Pareto(α=2.5, x_m=1)': (lambda size: (rng.pareto(2.5, size) + 1) * 1.0, 2.5/(2.5-1)),
}

S = len(specs)
running_mean_s_r_i = np.empty((S, R, I))
for s, (_, (sampler, _)) in enumerate(specs.items()):
    for r in range(R):
        x_i = sampler(I)
        running_mean_s_r_i[s, r] = running_mean(x_i)

fig, axes = plt.subplots(2, 2, figsize=(11, 6.5), sharex=True)
for s, (ax, (name, (_, mu_true))) in enumerate(zip(axes.flat, specs.items())):
    for r in range(R):
        ax.plot(running_mean_s_r_i[s, r], lw=0.8, alpha=0.85)
    ax.axhline(mu_true, color='k', ls='--', lw=1.0, label=fr'$\mu = {mu_true:.3f}$')
    ax.set_xscale('log')
    ax.set_title(name)
    ax.set_ylabel(r'$\bar X_I$')
    ax.legend(loc='upper right', fontsize=9)
axes[1, 0].set_xlabel('I (log scale)')
axes[1, 1].set_xlabel('I (log scale)')
plt.tight_layout()
plt.show()

Each panel shows five independent realizations of $\bar X_I$ for $I$ ranging from $1$ to $10\,000$, against a logarithmic $x$-axis. The convergence is visible in every panel, although the rate is conspicuously slower for the heavy-tailed Pareto distribution (which has finite mean only because $\alpha>1$).

3.3 A counterexample: the Cauchy distribution

The LLN can fail when the integrability assumption fails. The Cauchy distribution has no mean, $\mathbb{E}\lvert X_1\rvert=\infty$, and the sample mean of i.i.d. Cauchy draws is itself Cauchy-distributed (this is a striking property of the Cauchy: $\bar X_I\stackrel{d}{=} X_1$). Hence $\bar X_I$ does not converge to anything.

cauchy_running_mean_r_i = np.empty((R, I))
for r in range(R):
    x_i = rng.standard_cauchy(I)
    cauchy_running_mean_r_i[r] = running_mean(x_i)

fig, ax = plt.subplots(figsize=(8, 4))
for r in range(R):
    ax.plot(cauchy_running_mean_r_i[r], lw=0.8, alpha=0.85)
ax.axhline(0, color='k', ls='--', lw=1.0, label='median = 0')
ax.set_xscale('log')
ax.set_xlabel('I (log scale)')
ax.set_ylabel(r'$\bar X_I$')
ax.set_title('Sample mean of i.i.d. Cauchy: no convergence')
ax.legend()
plt.tight_layout()
plt.show()

The running sample means jump erratically and never settle: a direct consequence of the absence of a mean. The dashed line indicates the median, which is well-defined and toward which the sample median (not the mean) does converge.

Looking ahead

This lecture has introduced the first half of the asymptotic toolkit: the conditional expectation function and the Law of Iterated Expectations (the language of regression), the modes of convergence, and the Law of Large Numbers (consistency of sample averages).

Lecture 2 completes the toolkit with the Central Limit Theorem, the Continuous Mapping Theorem, and Slutsky's theorem, and combines all four into an asymptotic confidence interval for the mean log-wage. From Lecture 3 onward these tools are deployed on the OLS estimator and every estimator that follows.