1 Probability and Statistics

In statistical mechanics we will perform a replacement of the microscopic dynamics of a system with statistical average dynamical behavior, and obtain a distribution function from which thermodynamic properties of the system can be obtained by computing statistical expectations. In order to do this, we need three things.
1. Knowledge of the statistical methods associated with random variables.
2. Knowledge of the physics behind the microscopic dynamics.
3. A procedure for creating a suitable statistical replacement of the dynamics (Stosszahlansatz).

Basic Probability
Consider a sample space $S$ comprised of events or objects, and subsets $E$ and $F$ such that $E \subset S, F \subset S$ .

Example
Consider three coin tosses, all independent. What is the sample space of outcomes?

S = {(h h h), (h h t), (h t h), (t h h), (h t t), (t h t), (t t h), (t t t)}

We deﬁne the union of two subsets $E$ and $F$ of $S$ to be the collection of elements of $S$ that are either in $E$ , or in $F$ , or in both $E$ and $F$ , and denote the union as $E ⋃ F$ . The collection of events both in $E$ and $F$ we call the intersection and denote this one of two ways; $E ⋂ F = E F$ . The collection of events in $S$ but not in $E$ we call the complement of $E$ and denote it as $E^{c}$ .

Then we have two extremely useful and important statements

S = E ⋃ E^{c}

F = E F ⋃ E^{c} F

both of which you can prove with Venn diagrams. Notice that $E F$ and $E^{c} F$ are mutually exclusive sets; they have an empty intersection

E F ⋂ E^{c} F = \emptyset

Unions and intersections can be shown via Venn diagrams to satisfy basic properties of algebraic binary operations, such as associativity, commutativity and distribution property;

(E ⋃ F) ⋃ G = E ⋃ (F ⋃ G)

E ⋃ F = F ⋃ E

(E ⋃ F) G = E G ⋃ F G

E F = F E

E (F G) = (E F) G

Consider a sample space $S$ and an experiment whose outcome is always some element of $S$ . Repeat the experiment many times under identical conditions. Let $n (e)$ be the number of times in the ﬁrst $n$ repetitions that the result $e \in S$ we get outcome $e$ . We deﬁne the probability of this outcome as

℘ (e) = {lim}_{n \to \infty} \frac{n (e)}{n}

which must satisfy several fundamental axioms;

1 . 0 \leq ℘ (e) \leq 1

2 . ℘ (S) = 1

3 . ℘ (⋃_{n = 1}^{\infty} E_{n}) = \sum_{n} ℘ (E_{n})

in which $E_{n}$ are mutually exclusive (non-intersecting) subsets of $S$ . $S$ is the union of all of its non-intersecting subsets.

Fundamental properties of probabilities can be derived from the following propositions, all of which are fairly simple to prove;

1 . 1 = ℘ (S) = ℘ (E) + ℘ (E^{c})

2 . E \subset F \Rightarrow ℘ (E) \leq ℘ (F)

3 . ℘ (E ⋃ F) = ℘ (E) + ℘ (F) - ℘ (E F)

proof;

℘ (E ⋃ F) = ℘ (E ⋃ E^{c} F) = ℘ (E) + ℘ (E^{c} F)

this is certainly true since the two sets are mutually exclusive. However $F = E F + E^{c} F$ and so

℘ (F) = ℘ (E F) + ℘ (E^{c} F)

again since both of these sets are mutually exclusive. This implies that

℘ (E^{c} F) = ℘ (F) - ℘ (E F)

and we simply insert this into the ﬁrst line of the proof, thereby proving 3.

We deﬁne the conditional probability that $E$ will occur given that $F$ has already occurred as $℘ (E ∣ F)$ . Notice that

℘ (E ∣ F) \neq ℘ (F ∣ E)

in general.

Example. Suppose that on a multiple choice exam in which each question has $m$ possible choices, a student has a probability $p$ of knowing ( $K$ ) the answer to a problem , and a probability of $\frac{1}{m}$ of guessing the correct answer if he or she must guess. Let $℘ (C)$ be the probability that the student gets a problem correct. Let $℘ (K)$ be the probability that the student actually knows the correct answer. Then it is clear that

℘ (C ∣ K) = 1, ℘ (C ∣ K^{c}) = \frac{1}{m}

since the student will mark it correct if they know the answer, and must guess if they do not.

The fundamental tools for working with conditional probabilities are;
1.

℘ (E ∣ F) = \frac{℘ (E F)}{℘ (F)}

which implies

℘ (E F) = ℘ (E ∣ F) ℘ (F) = ℘ (F ∣ E) ℘ (E)

and
2. Bayes’ Theorem

℘ (E) = ℘ (E F) + ℘ (E F^{c}) = ℘ (E ∣ F) ℘ (F) + ℘ (E ∣ F^{c}) ℘ (F^{c})

= ℘ (E ∣ F) ℘ (F) + ℘ (E ∣ F^{c}) (1 - ℘ (F))

Example.
Under the conditions of the previous example, what is the probability that a student actually knows the answer if he or she did answer it correctly?

℘ (K ∣ C) = \frac{℘ (K C)}{℘ (C)} = \frac{℘ (C ∣ K) ℘ (K)}{℘ (C ∣ K) ℘ (K) + ℘ (C ∣ K^{c}) (1 - ℘ (K))}

= \frac{m p}{1 + (m - 1) p}

We say that two events $E$ and $F$ are independent if

℘ (E F) = ℘ (E) ℘ (F)

Example Consider an experiment in which independent trials, each with a probability of success equal to $p$ , will be performed in success. If $n$ trials are performed, ﬁnd the probability that there will be at least one success among them.

℘ (E_{1} E_{2} \dots E_{n}) = \prod_{i} ℘ (E_{i})

let each $E_{i}$ be failure;

℘ (E_{1} E_{2} \dots E_{n}) = {(1 - p)}^{n}

and so the probability of at least one success is

℘ = 1 - {(1 - p)}^{n}

Since each event is independent, we construct the generating function using $x$ as a simple place-holder

G_{1} (x) = ((1 - p) + x p)

such that

G (1) = 1

and

\frac{d G_{1} (1)}{d x} = p = ℘ (s u c c e s s)

Then since all trials are independent, all probabilities are multiplicative and

G_{n} (x) = {((1 - p) + x p)}^{n} = \sum_{m = 0}^{n} x^{m} ℘ (m s u c c e s s) = \sum_{m} (\binom{n}{m}) p^{m} {(1 - p)}^{n - m} x^{m}

This should look very familiar.

1.1 Distribution functions of random variables

The laws of random error propagation are obtained from statistics, and the notion that the precise outcome of each measurement of quantity $x$ is a random variable with some PDF or probability distribution function, $f_{x} (ξ)$ that is used to obtain the probability that the outcome of a measurement will be an $X$ value between $ξ$ and $ξ + d ξ$

f_{x} (ξ) = \frac{d ℘_{x} (ξ)}{d ξ}, 1 = \int_{- \infty}^{\infty} f_{x} (ξ) d ξ

In this notion, the variable whose probability distribution is $f_{x}$ is the subscript of the function, and the argument $ξ$ is a dummy variable or a particular value of $x$ . The probability that a randomly sampled $x$ will be found to lie between $ξ_{1}$ and $ξ_{2}$ is

℘ = \int_{ξ_{1}}^{ξ_{2}} f_{x} (ξ) d ξ

Generally speaking, there is no way for us to know what this PDF looks like, and since conditions of experimentation can change drastically, it may even depend on time. This may appear then to make it impossible to extract any meaningful conclusions from a set of measurements taken in the lab.

1.2 The distribution of averages

We are saved by the power of statistics. The relevance of the PDF to actual measurement can be virtually eliminated by performing the same measurement repeatedly and computing an average result. The PDF of an average of many random variables turns out to be virtually independent of the PDF of the individual data, a result known as the Central Limit Theorem. This is why a valid experimental measurement of a quantity must be the average of a set of measurements, the larger the set, the better.
This property of the average is fairly easy to demonstrate, and makes quite a bit of intuitive sense. We need some machinery ﬁrst.
We will refer to the mean of a randomly distributed variable with PDF $f_{x} (ξ)$ as what statisticians call the expectation

μ = E [x] = \int_{- \infty}^{\infty} ξ f_{x} (ξ) d ξ

Consider $N$ independently made measurements of the same quantity $x$ , each outcome $x_{i}$ was a randomly distributed variable with the same PDF. Statisticians construct the gadget

φ_{x} (t) = \int_{- \infty}^{\infty} e^{i t ξ} f_{x} (ξ) d ξ

since

(\frac{d}{d t} φ_{x} (t))_{t = 0} = (\int_{- \infty}^{\infty} \frac{d}{d t} e^{i t ξ} f_{x} (ξ) d ξ)_{t = 0} = i \int_{- \infty}^{\infty} ξ f_{x} (ξ) d ξ = i E [x] = i μ

which is called a generating function for the moments $m_{k}$ of the PDF

m_{k} = E [x^{k}] = \int_{- \infty}^{\infty} ξ^{k} f_{x} (ξ) d ξ

The moment $m_{1} = μ$ is called the mean of the distribution, and we will call $σ^{2} = m_{2}$ . Consider the generating function for the average $〈 x 〉 = \frac{1}{N} \sum_{i = 1}^{N} x_{i}$ of our $N$ variables

φ_{〈 x 〉} (t) = \int_{- \infty}^{\infty} e^{i t \frac{ξ_{1} + ξ_{2} + \dots + ξ_{N}}{N}} f_{x} (ξ_{1}) f_{x} (ξ_{2}) \dots f_{x} (ξ_{N}) d ξ_{1} d ξ_{2} \dots d ξ_{N}

= \prod_{i = 1}^{N} φ_{x} (\frac{t}{N}) = φ_{x}^{N} (\frac{t}{N})

This becomes particularly simple in the case of $N \to \infty$ ; we can replace the exponential with its Taylor series

e^{i \frac{t ξ}{N}} \approx 1 + i \frac{t ξ}{N} - \frac{1}{2} \frac{t^{2} ξ^{2}}{N^{2}} + \dots

and so

φ_{x} (\frac{t}{N}) = \int_{- \infty}^{\infty} e^{i \frac{t ξ}{N}} f_{x} (ξ) d ξ = \int_{- \infty}^{\infty} f_{x} (ξ) (1 + i \frac{t ξ}{N} - \frac{1}{2} \frac{t^{2} ξ^{2}}{N^{2}} + \dots) d x

\approx 1 + i \frac{μ t}{N} - \frac{1}{2} \frac{σ^{2} t^{2}}{N^{2}} + \dots \approx (1 + i \frac{μ t}{N}) (1 - \frac{1}{2} \frac{σ^{2} t^{2}}{N^{2}})

which allows us to write

φ_{〈 x 〉} (\frac{t}{N}) = {(1 + i \frac{μ t}{N})}^{N} {(1 - \frac{1}{2} \frac{σ^{2} t^{2}}{N^{2}})}^{N} \to e^{i μ t} e^{- \frac{1}{2 N} σ^{2} t^{2}}

by use of the Trotter formula ${lim}_{n \to \infty} {(1 - \frac{x}{n})}^{n} = e^{- x}$ .
Notice that the function $φ_{x} (t)$ is the Fourier transform of the PDF; we can inverse transform to get the PDF of the mean;

f_{〈 x 〉} (ξ) = \frac{1}{2 π} \int_{- \infty}^{\infty} e^{- i t ξ} φ_{〈 x 〉} (t) d t = \frac{1}{2 π} \int_{- \infty}^{\infty} e^{- i t ξ} e^{i μ t} e^{- \frac{1}{2 N} σ^{2} t^{2}} = \frac{1}{\sqrt{2 π \frac{σ^{2}}{N}}} e^{- \frac{{(ξ - μ)}^{2}}{2 \frac{σ^{2}}{N}}}

which is a stunning result. It says that the distribution of the average of many arbitrary random variables is a Normal distribution, peaked at the expectation or average computed from $f_{x} (ξ)$ , with a standard deviation that becomes smaller as the number of measurements is increased. You would see this particular derivation of the Central Limit theorem in any basic statistics course.
The signiﬁcance of this result is twofold. First and foremost, it renders the exact form of the probability distribution of the experimental variable, a random variable, irrelevant. As long as we take lots of measurements without introducing systematic errors, the average that we get will also be a random number, but very sharply and Normally distributed about a mean. Second is the fact that the outcome is a Normal distribution, a PDF that is easy to work with.
The Normal distribution is the Bell-curve which looks like the following for Normally distributed $x$

\frac{d ℘_{x} (ξ)}{d ξ} = f_{x} (ξ) = \frac{1}{\sqrt{2 π} σ} e^{- \frac{{(ξ - μ)}^{2}}{2 σ^{2}}}

The quantity $σ$ is a measure of the amount by which a randomly chosen $x$ differs from the average. If a given set of data has a standard deviation of sigma, then any randomly chosen value of $x$ has a better than $89 %$ chance of being between $μ - σ$ and $μ + σ$ . How would we compute this?

℘_{x} (μ - σ \leq ξ \leq μ + σ) = \int_{μ - σ}^{μ + σ} \frac{1}{\sqrt{2 π} σ} e^{- \frac{{(ξ - μ)}^{2}}{2 σ^{2}}} d ξ

If $σ$ is small, we say that the measurements were precise, meaning all pretty much the same. Precise does not mean accurate though.

Is the shooter suffering from a systematic error or random?

Example If we were to pick 200,000 Normal random deviates, whose mean $μ = 2.0$ and whose standard deviation $σ = 1.0$ , and create a frequency plot showing how many we get with values between $ξ$ and $ξ + d ξ$ , the plot would look like the following.

You should be able to visually inspect this and conclude that about $90 %$ of the numbers fall between $1.0$ and $3.0$ . The following pair of ﬁgure show frequency plots for 200,000 deviates that are themselves averages of 5 and 10 of these types of random numbers.

You can see that all three plots have the same mean, but the frequency plot, which is really the un-normalized PDF, for the averages gets narrower as the size of the set averaged gets bigger. The conclusion is that a reliable or precise experimental measurement will be gotten by averaging over the largest set of individual measurements that we can effectively manage.

1.3 Properties of distributions

A distribution has a mean or average value $μ = \bar{x}$ , a mode, which is the value of $x$ where the distributions PDF is maximal, and a median $x_{m e d}$ deﬁned as the “midpoint”

0.5 = \int_{- \infty}^{x_{m e d}} f_{x} (ξ) d ξ

The Normal distribution has all three of these points coinciding with one another.

We should illustrate the truth of the statements made above concerning the meaning of $μ$ and $σ$ , ﬁrst we show that $μ$ is the average $x$ ;

\bar{x} = \int_{- \infty}^{\infty} ξ f_{x} (ξ) d ξ = \int_{- \infty}^{\infty} ξ \frac{1}{\sqrt{2 π} σ} e^{- \frac{{(ξ - μ)}^{2}}{2 σ^{2}}} d ξ = \int_{- \infty}^{\infty} ((ξ - μ) + μ) \frac{1}{\sqrt{2 π} σ} e^{- \frac{{(ξ - μ)}^{2}}{2 σ^{2}}} d ξ

However

\int_{- \infty}^{\infty} (x - μ) \frac{1}{\sqrt{2 π} σ} e^{- \frac{{(ξ - μ)}^{2}}{2 σ^{2}}} d x = \int_{- \infty}^{\infty} y \frac{1}{\sqrt{2 π} σ} e^{- \frac{y^{2}}{2 σ^{2}}} d y = 0

since the distribution is symmetric about the mean

f_{x} (μ - ξ) = f_{x} (μ + ξ)

and so

\bar{x} = \int_{- \infty}^{\infty} ((x - μ) + μ) \frac{1}{\sqrt{2 π} σ} e^{- \frac{{(ξ - μ)}^{2}}{2 σ^{2}}} d ξ = μ

as we have stated.

We now compute the average deviation (squared) of a random $x$ from the average value $μ$ ;

\bar{{(x - μ)}^{2}} = \int_{- \infty}^{\infty} {(ξ - μ)}^{2} \frac{1}{\sqrt{2 π} σ} e^{- \frac{{(ξ - μ)}^{2}}{2 σ^{2}}} d ξ = \int_{- \infty}^{\infty} y^{2} \frac{1}{\sqrt{2 π} σ} e^{- \frac{y^{2}}{2 σ^{2}}} d y

using parametric integration methods

I_{a} = \int_{- \infty}^{\infty} e^{- a x^{2}} d x = \sqrt{\frac{π}{a}}, \int_{- \infty}^{\infty} x^{2} e^{- a x^{2}} d x = - \frac{d}{d a} I_{a} = \frac{1}{2} \sqrt{\frac{π}{a^{3}}}

with $a = \frac{1}{2 σ^{2}}$ . This results immediately in

\bar{{(x - μ)}^{2}} = σ^{2}

There is a general terminology for quantities such as

μ_{n} = \int_{- \infty}^{\infty} ξ^{n} f_{x} (ξ) d ξ

for the PDF $f_{x} (ξ)$ , these are called moments of the distribution, a concept that we introduced very early in our discussion, and $μ_{1} = μ$ is the mean, $μ_{2} - μ_{1}^{2} = σ^{2}$ making the second moment equal to the standard deviation for a distribution with zero mean. Two other moments are commonly used in statistics, the skew

γ_{1} = \frac{μ_{3}}{σ^{3}}

which measures the symmetry of a distribution about is mean, and the kurtosis

γ_{2} = \frac{μ_{4}}{σ^{4}} - 3

The kurtosis measures the extent to which the distribution differs from the Normal distribution.
Example Hopefully you recall the Maxwell-Boltzmann distribution from physics 205, which describes the probability that a randomly selected particle (per unit volume) will have a speed $v$ between $ξ$ and $ξ + d ξ$ ;

f_{v} (ξ) = 2 π ξ^{2} (\frac{m}{2 π k T})^{\frac{3}{2}} e^{- \frac{m ξ^{2}}{2 k T}}

Show that the average particle kinetic energy is $\frac{m}{2}$ times the second moment of this distribution.

Multi-variate distributions such as $f_{x, y} (ξ, η)$ are equally important in the analysis of laboratory data. We would hope that if $x$ and $y$ are completely different quantities being measured (such as one a voltage, the other a length), that they are independent, meaning that the measurement of one does not inﬂuence the measurement of the other. Statistically this means that the probability of picking a particular $x = ξ$ value is not inﬂuenced by having already chosen a $y$ value, and having gotten $y = η$ . This will be true if the joint PDF is a product of two separate PDFs

f_{x, y} (ξ, η) = f_{x} (ξ) f_{y} (η)

and under these conditions the correlation coefficient

C_{x, y} = \bar{\bar{(x - μ_{x})} \bar{(y - μ_{y})}} = \bar{x y} - μ_{x} μ_{y}

will be zero. It is absolutely not the case however that uncorrelated variables (variables with $C_{x, y} = 0$ ) are automatically independent.

1.4 Other important distributions

There are other PDF’s that come up in physics that have a comparable importance. Consider the simplest experiment that can be conceived of; within a vanishingly small time interval $d t$ , a process either occurs (such as a nuclear decay), with probability $p$ , or does not, with probability $(1 - p)$ . The probability $℘ (M)$ that within a macroscopic time $t = N d t$ , that $M$ such events occur, is generated by

{(p t + (1 - p))}^{N} = φ (t) = \sum_{M = 0}^{N} (\binom{N}{M}) p^{M} {(1 - p)}^{N - M} t^{M} = \sum_{M = 0}^{N} ℘ (M) t^{M}

resulting in the Binomial distribution

℘ (M) = (\binom{N}{M}) p^{M} {(1 - p)}^{N - M}

Notice that we say $℘ (M)$ rather than $\frac{d ℘ (M)}{d M}$ for a discrete distribution.
We call $φ (t) = {(p t + (1 - p))}^{N}$ the generating function of the distribution, which is very useful in obtaining moments. The expectation or mean of this distribution is

μ = \sum_{M = 0}^{N} M ℘ (M) = {[\sum_{M = 0}^{N} ℘ (M) t^{M}]}_{t = 1} = {[t \frac{d}{d t} \sum_{M = 0}^{N} ℘ (M) t^{M}]}_{t = 1}

= t \frac{d}{d t} φ (t) ∣_{t = 1} = t p N {(p t + (1 - p))}^{N - 1} = p N

This PDF has two important limits. First if we consider the case of $p \to 0$ but with $N \to \infty$ such that $p N = μ$ remains constant, we get

{lim}_{p \to 0} {lim}_{N \to \infty} {(p t + (1 - p))}^{N} = {lim}_{N \to \infty} {(1 + (t - 1) \frac{μ}{N})}^{N} = e^{μ t} e^{- μ}

and then

\sum_{M = 0}^{\infty} ℘ (M) t^{M} = e^{μ t} e^{- μ} = \sum_{M = 0}^{\infty} \frac{μ^{M}}{M!} e^{- μ} t^{M}

results in the Poisson distribution

℘ (M) = \frac{μ^{M}}{M!} e^{- μ} = P (μ, M)

with parameter $μ$ . The Poisson distribution for various parameters looks like the following.

Example The probability that an ISP gets a call from a subscriber in any given hour is $0.01$ . The ISP has $300$ subscribers. What is the probability that the ISP will get $4$ calls in the next hour?
First we see that the most probable number of calls is

μ = (300) (0.01) = N p = 3

in an hour. Then

℘ (4) = \frac{3^{4}}{4} e^{- 3} = 0.168

The second important limit of the binomial theorem is used when we want to explore the region around the maximum value $μ = n p$ , which is the most probable value. We will expand the natural log of the distribution around $μ = n p$ by treating the discrete variable $M$ as if it were continuous, calling it $ξ$ for consistency with our previous examples;

ln ℘ (ξ) = ln ℘ (μ) + \frac{d ln ℘ (ξ)}{d ξ} |_{ξ = μ} (ξ - n p) + \frac{1}{2} \frac{d^{2} ln ℘ (ξ)}{d ξ^{2}} |_{ξ = μ} {(ξ - n p)}^{2} + \dots

however the second term vanishes since $μ = n p$ is an extreme point of the distribution. Take the logarithm of the probability

ln ℘ (ξ) = ln n! - ln ξ! - ln (n - ξ)! + ξ ln p + (n - ξ) ln (1 - p)

To evaluate the derivatives we use Stirling’s approximation

\frac{d ln ξ!}{d ξ} = \frac{d}{d ξ} (- ξ + ξ ln ξ + ln \sqrt{2 π}) = ln ξ

\frac{d^{2} ln ξ!}{d ξ^{2}} = \frac{1}{ξ}, \frac{d^{2} ln ξ!}{d ξ^{2}} |_{ξ = n p} = \frac{1}{n p}

and so

ln ℘ (ξ) \approx ln ℘ (μ) + \frac{1}{2} (- \frac{1}{ξ} - \frac{1}{n - ξ}) |_{ξ = n p} {(ξ - n p)}^{2} + \dots

℘ (ξ) \approx ℘ (μ) e^{- \frac{{(ξ - n p)}^{2}}{2 n p (1 - p)}}

which we can normalize via

1 = \int_{- \infty}^{\infty} ℘ (ξ) d ξ

resulting in

℘ (ξ) = \frac{1}{\sqrt{2 π n p (1 - p)}} e^{- \frac{{(ξ - n p)}^{2}}{2 n p (1 - p)}}

which we recognize as the Bell curve (the Normal distribution), peaked around $x = μ = n p$ with standard deviation $σ = \sqrt{n p (1 - p)}$ . Notice that the Maxwell-Boltzmann distribution is a normal distribution in velocity.
Notice that if $p < < 1$ , then $σ = \sqrt{n p (1 - p)} \approx \sqrt{n p} = \sqrt{μ}$ and this plays an important role in hypothesis testing.

1.5 Experimental data

What does all of this preceding material on statistics have to do with experimental data? The assumption is that the vast number of uncontrollable random inﬂuences on an experiment result in the outcomes of measurements being random variables with some PDF. We do not know this PDF. We can however estimate its parameters using basic statistical methods.

Suppose that in the laboratory we measure a quantity $x$ that due to random inﬂuences is described by the PDF $f_{x} (ξ)$ . We have no knowledge as to what the nature of this function is.
Imagine that we take $N$ measurements of $x$ , obtaining a set of data ${x_{1}, x_{2}, \dots, x_{N}}$ . What can this set of data do for us? It tells us something about the PDF that governs the probability that a measurement of $x$ will give a particular value. We can estimate the mean $μ = \int ξ f_{x} (ξ) d ξ$ of the variables PDF by computing the average

\bar{x} = 〈 x 〉 = \frac{1}{N} \sum_{i = 1}^{N} x_{i}, synonymous notations

and estimate the standard deviation squared $σ^{2} = \int {(ξ - μ)}^{2} f_{x} (ξ$