Hypothesis Testing

Hypothesis Testing

Given a baseline population mean $\mu_0,$ a sample mean $\bar{x},$ a sample count $n,$ and either a sample standard deviation $s$ or a population standard deviation $\sigma,$ we might want to know if the sample data provides sufficient evidence to conclude that the true population mean $\mu$ differs significantly from $\mu_0.$

We could say that our null hypothesis is that the population mean is what we expect:

\[H_0: \mu = \mu_0,\]

and that the alternative hypothesis is that it is different from what we expect:

\[H_1: \mu \neq \mu_0.\]

We’d then compute a test statistic. Here, we will assume we know the population’s standard deviation $\sigma$

\[z = \frac{\bar{x} - \mu_0}{\sigma / \sqrt{n}}.\]

Then, since we want to know the p-value, that is, the probability of getting a test statistic ($z$) this extreme or more extreme if the actual mean is $\mu_0,$, we have a two-tailed test because the sample mean could either be lower than or higher than our actual mean. So, we do

\[\text{p-value} = P(|Z| \geq |z|) = 2 \times P(Z \geq |z|) = 2 \left(1 - \int_{-\infty}^{|z|} \frac{1}{\sqrt{2\pi}} e^{-\frac{t^2}{2}} \, dt \right)\]

or in Mathematica notation:

\[2 * (1 - CDF[NormalDistribution[0, 1], Abs[z]]).\]

Low $p-values$ (typically, p < 0.05 or so) lead us to reject the null-hypothesis and adopt the alternative hypothesis. A very low $p$-value means that the probability of getting the sample we got if $\mu = \mu_0$ is very low.

Instead of hypothesising that that $\mu \neq mu_0,$ we could hypothesize that $\mu > \mu_0.$, or that $\mu < \mu_0.$ These would be single-tailed tests and we’d use, for example:

\[H_0: \mu = \mu_0,\] \[H_1: \mu > \mu_0,\] \[1 - CDF[NormalDistribution[0, 1], z]\]

instead.

Additionally, if we don’t know $\sigma$ but only know $s,$ the standard deviation of the sample, we would use a Student T Distribution. We’d calculate a $t$ value instead of a $z$ value:

\[t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}}.\] \[1 - CDF[StudentTDistribution[n-1], t].\]

Note that if we wanted to test the alternative hypothesis $H_1: \mu < \mu_0$ we’d use

\[CDF[StudentTDistribution[n-1], t],\]

and similarly for when $\sigma$ is known, except we’d use the standard normal distribution and a $z$ value.