Hypothesis Test Errors
Type I errors, also known as false positives, is the error of rejecting a null hypothesis when it is actually true. This can be viewed as a miss being registered as a hit. The acceptable rate of this type of error is called significance level and is usually set to be
0.05 (5%) or
Type II errors, also known as false negatives, is the error of not rejecting a null hypothesis when the alternative hypothesis is the true. This can be viewed as a hit being registered as a miss.
Depending on the purpose of testing, testers decide which type of error to be concerned. But, usually
type I error is more important than
Sample Vs. Population Mean
In statistics, we often use the mean of a sample to estimate or infer the mean of the broader population from which the sample was taken. In other words, the sample mean is an estimation of the population mean.
Central Limit Theorem
The central limit theorem states that as samples of larger size are collected from a population, the distribution of sample means approaches a normal distribution with the same mean as the population. No matter the distribution of the population (uniform, binomial, etc), the sampling distribution of the mean will approximate a normal distribution and its mean is the same as the population mean.
The central limit theorem allows us to perform tests, make inferences, and solve problems using the normal distribution, even when the population is not normally distributed.
Hypothesis Test P-value
Statistical hypothesis tests return a p-value, which indicates the probability that the null hypothesis of a test is true. If the p-value is less than or equal to the significance level, then the null hypothesis is rejected in favor of the alternative hypothesis. And, if the p-value is greater than the significance level, then the null hypothesis is not rejected.
A univariate T-test (or 1 Sample T-test) is a type of hypothesis test that compares a sample mean to a hypothetical population mean and determines the probability that the sample came from a distribution with the desired mean.
This can be performed in Python using the
ttest_1samp() function of the
SciPy library. The code block shows how to call
ttest_1samp(). It requires two inputs, a sample distribution of values and an expected mean and returns two outputs, the t-statistic and the p-value.
from scipy.stats import ttest_1samp t_stat, p_val = ttest_1samp(example_distribution, expected_mean)
Tukey’s Range Hypothesis Tests
A Tukey’s Range hypothesis test can be used to check if the relationship between two datasets is statistically significant.
The Tukey’s Range test can be performed in Python using the
StatsModels library function
pairwise_tukeyhsd(). The example code block shows how to call
pairwise_tukeyhsd(). It accepts a list of data, a list of labels, and the desired significance level.
from statsmodels.stats.multicomp import pairwise_tukeyhsd tukey_results = pairwise_tukeyhsd(data, labels, alpha=significance_level)