Bootstrap confidence interval
Estimates the uncertainty of almost any statistic by resampling the data with replacement.
What problem am I solving?
How uncertain is a statistic, like a median, a ratio or a model metric, when there’s no simple formula for its standard error?
Why this method?
It works for statistics where no formula exists, and it makes no assumption about the shape of the data’s distribution. One recipe covers medians, percentiles, ratios and differences between models.
Why not something else?
- A formula-based interval, like the t interval for a meanUse it when a formula exists and its assumptions hold. It’s faster, exact under its assumptions, and easier for others to reproduce.
- Percentile bootstrapThe simplest bootstrap variant, but it can be off when the statistic is biased or skewed. BCa corrects for both and is SciPy’s default.
- Bayesian credible intervalBetter when I have prior knowledge worth using or want a direct probability statement about the parameter, at the cost of choosing a model and a prior.
What has to be true?
- The sample is representative of the population and observations are independent. Time series or grouped data need a block or cluster bootstrap.
- The sample is big enough to show the distribution’s shape. With very few points, resampling only recycles the same few values.
- The statistic is smooth. Extremes like the maximum are a known failure case.
How it works
- Draw a sample of the same size from the data, with replacement.
- Compute the statistic on it.
- Repeat thousands of times. The spread of those values approximates the sampling distribution, and its middle 95% gives the interval (in the percentile method).
In code
import numpy as np
from scipy import stats
res = stats.bootstrap((data,), np.median, n_resamples=10_000, method="BCa", random_state=0)
print(res.confidence_interval)Open questions
- Which block length to use for a block bootstrap on daily metrics?
Compared against this in
References
- Efron (1979), Bootstrap methods, another look at the jackknife. The Annals of Statistics.
- Efron and Tibshirani (1993), An Introduction to the Bootstrap.
- scipy.stats.bootstrap