Significance level

of a statistical test

The probability of incorrectly rejecting the basic hypothesis being tested, when it is valid. In the theory of statistical hypotheses testing (cf. Statistical hypotheses, verification of), the significance level is also called the probability of an error of the first kind. The concept first arose in connection with the problem of testing for compatibility of a theory with experimental data. For example, suppose that observations are being conducted on the values of $n$ random variables $X_1,\dots,X_n$ and that, on the basis of these data, it is required to test a hypothesis $H$, according to which the joint distribution of $X_1,\dots,X_n$ has some specific property. An appropriate statistical test is constructed with the aid of a suitably selected function $Y=f(X_1,\dots,X_n)$; this function usually assumes small values when $H$ is true, and large values when $H$ is false. In particular, if $X_1,\dots,X_n$ are the outcomes of independent measurements (with error) of some known constant $a$ and the hypothesis $H$ states that no systematic errors are involved, then a reasonable choice of $Y$ is $(2m-n)^2$, where $m$ is the number of measured values of $X_i$ that exceed the true value $a$. A large observed value of $Y$ may be considered a significant statistical refutation of the hypothetical agreement between the experimental outcome and the hypothesis $H$. The corresponding significance test is a rule according to which values of $Y$ are considered significant if they exceed a prescribed critical value $y$. In its turn, the choice of $y$ is governed by the significance level, which equals the probability of the event $\{Y>y\}$ in the case that the hypothesis $H$ is true.

Selection of a significance level should also take into account the unavoidable errors incurred when any specific significance level is employed. For example, if the significance level is excessively high, the main error will stem from rejection of a true hypothesis; but if the significance level is low, the error will usually arise from accepting a false hypothesis. In practice, the most commonly adopted significance levels in statistical calculations range from $0.01$ to $0.1$. Significance levels lower than $0.01$ are used, for example, in statistical detection of toxic medical preparates, and also in other special situations where the overriding purpose is to ensure against incorrect rejection of the hypothesis being tested. See also Confidence estimation.

References

[1]	H. Cramér, "Mathematical methods of statistics" , Princeton Univ. Press (1946)

References

[a1]	E.L. Lehmann, "Testing statistical hypotheses" , Wiley (1969)