Significance level

of a statistical test

The probability of incorrectly rejecting the basic hypothesis being tested, when it is valid. In the theory of statistical hypotheses testing (cf. Statistical hypotheses, verification of), the significance level is also called the probability of an error of the first kind. The concept first arose in connection with the problem of testing for compatibility of a theory with experimental data. For example, suppose that observations are being conducted on the values of random variables and that, on the basis of these data, it is required to test a hypothesis , according to which the joint distribution of has some specific property. An appropriate statistical test is constructed with the aid of a suitably selected function ; this function usually assumes small values when is true, and large values when is false. In particular, if are the outcomes of independent measurements (with error) of some known constant and the hypothesis states that no systematic errors are involved, then a reasonable choice of is , where is the number of measured values of that exceed the true value . A large observed value of may be considered a significant statistical refutation of the hypothetical agreement between the experimental outcome and the hypothesis . The corresponding significance test is a rule according to which values of are considered significant if they exceed a prescribed critical value . In its turn, the choice of is governed by the significance level, which equals the probability of the event in the case that the hypothesis is true.

Selection of a significance level should also take into account the unavoidable errors incurred when any specific significance level is employed. For example, if the significance level is excessively high, the main error will stem from rejection of a true hypothesis; but if the significance level is low, the error will usually arise from accepting a false hypothesis. In practice, the most commonly adopted significance levels in statistical calculations range from to . Significance levels lower than are used, for example, in statistical detection of toxic medical preparates, and also in other special situations where the overriding purpose is to ensure against incorrect rejection of the hypothesis being tested. See also Confidence estimation.

References

[1]	H. Cramér, "Mathematical methods of statistics" , Princeton Univ. Press (1946)

References

[a1]	E.L. Lehmann, "Testing statistical hypotheses" , Wiley (1969)