Distribution-Free Tests

Most statistical tests require to know the distribution of the statistic (i.e. a normal distribution, or an F distribution. Any deviation from normality can distort the results.

Usually the type I error-rate decreases when normality assumptions are violated. While this seems to be good at first sight, it also substantially decreases the power of the test.

In order to cope with these situations, tests have been developed which do not assume any special distribution (thus the name "distribution-free tests"). Distribution-free tests are also called non-parametric tests. These tests are always weaker than parametric tests (typically, the efficiency¹ of non-parametric tests falls into the range of 90 to 95 %).

Typical examples of non-parametric tests are the Kolmogorow-Smirnow test for normality, the Mann-Whitney U test for comparing means, the Wilcoxon test for comparing medians of two samples, or the runs test to check the randomness of a series of random numbers.

1	Note: The efficiency of a test is specified by the ratio of the number of observations required for a parametric test (to achieve a predefined level of significance) to the number of observations required by the non-parametric test.