a type of statistical significance test

无假设检验

1. Calculate the difference in means (or other test statistic) between the two samples

   the observed value of the test statistic.

2. pool (mix) sample A and B.

3. dividing these pooled values into (e.g. every possible) two groups of size $n_A$ and $n_B$.

4. calculate differences in means of each pair of newly sampled data.

   the set of these calculated differences is the exact distribution of possible differences under the null hypothesis.

   排序后称为null distribution

• One-sided p value: the proportion of sampled permutations where the difference in means was greater than or equal to observed value.
• Two-sided p value: the proportion of sampled permutations where the absolute difference was greater than or equal to observed value.

Monte Carlo test

refer to Monte Carlo testing

also known as randomization test.

The amount of all possible permutations is too large. Thus, in most settings, we randomly generate (via Monte Carlo sampling) some of the possible permutations (10000 or 40000).

蒙特卡洛方法在此的应用相当于按照某分布生成 index, 然后按照 index 决定数据的分组