Which test statistics is better? for two-sample hypothesis test
a simulation study
Lin Yu
2023-03-28
1 / 10
Distributions
we considered normal and nonnormal (uniform, weibull) distributions
2 / 10
Scenarios
We simulated different settings, simply put,
- sample size varies from 10(small) to 1000(large);
- different magnitude of diff between μ1 and μ2, sd1 and sd2;
- finally, both normal and nonnormal distribution were tested.
Details are noted in the following table:
| parameter | description | setting |
|---|---|---|
| μ1 | mean of population 1 | c(0.1,1,3,5) |
| μ2 | mean of population 2 | c(1,3,10) |
| sd1 | standard deviation of ppl 1 | c(1,5,10) |
| sd2 | standard deviation of ppl 2 | c(0.1,1,10) |
| nsample1 | sample size of ppl 1 | c(10,20,30,50,100,1000) |
| nsample2 | sample size of ppl 2 | c(10,20,30,100,1000) |
| dist | study population distribution | normal, uniform, and weibull |
3 / 10
Pseudo-code
function(seed,mu1,mu2,sd1,sd2,nsample1,nsample2,dist,Nsim){ #### for loop over all paramaters for (parameter in 1:length(mu1)){ ------------------------------------------------------- #### calculate the test stats and pval for Nsim times for(i in 1:Nsim){ dat1 = dat2 = test_stats = pval = } -------------------------------------------------------- #### calculate type I error and power by counting pval/Nsim }}4 / 10
Results
##wide table to long tablelibrary(tidyr)## Warning: 程辑包'tidyr'是用R版本4.1.3 来建造的long <- final_res %>% pivot_longer( cols = `t_equal`:`wilcox`, names_to = "test_stat", values_to = "value", )| distribution | mu1 | mu2 | sd1 | sd2 | nsample1 | nsample2 | t_equal | t_notequal | z | wilcox |
|---|
distribution | mu1 | mu2 | sd1 | sd2 | nsample1 | nsample2 | t_equal | t_notequal | z | wilcox | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | normal | 1 | 1 | 1 | 1 | 20 | 20 | 0.058 | 0.058 | 0.065 | 0.059 |
| 2 | normal | 1 | 1 | 5 | 5 | 20 | 20 | 0.049 | 0.049 | 0.056 | 0.047 |
| 3 | normal | 1 | 1 | 2 | 3 | 20 | 20 | 0.053 | 0.053 | 0.054 | 0.052 |
| 4 | normal | 0.1 | 0.1 | 5 | 5 | 20 | 20 | 0.047 | 0.047 | 0.053 | 0.051 |
| 5 | normal | 1 | 1 | 1 | 1 | 30 | 30 | 0.052 | 0.052 | 0.057 | 0.05 |
| 6 | normal | 1 | 1 | 5 | 5 | 30 | 30 | 0.052 | 0.052 | 0.056 | 0.061 |
| 7 | normal | 1 | 1 | 2 | 3 | 30 | 30 | 0.058 | 0.058 | 0.061 | 0.054 |
| 8 | normal | 0.1 | 0.1 | 5 | 5 | 30 | 30 | 0.055 | 0.055 | 0.058 | 0.052 |
| 9 | normal | 1 | 1 | 1 | 1 | 50 | 50 | 0.047 | 0.047 | 0.05 | 0.053 |
| 10 | normal | 1 | 1 | 5 | 5 | 50 | 50 | 0.059 | 0.059 | 0.062 | 0.059 |
Showing 1 to 10 of 60 entries
5 / 10
| distribution | mu1 | mu2 | sd1 | sd2 | nsample1 | nsample2 | t_equal | t_notequal | z | wilcox |
|---|
distribution | mu1 | mu2 | sd1 | sd2 | nsample1 | nsample2 | t_equal | t_notequal | z | wilcox | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | normal | 1 | 5 | 1 | 1 | 20 | 20 | 1 | 1 | 1 | 1 |
| 2 | normal | 0.1 | 0.8 | 5 | 5 | 20 | 20 | 0.082 | 0.082 | 0.091 | 0.08 |
| 3 | normal | 3 | 10 | 7 | 3 | 20 | 20 | 1 | 1 | 1 | 1 |
| 4 | normal | 1 | 5 | 1 | 1 | 30 | 30 | 1 | 1 | 1 | 1 |
| 5 | normal | 0.1 | 0.8 | 5 | 5 | 30 | 30 | 0.086 | 0.086 | 0.089 | 0.081 |
| 6 | normal | 3 | 10 | 7 | 3 | 30 | 30 | 1 | 1 | 1 | 1 |
| 7 | normal | 1 | 5 | 1 | 1 | 50 | 50 | 1 | 1 | 1 | 1 |
| 8 | normal | 0.1 | 0.8 | 5 | 5 | 50 | 50 | 0.115 | 0.115 | 0.117 | 0.115 |
| 9 | normal | 3 | 10 | 7 | 3 | 50 | 50 | 1 | 1 | 1 | 1 |
| 10 | normal | 1 | 5 | 1 | 1 | 100 | 100 | 1 | 1 | 1 | 1 |
Showing 1 to 10 of 45 entries
6 / 10
Type I error
7 / 10
8 / 10
Statistical power
9 / 10
10 / 10