Which test statistics is better? for two-sample hypothesis test

class: center, middle, inverse, title-slide

.title[
# Which test statistics is better? for two-sample hypothesis test
]
.subtitle[
## a simulation study
]
.author[
### Lin Yu
]
.date[
### 2023-03-28
]

---

# Distributions
we considered **normal ** and  **nonnormal ** (uniform, weibull) distributions

<img src="data:image/png;base64,#statistical_inference_simulation_study_files/figure-html/unnamed-chunk-1-1.png" style="display: block; margin: auto;" />
---
# Scenarios
We simulated different settings, simply put,         
- sample size varies from 10(small) to 1000(large);               
- different magnitude of diff between `\(\mu_1\)` and `\(\mu_2\)`,  `\(sd_1\)` and `\(sd_2\)`;        
- finally, both normal and nonnormal distribution were tested.

Details are noted in the following table:

|parameter|description|setting|
|---------|-----------|-------|
| `\(\mu_1\)` |mean of population 1|c(0.1,1,3,5)|
| `\(\mu_2\)` |mean of population 2|c(1,3,10)|
| `\(sd_1\)`  |standard deviation of ppl 1|c(1,5,10)|
| `\(sd_2\)`  |standard deviation of ppl 2|c(0.1,1,10)
| `\(nsample_1\)` |sample size of ppl 1 |c(10,20,30,50,100,1000)
| `\(nsample_2\)` |sample size of ppl 2 |c(10,20,30,100,1000)
| `\(dist\)`| study population distribution|normal, uniform, and weibull

---
# Pseudo-code

```r
function(seed,mu1,mu2,sd1,sd2,nsample1,nsample2,dist,Nsim){
  #### for loop over all paramaters
  for (parameter in 1:length(mu1)){
        -------------------------------------------------------
        #### calculate the test stats and pval for Nsim times
         for(i in 1:Nsim){
          dat1 = 
          dat2 = 
          test_stats = 
          pval = 
        }
        --------------------------------------------------------
        #### calculate type I error and power by counting
          pval/Nsim
    
  }
}
```

---
# Results

```r
##wide table to long table
library(tidyr)
```

```
## Warning: 程辑包'tidyr'是用R版本4.1.3 来建造的
```

```r
long <- final_res %>%
  pivot_longer(
    cols = `t_equal`:`wilcox`, 
    names_to = "test_stat",
    values_to = "value",
  )
```

<div class="datatables html-widget html-fill-item-overflow-hidden html-fill-item" id="htmlwidget-10abffa22c9b773fd5d0" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-10abffa22c9b773fd5d0">{"x":{"filter":"none","vertical":false,"extensions":["Buttons"],"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60"],["normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull"],["1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1"],["1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1","1","1","1","0.1"],["1","5","2","5","1","5","2","5","1","5","2","5","1","5","2","5","1","5","2","5","1","5","2","5","1","5","2","5","1","5","2","5","1","5","2","5","1","5","2","5","1","5","2","5","1","5","2","5","1","5","2","5","1","5","2","5","1","5","2","5"],["1","5","3","5","1","5","3","5","1","5","3","5","1","5","3","5","1","5","3","5","1","5","3","5","1","5","3","5","1","5","3","5","1","5","3","5","1","5","3","5","1","5","3","5","1","5","3","5","1","5","3","5","1","5","3","5","1","5","3","5"],["20","20","20","20","30","30","30","30","50","50","50","50","100","100","100","100","1000","1000","1000","1000","20","20","20","20","30","30","30","30","50","50","50","50","100","100","100","100","1000","1000","1000","1000","20","20","20","20","30","30","30","30","50","50","50","50","100","100","100","100","1000","1000","1000","1000"],["20","20","20","20","30","30","30","30","50","50","50","50","100","100","100","100","1000","1000","1000","1000","20","20","20","20","30","30","30","30","50","50","50","50","100","100","100","100","1000","1000","1000","1000","20","20","20","20","30","30","30","30","50","50","50","50","100","100","100","100","1000","1000","1000","1000"],["0.058","0.049","0.053","0.047","0.052","0.052","0.058","0.055","0.047","0.059","0.048","0.045","0.054","0.038","0.05","0.056","0.055","0.051","0.052","0.04","0.04","0.05","0.052","0.062","0.045","0.044","0.065","0.046","0.059","0.055","0.051","0.051","0.048","0.059","0.051","0.05","0.055","0.05","0.052","0.05","0.052","0.052","0.047","0.065","0.05","0.059","0.075","0.065","0.087","0.072","0.094","0.08","0.156","0.146","0.154","0.153","0.914","0.913","0.897","0.922"],["0.058","0.049","0.053","0.047","0.052","0.052","0.058","0.055","0.047","0.059","0.048","0.045","0.054","0.038","0.05","0.056","0.055","0.051","0.052","0.04","0.04","0.05","0.052","0.062","0.045","0.044","0.065","0.046","0.059","0.055","0.051","0.051","0.048","0.059","0.051","0.05","0.055","0.05","0.052","0.05","0.052","0.052","0.047","0.065","0.05","0.059","0.075","0.065","0.087","0.072","0.094","0.08","0.156","0.146","0.154","0.153","0.914","0.913","0.897","0.922"],["0.065","0.056","0.054","0.053","0.057","0.056","0.061","0.058","0.05","0.062","0.049","0.047","0.056","0.042","0.051","0.057","0.055","0.051","0.053","0.041","0.043","0.057","0.059","0.071","0.051","0.052","0.072","0.051","0.063","0.059","0.053","0.054","0.049","0.06","0.054","0.05","0.055","0.05","0.052","0.05","0.059","0.065","0.057","0.076","0.059","0.066","0.082","0.075","0.092","0.078","0.1","0.088","0.159","0.15","0.156","0.157","0.914","0.913","0.897","0.922"],["0.059","0.047","0.052","0.051","0.05","0.061","0.054","0.052","0.053","0.059","0.054","0.042","0.055","0.049","0.043","0.048","0.059","0.058","0.047","0.043","0.042","0.049","0.051","0.063","0.043","0.049","0.068","0.046","0.059","0.058","0.047","0.047","0.048","0.058","0.05","0.053","0.054","0.049","0.052","0.049","0.069","0.083","0.068","0.081","0.094","0.109","0.114","0.096","0.147","0.125","0.126","0.147","0.204","0.202","0.21","0.201","0.945","0.929","0.944","0.94"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>distribution<\/th>\n      <th>mu1<\/th>\n      <th>mu2<\/th>\n      <th>sd1<\/th>\n      <th>sd2<\/th>\n      <th>nsample1<\/th>\n      <th>nsample2<\/th>\n      <th>t_equal<\/th>\n      <th>t_notequal<\/th>\n      <th>z<\/th>\n      <th>wilcox<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"dom":"Blfrtip","buttons":["copy","csv","excel","pdf","print"],"scrollX":true,"lengthMenu":[[10,25,50,-1],["10","25","50","All"]],"order":[],"autoWidth":false,"orderClasses":false,"columnDefs":[{"orderable":false,"targets":0}]}},"evals":[],"jsHooks":[]}</script>

---

<div class="datatables html-widget html-fill-item-overflow-hidden html-fill-item" id="htmlwidget-42347b1dbb574c493c43" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-42347b1dbb574c493c43">{"x":{"filter":"none","vertical":false,"extensions":["Buttons"],"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45"],["normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","uniform","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull","weibull"],["1","0.1","3","1","0.1","3","1","0.1","3","1","0.1","3","1","0.1","3","1","0.1","3","1","0.1","3","1","0.1","3","1","0.1","3","1","0.1","3","1","0.1","3","1","0.1","3","1","0.1","3","1","0.1","3","1","0.1","3"],["5","0.8","10","5","0.8","10","5","0.8","10","5","0.8","10","5","0.8","10","5","0.8","10","5","0.8","10","5","0.8","10","5","0.8","10","5","0.8","10","5","0.8","10","5","0.8","10","5","0.8","10","5","0.8","10","5","0.8","10"],["1","5","7","1","5","7","1","5","7","1","5","7","1","5","7","1","5","7","1","5","7","1","5","7","1","5","7","1","5","7","1","5","7","1","5","7","1","5","7","1","5","7","1","5","7"],["1","5","3","1","5","3","1","5","3","1","5","3","1","5","3","1","5","3","1","5","3","1","5","3","1","5","3","1","5","3","1","5","3","1","5","3","1","5","3","1","5","3","1","5","3"],["20","20","20","30","30","30","50","50","50","100","100","100","1000","1000","1000","20","20","20","30","30","30","50","50","50","100","100","100","1000","1000","1000","20","20","20","30","30","30","50","50","50","100","100","100","1000","1000","1000"],["20","20","20","30","30","30","50","50","50","100","100","100","1000","1000","1000","20","20","20","30","30","30","50","50","50","100","100","100","1000","1000","1000","20","20","20","30","30","30","50","50","50","100","100","100","1000","1000","1000"],["1","0.082","1","1","0.086","1","1","0.115","1","1","0.168","1","1","0.874","1","1","0.963","1","1","0.997","1","1","1","1","1","1","1","1","1","1","0.056","0.051","0.047","0.064","0.075","0.059","0.074","0.085","0.087","0.154","0.132","0.161","0.917","0.921","0.919"],["1","0.082","1","1","0.086","1","1","0.115","1","1","0.168","1","1","0.874","1","1","0.963","1","1","0.997","1","1","1","1","1","1","1","1","1","1","0.056","0.051","0.047","0.064","0.075","0.059","0.074","0.085","0.087","0.154","0.132","0.161","0.917","0.921","0.919"],["1","0.091","1","1","0.089","1","1","0.117","1","1","0.172","1","1","0.874","1","1","0.971","1","1","0.997","1","1","1","1","1","1","1","1","1","1","0.065","0.056","0.053","0.068","0.079","0.063","0.077","0.089","0.091","0.158","0.137","0.162","0.917","0.921","0.92"],["1","0.08","1","1","0.081","1","1","0.115","1","1","0.166","1","1","0.864","1","1","0.92","1","1","0.99","1","1","1","1","1","1","1","1","1","1","0.087","0.098","0.081","0.101","0.099","0.114","0.126","0.125","0.132","0.239","0.219","0.187","0.939","0.937","0.949"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>distribution<\/th>\n      <th>mu1<\/th>\n      <th>mu2<\/th>\n      <th>sd1<\/th>\n      <th>sd2<\/th>\n      <th>nsample1<\/th>\n      <th>nsample2<\/th>\n      <th>t_equal<\/th>\n      <th>t_notequal<\/th>\n      <th>z<\/th>\n      <th>wilcox<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"dom":"Blfrtip","buttons":["copy","csv","excel","pdf","print"],"scrollX":true,"lengthMenu":[[10,25,50,-1],["10","25","50","All"]],"order":[],"autoWidth":false,"orderClasses":false,"columnDefs":[{"orderable":false,"targets":0}]}},"evals":[],"jsHooks":[]}</script>

---
class: center, reverse, middle
# Type I error
---
![](data:image/png;base64,#statistical_inference_simulation_study_files/figure-html/unnamed-chunk-10-1.png)

---
class: center, reverse, middle
# Statistical power
---
![](data:image/png;base64,#statistical_inference_simulation_study_files/figure-html/unnamed-chunk-11-1.png)