Inter-rater Repeatability

This vignette documents the analysis of the data gathered from surveying 5 journal articles, which all five authors evaluated for an inter-rater comparison. Five articles were evaluated by all five authors to understand the differences between ratings assigned by each of the paper’s authors. All original scores are kept with corrections being made to the software_avail column in software_avail_corrected (by AHS) based on the software that was recorded as being used in each paper. This step removes some disagreement between the ratings.

Set-up Workspace

Load libraries used and setting the ggplot2 theme for the document.

library(ggridges)
library(irr)
library(patchwork)
library(psych)
library(tidyverse)
library(pander)
library(Reproducibility.in.Plant.Pathology)

theme_set(theme_classic())

Import Data

Import the data.

rrpp <- import_interrater_scores()

Article Classifications

Check differences between evaluators in classifying articles as having a molecular focus or fundamental or applied research.

a <- ggplot(rrpp, aes(y = doi,
                      fill = molecular)) +
  geom_bar() +
  scale_fill_grey(name = "") +
  ylab("DOI") +
  xlab("Count") +
  ggtitle("Molecular Focus") +
  theme(legend.position = "top")

b <- ggplot(rrpp, aes(y = doi,
                      fill = art_class)) +
  geom_bar() +
  scale_fill_grey(name = "") +
  xlab("Count") +
  ylab("") +
  theme(
    axis.text.y = element_blank(),
    axis.ticks.y = element_blank()
  ) +
  ggtitle("Article Class") +
  theme(legend.position = "top")

p <- (a | b)
p

Reproducibility Criteria

Visualise differences between evaluators for the four criteria used to score reproducibility for each paper.

c <- ggplot(rrpp,
            aes(y = doi,
                x = software_cite)) +
  ylab("DOI") +
  xlab("") +
  geom_density_ridges() +
  ggtitle("Software Cited")

d <- ggplot(rrpp,
            aes(y = doi,
                x = software_avail)) +
  ylab("") +
  xlab("") +
  geom_density_ridges() +
  theme(
    axis.text.y = element_blank(),
    axis.ticks.y = element_blank()
  ) +
  ggtitle("Software Available")

e <- ggplot(rrpp,
            aes(y = doi,
                x = comp_mthds_avail)) +
  geom_density_ridges() +
  ylab("DOI") +
  xlab("Score") +
  ggtitle("Comp. Methods Available")

f <- ggplot(rrpp,
            aes(y = doi,
                x = data_avail)) +
  ylab("") +
  xlab("Score") +
  theme(axis.text.y = element_blank(),
        axis.ticks.y = element_blank()) +
  geom_density_ridges() +
  ggtitle("Data Available")

p <- (c | d) / (e | f)
p
#> Picking joint bandwidth of 1.98
#> Picking joint bandwidth of 1.97
#> Picking joint bandwidth of 0.658
#> Picking joint bandwidth of 0.652

Software Used

We can check if most authors found the same software in the papers that were evaluated. Perfect agreement is a 5, below that the software was not clearly and easily detected by the evaluator.

rrpp_software <-
  rrpp %>%
  transform(software_used = strsplit(software_used_cleaned, ",")) %>%
  unnest(software_used) %>%
  mutate(software_used = trimws(software_used)) %>%
  mutate(software_used = toupper(software_used))

tab <- table(rrpp_software$software_used)
tab_s <- as.data.frame(sort(tab))
tab_s <-
  tab_s %>%
  arrange(desc(Freq)) %>%
  rename("Software" = "Var1", "Frequency" = "Freq")

pander(tab_s)

Software	Frequency
GELCOMPAR II	5
GENALEX	5
PERMTEST	5
POPGENE	5
R	5
SAS	5
SDS	5
SPSS	5
STATISTICA	5
STRUCTURE	5
DARWIN	4
NTSYS	3
EXCEL	1
IDT PRIMER DESIGNER TOOL	1

Agreement Stats

Here will compute a simple percentage agreement between the reviewers using the irr package.

Software Citations

Percent agreement of software citations criteria rankings.

Agreement

rrpp %>%
  select(doi, assignee, software_cite) %>%
  pivot_wider(names_from = assignee, values_from = software_cite) %>%
  select(Adam, Emerson, Kaique, Nik, Zach) %>%
  as.matrix() %>%
  agree() %>%
  pander()

method: Percentage agreement (Tolerance=0)
subjects: 5
raters: 5
irr.name: %-agree
value: 0

Fleiss’ Kappa

rrpp %>%
  select(doi, assignee, software_cite) %>%
  pivot_wider(names_from = assignee, values_from = software_cite) %>%
  select(Adam, Emerson, Kaique, Nik, Zach) %>%
  as.matrix() %>%
  kappam.fleiss() %>%
  pander()

method: Fleiss’ Kappa for m Raters
subjects: 5
raters: 5
irr.name: Kappa
value: 0.02457
stat.name: z
statistic: 0.2342
p.value: 0.8148

Software Availability

Percent agreement of software availability criteria rankings.

Agreement

rrpp %>%
  select(doi, assignee, software_avail) %>%
  pivot_wider(names_from = assignee, values_from = software_avail) %>%
  select(Adam, Emerson, Kaique, Nik, Zach) %>%
  as.matrix() %>%
  agree() %>%
  pander()

method: Percentage agreement (Tolerance=0)
subjects: 5
raters: 5
irr.name: %-agree
value: 0

Fleiss’ Kappa

rrpp %>%
  select(doi, assignee, software_avail) %>%
  pivot_wider(names_from = assignee, values_from = software_avail) %>%
  select(Adam, Emerson, Kaique, Nik, Zach) %>%
  as.matrix() %>%
  kappam.fleiss() %>%
  pander()

method: Fleiss’ Kappa for m Raters
subjects: 5
raters: 5
irr.name: Kappa
value: 0.0165
stat.name: z
statistic: 0.1627
p.value: 0.8707

Software Availability Corrected

Percent agreement of corrected software availability criteria rankings.

Agreement

rrpp %>%
  select(doi, assignee, software_avail_corrected) %>%
  pivot_wider(names_from = assignee, values_from = software_avail_corrected) %>%
  select(Adam, Emerson, Kaique, Nik, Zach) %>%
  as.matrix() %>%
  agree() %>%
  pander()

method: Percentage agreement (Tolerance=0)
subjects: 5
raters: 5
irr.name: %-agree
value: 100

Fleiss’ Kappa

rrpp %>%
  select(doi, assignee, software_avail_corrected) %>%
  pivot_wider(names_from = assignee, values_from = software_avail_corrected) %>%
  select(Adam, Emerson, Kaique, Nik, Zach) %>%
  as.matrix() %>%
  kappam.fleiss() %>%
  pander()

method: Fleiss’ Kappa for m Raters
subjects: 5
raters: 5
irr.name: Kappa
value: 1
stat.name: z
statistic: 7.071
p.value: 1.537e-12

Data Availability

Agreement

Percent agreement of data availability criteria rankings.

rrpp %>%
  select(doi, assignee, data_avail) %>%
  pivot_wider(names_from = assignee, values_from = data_avail) %>%
  select(Adam, Emerson, Kaique, Nik, Zach) %>%
  as.matrix() %>%
  agree() %>%
  pander()

method: Percentage agreement (Tolerance=0)
subjects: 5
raters: 5
irr.name: %-agree
value: 100

Fleiss’ Kappa

rrpp %>%
  select(doi, assignee, data_avail) %>%
  pivot_wider(names_from = assignee, values_from = data_avail) %>%
  select(Adam, Emerson, Kaique, Nik, Zach) %>%
  as.matrix() %>%
  kappam.fleiss() %>%
  pander()

method: Fleiss’ Kappa for m Raters
subjects: 5
raters: 5
irr.name: Kappa
value: NA
stat.name: z
statistic: NA
p.value: NA

Computational Availability

Agreement

Percent agreement of computational methods availability criteria rankings.

rrpp %>%
  select(doi, assignee, comp_mthds_avail) %>%
  pivot_wider(names_from = assignee, values_from = comp_mthds_avail) %>%
  select(Adam, Emerson, Kaique, Nik, Zach) %>%
  as.matrix() %>%
  agree() %>%
  pander()

method: Percentage agreement (Tolerance=0)
subjects: 4
raters: 5
irr.name: %-agree
value: 100

Fleiss’ Kappa

rrpp %>%
  select(doi, assignee, comp_mthds_avail) %>%
  pivot_wider(names_from = assignee, values_from = comp_mthds_avail) %>%
  select(Adam, Emerson, Kaique, Nik, Zach) %>%
  as.matrix() %>%
  kappam.fleiss() %>%
  pander()

method: Fleiss’ Kappa for m Raters
subjects: 4
raters: 5
irr.name: Kappa
value: NA
stat.name: z
statistic: NA
p.value: NA

Colophon

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.2.2 (2022-10-31)
#>  os       macOS Ventura 13.1
#>  system   aarch64, darwin20
#>  ui       X11
#>  language en
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Australia/Perth
#>  date     2023-01-04
#>  pandoc   2.19.2 @ /opt/homebrew/bin/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package                            * version date (UTC) lib source
#>  assertthat                           0.2.1   2019-03-21 [2] CRAN (R 4.2.2)
#>  backports                            1.4.1   2021-12-13 [2] CRAN (R 4.2.2)
#>  broom                                1.0.2   2022-12-15 [2] CRAN (R 4.2.2)
#>  bslib                                0.4.2   2022-12-16 [2] CRAN (R 4.2.2)
#>  cachem                               1.0.6   2021-08-19 [2] CRAN (R 4.2.2)
#>  cellranger                           1.1.0   2016-07-27 [2] CRAN (R 4.2.2)
#>  cli                                  3.5.0   2022-12-20 [2] CRAN (R 4.2.2)
#>  colorspace                           2.0-3   2022-02-21 [2] CRAN (R 4.2.2)
#>  crayon                               1.5.2   2022-09-29 [2] CRAN (R 4.2.2)
#>  curl                                 4.3.3   2022-10-06 [2] CRAN (R 4.2.2)
#>  DBI                                  1.1.3   2022-06-18 [2] CRAN (R 4.2.2)
#>  dbplyr                               2.2.1   2022-06-27 [2] CRAN (R 4.2.2)
#>  desc                                 1.4.2   2022-09-08 [2] CRAN (R 4.2.2)
#>  digest                               0.6.31  2022-12-11 [2] CRAN (R 4.2.2)
#>  dplyr                              * 1.0.10  2022-09-01 [2] CRAN (R 4.2.0)
#>  ellipsis                             0.3.2   2021-04-29 [2] CRAN (R 4.2.2)
#>  evaluate                             0.19    2022-12-13 [2] CRAN (R 4.2.2)
#>  fansi                                1.0.3   2022-03-24 [2] CRAN (R 4.2.2)
#>  farver                               2.1.1   2022-07-06 [2] CRAN (R 4.2.2)
#>  fastmap                              1.1.0   2021-01-25 [2] CRAN (R 4.2.2)
#>  forcats                            * 0.5.2   2022-08-19 [2] CRAN (R 4.2.2)
#>  fs                                   1.5.2   2021-12-08 [2] CRAN (R 4.2.2)
#>  gargle                               1.2.1   2022-09-08 [2] CRAN (R 4.2.0)
#>  generics                             0.1.3   2022-07-05 [2] CRAN (R 4.2.2)
#>  ggplot2                            * 3.4.0   2022-11-04 [2] CRAN (R 4.2.0)
#>  ggridges                           * 0.5.4   2022-09-26 [2] CRAN (R 4.2.1)
#>  glue                                 1.6.2   2022-02-24 [2] CRAN (R 4.2.2)
#>  googledrive                          2.0.0   2021-07-08 [2] CRAN (R 4.2.0)
#>  googlesheets4                        1.0.1   2022-08-13 [2] CRAN (R 4.2.0)
#>  gtable                               0.3.1   2022-09-01 [2] CRAN (R 4.2.2)
#>  haven                                2.5.1   2022-08-22 [2] CRAN (R 4.2.2)
#>  highr                                0.10    2022-12-22 [2] CRAN (R 4.2.2)
#>  hms                                  1.1.2   2022-08-19 [2] CRAN (R 4.2.2)
#>  htmltools                            0.5.4   2022-12-07 [2] CRAN (R 4.2.2)
#>  httr                                 1.4.4   2022-08-17 [2] CRAN (R 4.2.0)
#>  irr                                * 0.84.1  2019-01-26 [2] CRAN (R 4.2.2)
#>  jquerylib                            0.1.4   2021-04-26 [2] CRAN (R 4.2.2)
#>  jsonlite                             1.8.4   2022-12-06 [2] CRAN (R 4.2.2)
#>  knitr                                1.41    2022-11-18 [2] CRAN (R 4.2.0)
#>  labeling                             0.4.2   2020-10-20 [2] CRAN (R 4.2.2)
#>  lattice                              0.20-45 2021-09-22 [2] CRAN (R 4.2.2)
#>  lifecycle                            1.0.3   2022-10-07 [2] CRAN (R 4.2.2)
#>  lpSolve                            * 5.6.17  2022-10-10 [2] CRAN (R 4.2.2)
#>  lubridate                            1.9.0   2022-11-06 [2] CRAN (R 4.2.2)
#>  magrittr                             2.0.3   2022-03-30 [2] CRAN (R 4.2.2)
#>  memoise                              2.0.1   2021-11-26 [2] CRAN (R 4.2.2)
#>  mnormt                               2.1.1   2022-09-26 [2] CRAN (R 4.2.2)
#>  modelr                               0.1.10  2022-11-11 [2] CRAN (R 4.2.2)
#>  munsell                              0.5.0   2018-06-12 [2] CRAN (R 4.2.2)
#>  nlme                                 3.1-161 2022-12-15 [2] CRAN (R 4.2.2)
#>  pander                             * 0.6.5   2022-03-18 [2] CRAN (R 4.2.2)
#>  patchwork                          * 1.1.2   2022-08-19 [2] CRAN (R 4.2.0)
#>  pillar                               1.8.1   2022-08-19 [2] CRAN (R 4.2.2)
#>  pkgconfig                            2.0.3   2019-09-22 [2] CRAN (R 4.2.2)
#>  pkgdown                              2.0.7   2022-12-14 [2] CRAN (R 4.2.2)
#>  psych                              * 2.2.9   2022-09-29 [2] CRAN (R 4.2.2)
#>  purrr                              * 1.0.0   2022-12-20 [2] CRAN (R 4.2.2)
#>  R6                                   2.5.1   2021-08-19 [2] CRAN (R 4.2.2)
#>  ragg                                 1.2.4   2022-10-24 [2] CRAN (R 4.2.2)
#>  Rcpp                                 1.0.9   2022-07-08 [2] CRAN (R 4.2.2)
#>  readODS                              1.7.0   2020-07-10 [2] CRAN (R 4.2.2)
#>  readr                              * 2.1.3   2022-10-01 [2] CRAN (R 4.2.2)
#>  readxl                               1.4.1   2022-08-17 [2] CRAN (R 4.2.2)
#>  reprex                               2.0.2   2022-08-17 [2] CRAN (R 4.2.1)
#>  Reproducibility.in.Plant.Pathology * 1.0.0   2023-01-04 [1] local
#>  rlang                                1.0.6   2022-09-24 [2] CRAN (R 4.2.2)
#>  rmarkdown                            2.19    2022-12-15 [2] CRAN (R 4.2.2)
#>  rprojroot                            2.0.3   2022-04-02 [2] CRAN (R 4.2.2)
#>  rstudioapi                           0.14    2022-08-22 [2] CRAN (R 4.2.2)
#>  rvest                                1.0.3   2022-08-19 [2] CRAN (R 4.2.0)
#>  sass                                 0.4.4   2022-11-24 [2] CRAN (R 4.2.0)
#>  scales                               1.2.1   2022-08-20 [2] CRAN (R 4.2.2)
#>  sessioninfo                          1.2.2   2021-12-06 [2] CRAN (R 4.2.2)
#>  sodium                               1.2.1   2022-06-11 [2] CRAN (R 4.2.2)
#>  stringi                              1.7.8   2022-07-11 [2] CRAN (R 4.2.2)
#>  stringr                            * 1.5.0   2022-12-02 [2] CRAN (R 4.2.2)
#>  systemfonts                          1.0.4   2022-02-11 [2] CRAN (R 4.2.2)
#>  textshaping                          0.3.6   2021-10-13 [2] CRAN (R 4.2.2)
#>  tibble                             * 3.1.8   2022-07-22 [2] CRAN (R 4.2.2)
#>  tidyr                              * 1.2.1   2022-09-08 [2] CRAN (R 4.2.0)
#>  tidyselect                           1.2.0   2022-10-10 [2] CRAN (R 4.2.2)
#>  tidyverse                          * 1.3.2   2022-07-18 [2] CRAN (R 4.2.0)
#>  timechange                           0.1.1   2022-11-04 [2] CRAN (R 4.2.2)
#>  tzdb                                 0.3.0   2022-03-28 [2] CRAN (R 4.2.2)
#>  utf8                                 1.2.2   2021-07-24 [2] CRAN (R 4.2.2)
#>  vctrs                                0.5.1   2022-11-16 [2] CRAN (R 4.2.2)
#>  withr                                2.5.0   2022-03-03 [2] CRAN (R 4.2.2)
#>  xfun                                 0.36    2022-12-21 [2] CRAN (R 4.2.2)
#>  xml2                                 1.3.3   2021-11-30 [2] CRAN (R 4.2.2)
#>  yaml                                 2.3.6   2022-10-18 [2] CRAN (R 4.2.2)
#> 
#>  [1] /private/var/folders/hc/tft3s5bn48gb81cs99mycyf00000gn/T/RtmpwnOTN6/temp_libpath4da5ab32fec
#>  [2] /Users/adamsparks/Library/R/arm64/4.2/library
#>  [3] /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Adam H. Sparks

2023-01-04

Set-up Workspace

Import Data

Article Classifications

Reproducibility Criteria

Software Used

Agreement Stats

Software Citations

Agreement

Fleiss’ Kappa

Software Availability

Agreement

Fleiss’ Kappa

Software Availability Corrected

Agreement

Fleiss’ Kappa

Data Availability

Agreement

Fleiss’ Kappa

Computational Availability

Agreement

Fleiss’ Kappa

Colophon