Pre-Registered Analysis Plan: Is the Inter-Researcher Variability in Social Scientific Results Explicable? An Adversarial Collaboration and Joint Effort to Parse Model and Estimate Dispersion

We plan a collaborative research analysis of the original ‘many analysts’ study by Silberzahn et al.1 testing whether European soccer referees were skin-tone biased in giving red cards. Their study concluded that the answer has great model dependency, and in this sense is unclear. Auspurg and Brüderl2 re-analysed the methods and data and argued the answer is clearer when attending to causal inference and outliers.

Researchers involved in the original study and follow-up agreed to an adversarial collaboration to better answer this pertinent research question and sources of variability in answers. They recruited other scholars to provide independent methodological and theoretical insights. The result is this study where together we jointly tackle the skin-tone bias hypothesis and the ‘parsing problem’ of identifying meaningful moderators that explain effect size dispersion across a multiverse of alternative specifications.

We build on Auspurg and Brüderl’s proposal that analysts in the original study had very different interpretations of the conceptual research question, leading them to different empirical research questions, which in turn produced results that are not directly comparable. This pre-analysis plan describes our motivation, methods, and plan to contextualize our findings in larger discussions of race bias, multiverse analysis parsing, and subjectivity in science.