I have done many replications. That means checking somebody else’s empirical research using the same or better data and the same or improved analysis methods.[1],[2] My replications are listed below.
Replications aren’t good for the replicator’s career.[3] Replications take as much time as original research.[4] But replications offer none of the rewards in terms of developing and publicizing one’s own ideas or gathering citations.[5] It also risks upsetting the original researchers, which isn’t fun unless you enjoy annoying other people (which I don’t; otherwise, I could have earned a lot more money as a litigator). So why do I do it?
Because I want to stand on the shoulders of giants, not on a giant pile of manure. I can’t help it: I care about truth. That’s why I became an academic. What’s the point theorizing about a factoid that is likely just an error or randomness, intentional or not?
Many believe replications are an attack on the original researchers. They aren’t, or at least they shouldn’t be.[6] The point is to get the research right. We researchers all have a stake in this, and it’s all we should care about.
List of my Replications
(For code, data, and other additional information, see my Publications page.)
- Comment on “Temperature and Decisions: Evidence from 207,000 Court Cases, 14(4) AEJ: Applied Economics 519-528 (2022)
- Does the Supreme Court Really Not Apply Chevron When It Should?, 57 Int. Rev. L. & Econ. 81-89 (2019) (with N. Salmanowitz)
- Are Sleepy Punishers Really Harsh Punishers? Comment on Cho, Barnes, and Guanara (2017), 29 Psychological Science 1006-1009 (2018)
- The ‘Anti-Director Rights Index’ Revisited, 23 Rev. Fin. Stud. 467-486 (2010)
- Legal Origin, Civil Procedure, and the Quality of Contract Enforcement, 166 J. Institutional & Theoretical Econ. 149-165 (2010)
- On Inference When Using State Corporate Laws for Identification (HODP 1024/2019; ECGI FWP 644/2019) (not technically a replication because not targeted at any particular original study, but a methodological point that affects many papers simultaneously)
[1] For a systematic review of the role of replications and related issues, see, e.g., Garret Christensen & Edward Miguel, “Transparency, Reproducibility, and the Credibility of [Economics] Research,” Journal of Economic Literature 56(3):920-80. You can also watch a video of me discussing these topics at Berlin’s Institute for Advanced Studies (Wissenschaftskolleg), but unfortunately they asked me to do it in German.
[2] In experimental fields of research like psychology, “replication” tends to have the narrower meaning of repeating the experiment. In fields such as economics or law where most empirical work uses observational data, albeit selected to act as a “natural experiment,” replications are more likely to consist of repeating the analysis on the same data, be it using the exact procedures specified by the original researchers (checking reproducibility) or clearly improved methods (reanalysis).
[3] I am referring to the current state of social and some other sciences. In fields such as physics, replication of any novel result in various independent labs is standard. Most fields would love to be physics …
[4] Sometimes, the replicator can build on data collection etc. done by the original researcher, as in my replication of Eskridge and Baer. In that case, the replication is easier. However, in other cases, the replicator starts from scratch. Indeed, the replicator is likely to spend more time on data collection and analysis because doing it right is the whole point of the exercise; it is also the only way not to make a fool of oneself if one ends up disagreeing with the original researchers. Against all this, some say that replications don’t require the creativity of original research. Frankly, most empirical research isn’t that creative, at least not the type that I have replicated.
[5] If the replication confirms the original finding, the replication becomes at best a footnote in the literature. If the replication disconfirms the original finding, neither will tend to be cited going forward. A partial exception is that a famous original finding’s failure to replicate can become notorious in the literature and be remembered as such for a while. But that only happens if the replicator waits long enough for the original finding to become famous before attempting the replication. For science, that’s too late.
[6] Some replicators strike an unnecessarily aggressive tone, or tweak the analysis to increase the distance between their and the original results. In that case, the original researchers are right to be upset. In my replications, I have always striven to be as neutral and charitable as possible. Sometimes, editors or defensive reactions from the original researchers have forced me to say more about mistakes by the original researchers than I think was necessary for scientific progress.