I lowered my estimation of Russ Roberts and did the reverse for Robin Hanson after listening to this episode of EconTalk. Roberts says he wants to discourage people from being persuaded by empirical arguments. I would like to discourage people from being persuaded by Roberts in that case. Here is my rank ordering of persuasive evidence:
1: Controlled, replicated double-blind experiments, or (even better) meta-studies of such
2: Non-manipulated (meant in a non pejorative sense) non-experimental empirical evidence
2.5: Case studies (may be considered a subset of 2)
3: Simple regressions/correlations with few controls from non-experimental empirical evidence
3.5: Same as above, but with more steps/complications
4: Mathematical models
5: Verbal arguments relying on logic
6: Verbal arguments relying on analogies
7: Catch-all for things I haven’t mentioned
Last: Proof by assertion
Roberts and I agree that simple empirical evidence is better than fancy complicated analysis of data. The reason is that the more additional input the analyst brings to bear, the closer their analysis gets to logical argument. The point about meta-studies should also be applied to numbers other than 1, but I didn’t feel like repeating myself. One should resort to a low number only when a higher number is not available.
UPDATE: A reader asks Andrew Gelman for his favorite example of an experimental study debunking a causal relation found in a regression.