Via Scott Sumner comes the question “What took the ‘con’ out of econometrics?“. I bet many people question whether that actually occurred, usually those ignorant of math and/or economics. Other ignorant people also question the application of quantitative social science to the study of violent conflict. The answer to the aforementioned question is better identification through mechanisms like randomized experiments or “natural experiments”. I mentioned some heretics who fail to pay homage to randomized trials here. Among the party-poopers is a recent commenter here, Thorfinn, in “When Numbers Fail“. My view is that people are apt to fool themselves, and numbers can provide a constraint, though given sufficient ingenuity and lack of countervailing incentives they will tend to be undermined.
On a completely different note for people more interested in medieval history & philosophy of science than numbo-jumbo, check out this.
The title of this post courtesy of Penn Jillette.
UPDATE: Most of the trackbacks I get are spam, but I don’t bother turning them off anyway. This post got an amazing 130 of them, and actually took a good deal of processing time to clear them all out. They are now off for this post.
March 19, 2010 at 1:19 am
the increased transparency of the internet has not yet led to an increase in scientific rigor. perhaps this is because of the number of paywalls (sci journals) or simply because there aren’t enough people trained in rigorous analysis to be fact checking everything.
of course it always amazes me when people *don’t* adjust their priors upon speaking to someone (credible) with more hard data on a subject than themselves. people generally ask “what is credibility?”. it seems fairly straightforward: someone is credible if they filter evidence as rigorously as you do. of course this can lead to echo chambers, so in general you should be seeking out people *more* rigorous than you in any given field.
March 19, 2010 at 11:39 am
“the increased transparency of the internet has not yet led to an increase in scientific rigor.”
If it’s become much cheaper to supply that, then there could have been a fall in demand for it.
March 20, 2010 at 12:33 am
“the increased transparency of the internet has not yet led to an increase in scientific rigor”
Do we have good before-and-after evaluations?
“there aren’t enough people trained in rigorous analysis to be fact checking everything”
Not only that, but a lot of people just aren’t interested in it.
March 21, 2010 at 11:38 am
Here’s what I found:
“Pre-publication peer review on the Internet
Open peer review consists of making reviewers� comments public and/or inviting post-publication comments from readers, which are then published. Such a system was tested at the Medical Journal of Australia, where 60 (81%) authors and 150 (92%) reviewers agreed to take part in a study using website access for pre-publication peer review3. Although informal feedback from readers was positive, only 29% of comments led authors to change their manuscripts.”
from: http://www.nature.com/nature/peerreview/debate/nature05031.html
29% doesn’t sound like an *only* to me. it sounds excellent. but pre-publication peer review has not been widely adopted yet. by and large it seems that the same groups are submitting to the same journals as they were before the internet.