Reproducibility in the Spotlight:

Psychology is still reeling from the results of a massive study of reproducibility which found that less than half of statistically significant findings recently published in the top psychology journals remain significant when independently reproduced. If this is new to you, Jesse Singal wrote a great summary about it here.

Biology is not exactly doing that much better. Ioannidis first highlighted shoddy research practices in biology with his aptly named paper Why Most Published Research Findings Are False.

The latest entry in this saga is discussed in this excellent article from Nature. In papers that rely on animal models, the results are too fragile and unreliable to be used as a basis for further research because the statistical design of the experiments is often critically flawed. This is unfortunately not a huge surprise: scientists at Bayer had already warned that published data had become progressively less reliable as basis for drug research.

In a previous editorial, Malcolm Macleod points out that:

The most reliable animal studies are those that: use randomization to eliminate systematic differences between treatment groups; induce the condition under investigation without knowledge of whether or not the animal will get the drug of interest; and assess the outcome in a blinded fashion. Studies that do not report these measures are much more likely to overstate the efficacy of interventions. Unfortunately, at best one in three publications follows these basic protections against bias. This suggests that authors, reviewers and editors accord them little importance.

For those that aren't familiar with experimental design - these aren't hyper advanced techniques. Those are all things which are taught in a 1st year graduate experimental design course and that every senior scientist should be very familiar with.

Animal Models In Science:

What makes this particularly frustrating is that in this case, the scientists aren't just wasting public money, but they are killing loads of animals for absolutely no public good. Until computational models massively improve, research with animal models is absolutely necessary, especially in the later stages, when we wish to validate promising drug targets or test drug safety.

In theory, the use of animal models in research in the US is strictly regulated by the NIH. In practice, while charges of animal cruelty are taken seriously, poor protocol design leading to a waste of model animals is very rarely sanctioned.

You Always Get What You Measure:

Why do really smart scientists make such stupid mistakes?

• Arrogance: Some oldschool PIs are so confident in their own theories that they see statistics not as a critical way to evaluate their data, but as a simple threshold (p < 0.05) they gotta cross to be able to publish. This is thankfully not very common, but if you go to any bioinformatics conference and talk over drinks to some of the junior people there, you'll hear all sorts of horror stories about being the only bioinformatician in a group and having to somehow come up with a statistical test that gives p<0.05.

• Misaligned Incentives: This is unfortunately very common. The currency of scientific careers are publications in high impact journals, and the easiest way to publish in those journals is to produce splashy results. Unfortunately, results are the one thing that scientists have no actual control over - if you execute a well designed experiment to test a reasonable hypothesis, the actual outcome is up to the Universe. By dis-proportionally rewarding splashy results, we punish the scientists that actually do research properly, since they will have a much lower fraction of positive results.

The irony that Nature, one of the most prestigious scientific journals in the world, acknowledges that scientists push splashy but unreliable findings (like say, these) instead of solid but boring ones is not lost on me. It reminds me of the time a Facebook executive complained that the Internet was drowning in shitty click-baity articles, when the product he was responsible for was the main driver of that behaviour.

It's important to add that the quest for high impact publications is not just motivated by a selfish desire for success. Postdocs are facing a very difficult job market and are desperate to have a high impact paper before applying for tenure track positions. PIs are under an immense amount of pressure by their institutions to obtain grants, and it's getting harder and harder.

Further, missing grants doesn't just hurt the career of the PI. Very few people in science have the luxury of fixed positions - even lab technicians and staff scientists are often funded by grants, and not getting a grant renewed can mean having to fire people people you've worked with for many years.