No they didn't (part 1)

As today's unfortunate headline, I present, from Phys.org,

Study finds scientific reproducibility does not equate to scientific truth

I call this an unfortunate headline because it's usual with such statements that they don't accurately represent what was actually found.

The referenced study consisted of modeling various scientific communities with a variety of research strategies to discover a modeled "scientific truth"... in this case, the shape of a target polygon. Keep in mind that these were not actual scientific communities, but computer models. As with all such models, certain assumptions are made and biases can be introduced by the programmers. Such models may, but do not necessarily, accurately reflect reality. As is the way with models, the model is extremely simplified. Reality is more variable, and this can muddy the water.

For my part, a statement to be closely considered is the following:

"We found that, within the model, some research strategies that lead to reproducible results could actually slow down the scientific process, meaning reproducibility may not always be the best—or at least the only—indicator of good science," said Erkan Buzbas, U of I assistant professor in the College of Science, Department of Statistical Science and a co-author on the paper. "Insisting on reproducibility as the only criterion might have undesirable consequences for scientific progress."

Notice that what Buzbas says here does not support the headline... not by a long shot.  Buzbas isn't complaining about the insistence on reproducibility: rather, he's opining on the insistence on reproducibility alone... which in practice, nobody does.

We all know from numerous historical examples that scientific discoveries can be made by accident. The discoveries of X-rays and penicillin are two such examples. A careful reading of the above statement reveals nothing more or less.  If you are not rigorous in your method, then you might discover something new more quickly than otherwise. And indeed, the application of rigorous method may slow progress.

However, this says nothing about the value of reproducibility. "Innovative" (let's face it: sloppy) method might get you the correct answer by accident, but it doesn't prove a damned thing. The problem is that with irreproducible experiments, you can't show a definitive cause for the observation you witnessed. Any number of uncontrolled environmental or experimental factors might be the cause. This can skew the researchers' hypotheses and lead to the wrong explanations for observed effects. So long as the research is irreproducible, it has dubious explanatory value and no practical applications. Obviously, you can't reliably use a principle that you can't reliably reproduce. That part of science... the part that leads to engineering... is just common sense.

On the other hand, if you can control and reliably bring into existence an effect based on manipulation of predictable and consistent factors, then you are far more likely to have discovered the truth than if you were to have stumbled across an effect that no one else can observe or make happen.

Far from the implied message of this Phys.org article, "innovative experiments" do not supplant the value of rigorous, controlled experiments. Every accidental discovery must be followed by rigor if you want to claim that what you discovered is most likely the Truth. And if you cannot reproduce your results, you don't get to shelter under the Umbrella of Laziness and whine that your progress is being impeded.  Oh yes, Mr. Innovator, your discovery of Cold Fusion (or whatever) could change the world... but if and only if you can make it happen on demand.

The history of serendipitous discoveries alone tells us that it is not, nor has it ever been, innovation that is criticized when scientists talk about the replication crisis. Rather, it's the hyping and premature conclusions that are drawn when innovative experiments are not followed up with sound method.

If you truly "fucking love science", avoid