Stardate 20020123.1200

(On Screen): One of the big hazards in research is that if you have a preconceived notion of what you're supposed to find, then there's a good chance that you'll find it even if it isn't there. We are not perfect observers. We tend to see what we expect to see. When we know the answer, then confirming evidence will look large and contradicting evidence will be deemphasized.

There's a classic joke about this: Two students were asked to prove or disprove the statement, "All odd numbers are prime." The student of mathematics said, "1 is prime, 3 is prime, 5 is prime, 7 is prime, 9 is not prime; therefore the statement is false." The physic student said "1 is prime, 3 is prime, 5 is prime, 7 is prime, 9 is experimental error, 11 is prime, 13 is prime; the statement is true."

Physics is less prone to this kind of thing than some fields, and generally speaking in the 20th century Physics hasn't made these kinds of whoppers. Cold Fusion is the classic example. Pons and Fleischmann went public with their claim, and physicists all over the world jumped on it. It was exciting as hell. I remember reading the debate in sci.physics as people wrestled with the concept and attempted to determine whether it made sense in terms of quantum mechanics, and what other kinds of effects would be observed. (Conclusion: huge numbers of neutrons would be emitted. If their cells really were generating excess power at the rates they claimed, they should both have been dead from radiation poisoning.)

More important, though, was that other groups around the world immediately attempted to duplicate their results -- and failed. And they kept on failing. There are still true believers, and there are still people doing research on it. But I think the general consensus now is that Pons and Fleischmann were not attempting to perpetrate a fraud; rather, they had rushed into print with preliminary results so as to establish primacy in case it became possible to file patents. They were seduced by dollar signs; if their result had been real, it would have been fantastically valuable.

Science is a human activity, and humans will make mistakes. It can be routinely proved that pollsters who have a preconceived notion of what they will find will actually find it. Commission two groups of students to go out and interview people, and tell them to test a certain attitude. Casually mention what you think the result will be, but make it opposite for the two groups. And when they come back with their results, you'll find that they polarize along the lines you set out, even though the students thought they were being honest.

Tell someone who believes in ESP that they are watching dice being rolled under the control of someone with TK who is trying to increase the number of 7's being rolled, and then tell them to log the dice rolls, and when you tally their results there will be a statistically disproportionate number of 7's in their log. An honest accounting of the same sequence of dice rolls will show no such anomaly.

These people are not lying. They're reporting what they actually think they saw. But what we see is filtered through our preconceptions. That's why science has established certain elaborate rituals to try to eliminate this kind of bias. (And honest studies using those rituals have demonstrated the effects I just mentioned.)

The best science starts with an anomalous fact and tries to explain it. The researchers don't know what they'll conclude; they just try to explain what they've seen. That kind of process can yield quite revolutionary results. The Alvarez impact theory began with an anomalously large amount of iridium in the K-T boundary layer clay, for instance, and Luis Alvarez considered and rejected a large number of explanations before he and his team settled on the idea of an impact. All the other explanations were shot down upon short consideration, but the impact theory appeared to be worth further study. That is science at its best.

In a sense, Cold Fusion showed the other side of science at its best. To their credit, Pons and Fleischmann revealed enough about what they were doing to permit others to attempt to duplicate their experiments, and when the others failed, the ultimate conclusion was "experimental error". Scientists collectively took the challenge seriously; they didn't reject the proposal out of hand. This was no "Heretics! Away from our temple!" reaction. Rather, they went ahead and tried to check it to see if it was true. It turned out not to be.

On the other hand, scientists in numerous fields also immediately took the Alvarez theory and started to test it. It was a very fruitful theory; it predicted many things which could be tested. And as people looked at those predictions, they kept being sustained by direct research. It's accepted now that it really happened. (Finding the impact crater pretty much sealed it. But there are still people working to disprove it, and that is healthy too. They may yet succeed, but I'm doubtful.)

Social science has not done as well. One problem is that too much of social science's research has been done by people with an agenda, who know the answer ahead of time and set out to prove it. Randolph sends me an article about two recent cases of what appears to have been deliberate falsification in major social science studies.

On the scale of professorial failings, publishing fraudulent data is a far more deadly sin t