Over the past forty years of striving to make computers do what I want, I have gained a few hard-won truths. One is that we can be very poor critics of own ideas. We love those ideas; we poured our own blood, sweat and tears into them. Of course they must be right; of course the computer must have misunderstood what we intended to happen.
Well, no. Whether we refer to computer programs, science, politics or economics, when we get stuck, or even when we think everything is going swimmingly, we need to bring in other pairs of eyes to take a fresh look, to spot what we may have blocked ourselves from seeing.
As I write this, Brian Wansink has made the news in a very unfortunate manner; he unwittingly invited people to scrutinize his research and his writings, and the results were not pretty. Now, I must stress that his conclusions might be right; we don’t know one way or the other. What we do know is that he didn’t take care to prove his points; his methods were not up to the task.
Brian Wansink, bless him, is probably a good fellow who was trying to do the right thing. But he admits that he hasn’t kept up with what we now know as the Replication Crisis – the fact that many research papers, especially in the social sciences, cannot be replicated.
I say to anybody in search of the truth: invite criticism. Borrowing from the experiences of those who create and use Open Source Software, open up your data, your methods, your research; your little “tricks” and “kludges” and “smoothing” algorithms. If you had good, solid reasons for your tweaks, they’ll stand the light of scrutiny. As Eric Raymond put it, given enough eyeballs, all bugs are shallow – or perhaps it is better to say that problems are easily solved if you can find the right pair of eyes with the right expertise to examine them. Many people, it seems, assume that they already have sufficient expertise; they hoard their data, their algorithms, their thinking processes, and declare that the “science is settled” rather too easily.