Either that, or it’s stellar. I don’t know, it’s just a guess. The point is that almost no one else knows, either.
Software is a mainstay of modern research. In the computing fields in particular, but also in others, custom software is frequently rolled for research-oriented projects. This software is typically used to gather or process the data on which the results of the research are based. For reference, these phases are highlighted in the (oversimplified) research process shown below.
Whereas the results of using such software beget many articles in The Literature, the source of the software itself is rarely published. This suggests an easy avenue of attack when attempting to dispute research results: Simply ask for the source. An antagonist may use the strategy shown below to attack the results.
It goes something like this: The published results of your research contradict my worldview. I don’t care if your research is actually valid. At this point I don’t even care what data you have. I am going to ask for the source of any software that you used to gather or process the data on which the results are based. If you can’t/won’t give me the source, I am going to claim that the results are made up. In order to disprove me, you now have to release the source.
If you do release the source, I am going to subject it to analysis tools like FindBugs or Clang, or tools such as those produced by companies like Coverity and Parasoft who make a good business out of finding software defects. I am likely to find hundreds, possibly thousands of defects in your source (because any non-trivial software system has bugs), at which point I am going to claim that the results are invalid because they were produced using untrustworthy software. In order to disprove me, you now have to demonstrate that no defect (or combination of defects) affected the results, a tedious if not impossible task.
If I don’t find any defects, then I am probably using the tools incorrectly. This is not a statement of the inability of researchers to produce quality software, but rather of the complexity inherent to software development. Regardless, I am now going to ask for your data.
If you can’t/won’t give me the data, again, I am going to claim that the results are made up. In order to disprove me, you now have to release the data. If you do release the data, I am going to attempt to use your software.
If your software was used to gather the data in some way that cannot be automated, such as may be the case with a user study, I’ll probably make the standard complaints about methods and sample size, but my grumbling would be effectively neutralized by correct sampling and good statistical techniques. If your software has a UI that determines or influences the data gathered, I am going to subject it to analysis techniques such as GOMS, which may enable me to make some weak argument about bias in the results.
On the other hand, if your software was used to gather the data in some way that could be automated, or if your software was used to subsequently process the data, something that should be automated, but you have provided no automation with either the source or the data, I will claim that the results are unverifiable. (Software developers may recognize this as a variant of automated testing.) As the complexity of processing the data increases, so too does the likelihood that a mistake of process produces incorrect results, even supposing perfect software. In order to disprove me, you now have to explicate the process and provide some assurance that it was followed precisely. As any QA engineer can attest, this is no easy task.
This leaves the scenario that you have provided me with both the source and the data, that the source (and possibly the user interface) passes rigorous analysis, and that processing the data is automated. I can’t claim anything negative about the results, so I will compare you to Hitler.
I was picking on academia in the title, but this problem is not specific to academia. The credibility of public research is at stake, in particular research in fields that are politically charged. The havoc wreaked by “Climategate” was due to the leaking of emails and other ancillary documents that had little to do with the actual research. Despite the media coverage, the criticism was poorly received except by like-minded opponents of climate science. What happens when these opponents obtain the relevant software and start tearing it apart, finding thousands of defects? What happens if that receives the same media coverage?
As an opponent of some field of research, my goal would not be to negate the research but to defund it. I wouldn’t need to negate the research. I wouldn’t need to persuade supporters to become opponents like myself. I would need only to sway public opinion enough to increase the political capital necessary to fund the research, such that the funding remains below a critical mass. I could do that, at least in part, with a credible argument that receives adequate media coverage. I submit that attacking research by means of its supporting software is such an argument.
This is the reality of the environment in which public research must be conducted. So what to do?
The more obvious solution is to publish (under an appropriate license) the source of any supporting software alongside the corresponding research results, be they positive or negative. Publishing source should be a push activity, not a pull activity. In other words, source should always be published alongside results, regardless of whether anyone asks for it. This allows the software to undergo continuous scrutiny, evaluation, and improvement. Coupled with automated processing of the data, results can be easily reproduced in response to changes to the software.
The less obvious solution is to use only software for which the source is already published. In this regard the importance of OSS to public research cannot be underestimated. Though I have been focusing on custom software, ostensibly written by the researcher, the arguments above apply to any closed-source software. Many fine companies sell proprietary packages to institutions for research purposes. However, use of these packages is inconsistent with one of the core tenets of public research:
Using closed-source software to gather or otherwise process the data upon which the results of research are based is a violation of the principal of full disclosure.
In other words, the software used to support public research should be open-source software. If the software is written specifically to support the research, it should be released as open-source software. Using software whose internal processes cannot themselves be observed and analyzed is tantamount to hiding some portion of a physical experiment and then declaring “trust me, I’m getting correct numbers from this thing”.
Not only is this kind of openness essential to the integrity of public research, but alongside additional effort (such as the automated processing of data) it is likely to blunt a latent tactic that otherwise has the potential to erode support for such research.