Mon 26 Oct 2015 15:30 - 15:50 at Grand Station 5 - Group Discussion Chair(s): Joshua Sunshine, Craig Anslow, Thomas LaToza

While it looks like controlled trials with human envolvement are more and more often applied in software science, there is the problem that there are hardly any explicitly documented community standards that need to be addressed by such trials. This leads to a number of problems: experimenters cannot be sure whether an experiment they perform does represent the current state-of-the-art, reviewers have no guidelines to check whether a critique they have in mind is valid or not, and readers from experiments have hardly any chance to check, whether the results of an experiment, they are currently reading, should be taken serious and should be permitted to chance the reader’s perspective on a certain topic. This paper discusses the problem of missing community agreements and makes some first proposals with respect to subjects, training, measurements, experimental designs, and documentation.

On the Need to Define Community Agreements for Controlled Experiments with Human Subjects (plateau2015-hanenberg.pdf)91KiB

Mon 26 Oct