recentpopularlog in

jabley : measurement   13

Producing Wrong Data Without Doing Anything Obviously Wrong!
This paper presents a surprising result: changing a seemingly
innocuous aspect of an experimental setup can cause a systems
researcher to draw wrong conclusions from an experiment.
What appears to be an innocuous aspect in the experimental
setup may in fact introduce a significant bias in an
evaluation. This phenomenon is called measurement bias in
the natural and social sciences.
Our results demonstrate that measurement bias is significant
and commonplace in computer system evaluation. By
significant we mean that measurement bias can lead to a performance
analysis that either over-states an effect or even
yields an incorrect conclusion. By commonplace we mean
that measurement bias occurs in all architectures that we
tried (Pentium 4, Core 2, and m5 O3CPU), both compilers
that we tried (gcc and Intel’s C compiler), and most of the
SPEC CPU2006 C programs. Thus, we cannot ignore measurement
bias. Nevertheless, in a literature survey of 133 recent
papers from ASPLOS, PACT, PLDI, and CGO, we determined
that none of the papers with experimental results
adequately consider measurement bias.
Inspired by similar problems and their solutions in other
sciences, we describe and demonstrate two methods, one
for detecting (causal analysis) and one for avoiding (setup
randomization) measurement bias.
paper  filetype:pdf  experiment  bias  measurement 
september 2017 by jabley

Copy this bookmark:

to read