“For user studies in software engineering holds: it is better to have one user in the field, than ten in the lab.”

Runkel & McGrath have proposed a four-quadrant taxonomy of empirical methods with two axes, obtrusive vs. unobtrusive and abstract vs. concrete.

They further add three different criteria we might have for our methods, generalizability, precision, and realism. [1]

Many artifacts in software engineering are validated with studies in the lab: ranging from testing an algorithm on a given code base to testing a method with colleagues or students. I am no stranger to these methods. In my first paper [2] we evaluated a method to extract class diagrams from spreadsheets by comparing our algorithm to our own manual performance. While the algorithm worked quite well, we did not think about how it would be perceived by users. It wasn’t until we started to use it in practice that we found that users did not want to transform spreadsheets! So the algorithm worked, but was quite useless.

This experience made me convinced that we as software engineers should focus on maximizing realism in user studies. Think case studies, interviews or tests in a company, basically everything you can do outside of the university. Too often I read great papers with well-performed lab evaluations, that solve problems of which I question they exist or really need solving. I know it is hard to perform a case study in industry, but software engineering is concerned with how people build and maintain software, so we should study how this is actually happening and how our newly developed tools support this. We shouldn’t focus on ideal lab circumstances or untrained students. As Erik Meijer states:

“In practice most development effort goes into the ‘noise’ that researchers abstract away.” 

We should embrace this noise and start to actively look for it in our field work. This will bridge the gap between industry and academia (a bit) and make sure our methods and tools have a basis in reality. Want to know more? [3] is a great place to start.

Addendum

I forgot to mention, there is a great workshop at ICSE this year about user studies: USER 2013.

References

[1] http://www.ugrad.cs.ubc.ca/~cs344/current-term/resources/supp-userStudyMethods.html#empiricalMethods
[2] Automatically extracting class diagrams from spreadsheets
[3] Benefits and barriers of user evaluation in software engineering research