In March, I blogged about the industry track of ICSME (the international conference on software maintenance and evolution), offering advice on how to write a paper for an industry focused track. In this track, we did something relatively new in software engineering conferences, we used a double blind reviewing process. In this blog post, I will explain our reasons and process, while in a next one, I will  share our experiences, and the opinions of reviewers and authors.

What is double blind?

Maybe I should firstly explain what single blind is?

What is single blind?

If you submit a paper to a conference, you as an author typically do not know who the reviewer is. This is done to protect the reviewers, because a bad review could easily be retaliated later. Communities are relatively small, maybe a few hundred researchers are active in a field, so chances are indeed high that someone you once reviewed will review your paper. Also, many conferences have acceptance rates between 20 and 50% meaning that 80 to 50% of papers get rejected. At ICSME around 100 papers get submitted, and about 70 get rejected. That are a lot of people potentially pissed of. And then there also are reference letters, grant proposals etc. So protecting the reviewers does make some sense.

OK, now what is double blind?

In double blind, the reviewers do not know who the authors are. This means they remove their names from the first page of the paper, but they also have to remove other details that might reveal their identity. For example, in a single blind paper, I could write about my own work like this: “in our previous work, we have examined the effect of code smells on the performance of high school children”. In a double blind submission, however, I should write “Hermans et al. examined the effect of code smells on the performance of high school children” to conceal the fact that I wrote that previous paper.

What happens if the paper is accepted?

Important to know is that, if the paper is accepted, the authors can make changes to the paper, for example, they might change the sentences referring to their own previous work anonymously to more common sentences, or add details about experimentation subjects that might have been revealing, like “in our experiment, 23 students from Delft University of Technology participated”.

Why does this matter? 

There is ample research that shows that people are not entirely unbiased, Claire Le Goues has a great list on her website, and my colleagues Moritz and Alberto wrote a paper containing background. Many studies show that gender of the authors can play a big role, as can nationality and the rank of the university of the authors, or the status of the company they work for. I have really experienced this while reviewing a subpar paper by a hot shot in the community. You do seem to give them the benefit of the doubt, where you might not with lesser names.

Blinding a paper is relatively low effort, while it will most likely make the world more fair.

That does sound nice! So why aren’t we doing this yet? I think the main reason is that it is just not common to do it, and it something is a certain way and has been for years, you need effort to change it. However, there are some arguments against double blind which heard over the course of organizing this track.

But in software engineering, we make tools and share data!

That indeed is an issue, if you want to share data or software, you might accidentally reveal your identity. Also, of course, the fact that you build upon an existing code base must mean you have access to it. Luckily we live in an age of more and more research software shared via GitHub, so other people nowadays could realistically even use someone else’s software. (Someone just used our Excel formula Parser, so there!)

A solution here could be to share your data in an anonymous way, on a fresh GitHub account (I have done this) or to promise in the paper that the data will be added upon acceptance.

But in industry tracks, it is important to share the context!

This is a comment that specifically pertains to industry track papers, which often talk about case studies performed within companies, or about data obtained from companies. If you need to make your paper double blind, you might be forced to remove details that are important in understanding the paper.

This indeed is true, and it might be hard to fully appreciate the paper without details. We were unsure how this would work out, this is something you will see in the survey in my next post 🙂

But, I know who it wrote anyway!

Sometimes, yes after a while you will figure it out. Although it is not always obvious, I once thought I knew and I was wrong! And also, we figured that every page that the reviewer does not know, is a win. This too will be addressed in the survey.

How does it work practically?

We made an extensive FAQ for reviewers, see here: http://icsme2016.github.io/cfp/dbr-faq.html