SERQco: Software Engineering Research Quality Coalition

1 Mission

Background: The Software Engineering Research Quality Coalition (SERQco, pronounced “Sirco”) works based on the perception that many empirical SE research contributions have (some aspects of) avoidably low quality.
Objective: SERQco creates evidence of research quality issues and makes suggestions how to improve.
Method: SERQco operationalizes reasonable lower bounds for some aspects of research quality and then measures research quality over many published empirical SE research articles.

2 Methods

2.1 Research approach

SERQco mostly analyzes published research literature, using methods that work with qualitative data but aim at quantitative results. A common approach for simple cases is open coding plus subsequent content analysis. More complicated cases will use individually crafted combinations of methods.

2.2 Epistemological stance

In the normative aspects of its work, SERQco predominantly uses pragmaticist and mildly antipositivist epistemological stances.

2.3 Organization

SERQco is a loose collective of SE researchers who work together on a project-by-project basis. A project usually aims at doing a single study and producing one or two publications from it.

2.4 Open science

SERQco works transparently: We identify each analyzed article clearly and publish raw data. This will embarrass some authors, but our research is about works, not people, and is interested in general trends, not in individual cases (which we may still use for illustration).

3 Possible topics

There are many questions that SERQco might tackle. Some are straightforward to investigate and the results should be acceptable for everybody; others will be trickier or the outcomes more controversial. SERQco considers them all and picks those where the benefit ratio appears favorable.

Examples (in order of increasing difficulty):

Abstracts that do not summarize
Conclusions that do not draw conclusions
Confusing engineering with science: not explaining relevance
Use of causational language in studies that are merely correlational
Imprecise method descriptions
Questionable tacit assumptions

4 The logo: “Defining a bar”?

In some countries, “bar” is a legal term referring to the community of legal practitioners and connoting the line in a courtroom that separates the spectators from the participants of a trial;
the “bar exam” is an examination that people with a university legal degree need to pass to become members of the bar and be admitted to court or at least to higher courts.

In this spirit, SERQco will demonstrate some conditions that an SE research article should fulfill before it can be considered good research.