Home Publications CV Education


Empirical Evidence in Software Testing

Zoltan Micskei

Hungarian Software Testing Forum (HUSTEF) poster (2017-11-14)


Is TDD better than test-last? Which test design method to use? There are many myths, but luckily more and more empirical evidence is provided by researchers. The poster will give an overview on available empirical methods and explain what kind of insights to expect from them. Moreover, it will present example studies on testing habits, methods and techniques offering practical suggestions.



After a short motivation, the poster depicts two things:

  1. There are different empirical methods that are suitable to study various software testing methods and practices, and
  2. There are empirical studies that can offer relevant suggestions for practitioners.

Part 1

The following main methods will be briefly presented:

When assessing the evidence provided by such studies a significant concern is their validity. Are the conclusions valid and supported by the data? Can we generalize the results and use them in different settings? Unfortunately, one study cannot mitigate all validity threats, and therefore has to find a balance between these two (e.g. an experiment tries to control every factor and have a well-defined procedure, but this can result in a non-realistic setting). Therefore, it is vital to understand the typical validity threats to assess the applicability of the results of an empirical study.

Finally, reliable evidence can be obtained by integrating the results of different studies taking into account their findings and limitations. One way to achieve this is to perform a systematic literature review (SLR). SLRs offer valuable information for testers on what works in different settings.

Part 2

A few studies and their main results are presented to illustrate what kind of information can be obtained from empirical studies and how they can influence practice.

More information

Where to find such studies:

Books if you are interested in the topic:

Last modification: 2017-11-13