Home Publications CV Education

Robustness testing

Definition: Robustness is defined as the degree to which a system operates correctly in the presence of exceptional inputs or stressful environmental conditions. [IEEE Std 24765:2010]

Goal: The goal of robustness testing is to develop test cases and test environments where a system's robustness can be assessed.

Robustness testing approaches

Many research projects worked on this topic, the two major approaches are:

I. Interface robustness testing: bombarding the public interface of the application/system/API with valid and exceptional inputs. The success criteria is in most cases: "if it does not crash or hang, then it is robust", hence no oracle is needed for the testing. Examples:

More information

II. Dependability benchmarking: the aim is to develop a public benchmark specification which focuses on evaluating the dependability of the system. It is a much broader field, than robustness, it contains the other attributes of dependability like availability and maintainability. The common method is to create a workload, which resembles the normal operation of the system under benchmark. Then define a faultoad, which contains typical faults (hardware, software, operator, etc.) and the exact time period when they should be instered. The specification includes also what dependability measures should be collected.

More information

Our robustness testing experiments of HA middleware

Together with Nokia Research Center we were working on the robustness testing of HA middleware systems. The general idea is illustrated on the following figure. On the left side you can see the typical fault sources of an HA middleware, on the right side the means to test against these fault sources.


Test suite and results:

Readme: README for the test suite

The code of the test suite for openais trunk (2007.10.02.)

Older codes for previous versions (just for backup, try to use the above one, we updated the code and scripts slightly since these versions): openais trunk 20061211 and openais release 0.80.1


The test results obtained for the openais platform:

Last modified: 2013. 10. 02.