SA Forum AIS robustness test suite ---------------------------------- I. Introduction The test suite is intended to test the robustness (i.e. the degree to which a system operates correctly in the presence of exceptional inputs or stressful environmental conditions.) of middleware systems implementing Service Availability Forum's Application Interface Specification, in particular the Availability Management Framework. The test suite was created by a research project conducted by Nokia Research Center and Budapest University of Technology. For details about the project and its tools see [1] and [2]. The current test suite works in a way, that it uses a combination of valid of invalid values for the types in the API and generates a large number of calls to the API functions using the combination of these values. The stucture of the test suite is the following. - conf: contains the openAIS configuration files with an AMF configuration containing one component, the actual test case. The configuration files from the trunk version at 2007. 10. 01. was used, in other versions the configuration files may slightly change. - include, lib: common functions for the test cases - typeSpecificTests: contains the test cases - bin: contains the test case binaries and the log files in a folder called log - src: - generateTypes.c: contains the valid and invalid values for the types used in the API - test_*.c: the actual test cases, which (i) wait for a predefined time before starting the calls to let the application start up properly, (ii) fork a new process for each call with a combination of the values returned by the functions in generateTypes.c, (iii) when they are finished they mark it by creating a file named finished in the bin directory. (Note, that both the generateTypes.c and test_*.c files were automatically generated by the tools created in the project.) II. How to use the test suite 1. Extract the contents of test suite in /opt/aisrobustness 2. Set bindnetaddr in conf/aisrobustness_openais.conf according to your actual IP address 3. Set the MIDDLEWAREROOT variable in Makefile and typeSpecificTetsts/Makefile to point to your openais directory 4. Modify the LOG_FILES_PREFIX and FINISHED_FILE_NAME in include/testcommon.h if needed 5. Set the MIDDLEWAREROOT and TESTROOT variables correctly in start.sh 6. Run start.sh and redirect its output to a file 7. Wait, the running of the whole test suite can take at least half an hour. You can watch the progress by seeing files created in the log directory In the end, the following log files are created - the output of the start.sh, it contains which files was run, basicly you only need this if there is an error with running the tests - ais-test_*.log files in the log folder: contains the output of the middleware - log_*.csv: contains the result of each call. The structure of a line is: functionName;param1,param2,..paramN;saErrorT;processStatusCode functionName: name of the function called param1,param2,...paramN: values of the parameters separated by coma. Pointers are written in "value (address)" form. Members of structures are enclosed with []. saErrorT: saErrorT returned by the call processStatusCode: exit status of a process, 0 means success, 11 means segmenetation fault and 14 means timeout. How to use the result: - load the csv log files in a spreadsheet application and filter for 11 and 14 process status codes. Search for the parameter value that is fixed in all the different failed calls, e.g. a NULL value, in this way you can identify problematic points. - check the saErrorT values for SA_AIS_ERR_LIBRARY, it usually means also a problem - check the ais-test_*.log files for middleware crashes. - check the output of start.sh in the block, if it states "aisexec: no process killed" then the middleware crashed during the tests. III. ToDo There can be a lot of improvement for the tests: - if there is two much combination for a given test, then use random selection instead of choosing the first MAX_NUMBER_OF_CALLS - in start.sh watch not only the creation of the finished file, watch that the middleware is also running - when logging the values for a call, log also the loop1..loopN variables to help finding the values causing a failure Good luck! IV. References [1] Z. Micskei, I. Majzik, F. Tam: Comparing Robustness of AIS-Based Middleware Implementations, In Proceedings of International Service Availability Symposium (ISAS 2007), LNCS 4526, Durham, New Hampshire, USA, May 21-22, 2007. http://www.springerlink.com/content/124123148u534246/?p=9aeb3b69e6154e018aae199330338c2b&pi=2 [2] Z. Micskei, I. Majzik, F. Tam: Robustness Testing Techniques For High Availability Middleware Solutions, In Proceedings of International Workshop on Engineering of Fault Tolerant Systems (EFTS 2006), Luxembourg, Luxembourg, June 12-14, 2006. http://mit.bme.hu/~micskeiz/papers/micskei-majzik-tam_efts2006.pdf