Home Publications CV Education

Code-based test generation

By

Overview

In software testing one of the challenging tasks is to select the test inputs. There are several methods that are capable of automatically generating test input based on the source code of the program under test. (However, these methods usually cannot generate the expected output for these inputs, as they do not have a specification or model). These methods usually try to satisfy kind of coverage (e.g. different branches and paths of the code under test).

Note: these tools use the code/binary of the program under test; see this list for model-based test generation tools.

Several methods can be used for code-based test generation, for example:

There are several good overviews and surveys for the topic:

Comparison: for an evaluation and comparison of some tools see our SETTE project.

Tools

A far from complete list of code-based test generation tools. (If you know about a tool that is not in the list, feel free to send me an email.)

ToolAgitarOne
Mod.2015
Ref.[BDS06]
InputJava
Method
TypeCOMM
Desc.AgitarOne is a commercial tool for Java combining dynamic invariant detection and test-input generation. It exercises classes with various input data and create 'observations' that represent the behavior of the unit, which developers can convert to assertions.
 
ToolAutoTest
Mod.2015
Ref.[M+09]
InputEiffel
MethodANN
TypeCOMM, OSS
Desc.AutoTest uses assertions in Design by Contract to generate test cases. It automates test creation, test minimization. It is now integrated into the EiffelStudio IDE.
 
ToolCATG
Mod.2015
Ref.
InputJava
MethodSE
TypeOSS
Desc.CATG is a concolic unit testing engine for Java programs (built by the developers of jCUTE).
 
ToolCAUT
Mod.2014
Ref.[S+15]
InputC
MethodSE
TypeACAD
Desc.CAUT is a DSE (dynamic symbolic execution)-based tool to automatically generate test data for C program at unit/program testing level. It currently supports the coverage-driven testing on both control flow and data flow coverage, e.g., branch, MC/DC and all def-use criteria.
 
ToolCREST
Mod.2014
Ref.[BS08]
InputC
MethodSE
TypeOSS
Desc.CREST is an automatic test generation tool for C. It instruments the source code to perform symbolic execution, and uses a constraint solver to search for unexplored program paths.
 
ToolEvoSuite
Mod.2015
Ref.[FA13]
InputJava
MethodSB
TypeACAD, OSS
Desc.EvoSuite is a tool that automatically generates test cases with assertions for classes written in Java code. To achieve this, EvoSuite applies a novel hybrid approach that generates and optimizes whole test suites towards satisfying a coverage criterion.
 
ToolGRT
Mod.2015
Ref.[M+15]
InputJava
MethodRAND
TypeACAD
Desc.GRT (Guided Random Testing) is an automatic test generation tool for Java code, which leverages static and dynamic program analysis to guide run-time test generation. It uses feedback-directed random testing and is based on Randoop.
 
ToolGUITAR
Mod.2015
Ref.[NRBM13]
InputGUI application (JFC, Web)
Method
TypeOSS
Desc.GUI Testing frAmewoRk: "rips" the GUI of an application and creates an event-flow graph model from it. GUITAR is open-source, there is a lot of publications and case studies with it. Beta versions are available for testing other types of applications (e.g. iPhone).
 
ToolJalangi
Mod.2015
Ref.[SKBG13]
InputJavaScript
MethodSE
TypeOSS
Desc.Jalangi is a framework for writing heavy-weight dynamic analyses for JavaScript, like concolic testing or taint analysis.
 
Tool JSeft
Mod. 2014
Ref. [MMP15]
Input JavaScript
Method
Type OSS, ACAD
Desc. JSeft generates function-level unit tests and DOM-level event-based test cases for JavaScript code.
 
ToolJtest
Mod.2015
Ref.
InputJava
Method?
TypeCOMM
Desc.Jtest is an automated Java software testing and static analysis product that is made by Parasoft. The product includes technology for Unit test-case generation and execution, static analysis, regression testing, runtime error detection, code review, and Design by contract.
 
ToolJTExpert
Mod.2015
Ref.[SPG15]
InputJava
MethodSB
TypeACAD
Desc.JTExport can automatically generate a whole test suite towards satisfying the branch coverage criterion on a given Java class.
 
ToolKLEE
Mod.2015
Ref.[CDE08]
InputC (LLVM bitcode)
MethodSE
TypeOSS
Desc.KLEE is capable of automatically generating tests that achieve high coverage on a diverse set of complex and environmentally-intensive programs. KLEE is the successor of EXE.
 
ToolMergePoint/Mayhem
Mod.2013
Ref.[A+14]
Inputbinary (32bit, Linux)
MethodSE
TypeCOMM?
Desc.MergePoint is a tool that combines dynamic and static symbolic execution. It is built on top of Mayhem (a tool that is used for exploit generation). MergePoint was run on more than 30000 binaries from Debian.
 
ToolPathCrawler
Mod.2013?
Ref.[WMMR05]
InputC
Method
TypeACAD
Desc.PathCrawler is a method to automatically generate path tests for the all-feasible-paths criterion with a user-defined limit on the number of loop iterations in the covered path. It has an online version and a version integrated into Frama-C.
 
ToolPex
Mod.2015
Ref.[TH08]
Input.NET code
MethodSE
TypeACAD, COMM
Desc.With Pex testers create parametrized unit tests and then Pex finds input parameters for these test that result in high coverage, boundary conditions, exceptions and assertion failures. Pex applies dynamic symbolic execution and an underlying constraint solver. Pex is now included in Visual Studio as IntelliTest.
 
Tool QuickCheck
Mod. 2015
Ref. [CH11]
Input Haskell
Method RAND
Type OSS
Desc. QuickCheck is a library for random testing of program properties. The programmer provides a specification of the program, in the form of properties which functions should satisfy, and QuickCheck then tests that the properties hold in a large number of randomly generated cases. There are commercial versions for C, Erlang developed by QuviQ
 
ToolRandoop
Mod.2015
Ref.[P+07]
InputJava
MethodRAND
TypeOSS
Desc.Randoop generates unit tests using feedback-directed random test generation. Randoop executes the sequences it creates, using the results of the execution to create assertions that capture the behavior or the program and to catch bugs.
 
ToolSAGE
Mod.2013?
Ref.[GLM12]
Inputx86 binary
MethodSE
TypeCOMM
Desc.Scalable Automated Guided Execution (SAGE) is a whitebox fuzzing tool, which extends systematic dynamic test generation to whole-program security testing. It is internally used in Microsoft.
 
ToolSymbolic PathFinder
Mod.2015
Ref.[P+13]
InputJava
MethodSE
TypeOSS
Desc.Symbolic PathFinder (SPF) combines symbolic execution with model checking and constraint solving for test case generation.
 
ToolT3
Mod.2015
Ref.[PVB08]
InputJava
MethodRAND
TypeOSS
Desc.T3 (previously T2) is able to randomly generate sequences of calls to Java class' methods to test it.

Tools not developed any more

ToolCodePro Analytix
Mod.2009
Ref.
InputJava
Method
TypeCOMM
Desc.The CodePro JUnit Test Case Generation facility allows you to automate the creation of comprehensive JUnit regression test cases. The tool was acquired by Google, its development stopped.
 
ToolCUTE, jCUTE
Mod.2006
Ref.[SA06]
InputC, Java
MethodSE
TypeACAD
Desc.A concolic unit testing engine for C (CUTE) and Java (jCUTE), which could handle multi-threaded programs and pointer operations.
 
ToolDART
Mod.2005
Ref.[GKS05]
InputC
MethodSE
TypeACAD
Desc.DART (Directed Automated Random Testing) was one of the first tools to use dynamic symbolic execution. (Tool was not publicly available.)
 
Tool DOTgEAr
Mod. 2007
Ref. [OS06]
Input Java
Method SB
Type ACAD
Desc. Dataflow-oriented Test Case Generation for Object-oriented Software Systems by Evolutionary Algorithm.
 
Tool DSD-Crasher
Mod. 2007
Ref. [SC07]
Input Java
Method
Type ACAD
Desc. DSD-Crasher is the successor of JCrasher. It adds two steps before the test generation: runs existing tests to capture their behaviour, then uses static analysis to explore the paths of the program.
 
Tool Euclide
Mod. 2009
Ref. [G09]
Input C
Method SE
Type ACAD
Desc. Test generator for C, implemented as a web application. Uses contraint solvers to generate a test reaching a given point in the program.
 
Tool EXE
Mod. 2006
Ref. [C+08]
Input C
Method SE
Type ACAD
Desc. EXecution generated Executions (EXE) is a bug-finding tool that uses symbolic and concrete execution by turns. EXE was later redesigned in KLEE.
 
Tool JCrasher
Mod. 2007
Ref. [CS04]
Input Java
Method
Type ACAD
Desc. JCrasher is an automatic robustness tester for Java. JCrasher examines the type information of a set of Java classes and constructs code fragments that will create instances of different types to test the behavior of public methods under random data.
 
Tool JET
Mod. 2010
Ref. [CR07]
Input Java annotated with JML
Method ANN
Type ACAD
Desc. JET is an automated unit testing tool for Java classes annotated with Java Modeling Language (JML) specifications. It generates random tests and uses the JML constraints as test oracles.
 
Tool jFuzz
Mod. 2009
Ref. [JHGK09]
Input Java
MethodSE
Type ACAD
Desc. jFuzz is a concolic whitebox fuzzer, built on the NASA Java PathFinder.
 
Tool Korat
Mod. 2011
Ref. [BKM02]
Input Java
Method
Type OSS
Desc. Korat is a tool for constraint-based generation of structurally complex test inputs for Java programs. Korat requires (1) an imperative predicate that specifies the desired structural constraints and (2) a finitization that bounds the desired test input size.
 
Tool LCT
Mod. 2012
Ref. [K+11]
Input Java
Method SE
Type ACAD
Desc. The LIME testbench is a collection of tools that allows concolic testing of Java programs.
 
Tool Mika
Mod. 2011
Ref.
Input Ada
Method ?
Type COMM
Desc. Mika is an entirely automatic tool that analyses your Ada code and generates test inputs that will exercise all the branches, decisions or MC/DCs within your code at a level suitable for integration testing.
 
Tool Palus
Mod. 2012
Ref. [Z+11]
Input Java
Method SE
Type OSS
Desc. Palus is an automated test generation tool, which combines various static and dynamic analysis to create tests for Java programs in the JUnit form.
 
Tool PET, jPET
Mod. 2011
Ref. [AGP10]
Input Java
Method
Type ACAD
Desc. Partial Evaluation-based Test Case Generator for Bytecode (PET) transforms Java bytecode to equivalent CLP code, and generates test cases from it. It is implemented in Prolog, jPET is an Eclipse GUI for it.
 
Tool TAO
Mod. 2004
Ref.
Input C
Method ANN
Type ACAD
Desc. TAO can generate test cases based on structural coverage, or from pre- and postconditions (with the CAVEAT tool). (Tool web page not available any more.)

Further tools not developed any more: GenUTest [PTY09], Klover [LGR11], TestGen4J, TestEra

References

[A+13] S. Anand et al. (2013) "An orchestrated survey of methodologies for automated software test case generation," Journal of Systems and Software, 86:8, pp. 1978–2001. DOI: j.jss.2013.02.061
[AGP10] E. Albert, M. Gomez-Zamalloa, and G. Puebla (2010) "PET: A Partial Evaluation-based Test Case Generation Tool for Java Bytecode." In Proc. PEPM, ACM, pp. 25-28. DOI: 10.1145/1706356.1706363
[A+14] T. Avgerinos, A. Rebert, S. K. Cha, and D. Brumley (2014) "Enhancing Symbolic Execution with Veritesting", In Proc. ICSE'14, pp. 1083-1094. DOI: 10.1145/2568225.2568293
[BDS06] M. Boshernitsan, R. Doong, A. Savoia (2006) "From Daikon to Agitator: lessons and challenges in building a commercial tool for developer testing", In Proc. of ISSTA '06, pp. 169-180. DOI: 10.1145/1146238.1146258
[BKM02] C. Boyapati, S. Khurshid, and D. Marinov (2002) "Korat: Automated testing based on Java predicates", In Proc. ISSTA'02, pp. 123-133. DOI: 10.1145/566172.566191
[BS08] J. Burnim and K. Sen (2008) "Heuristics for Scalable Dynamic Test Generation," In Proc. ASE'08, IEEE, pp. 443-446. DOI: 10.1109/ASE.2008.69
[C+08] C. Cadar et al. (2008) "EXE: Automatically Generating Inputs of Death." ACM Trans. Inf. Syst. Secur. 12:2. DOI: 10.1145/1455518.1455522
[C+11] C. Cadar et al. (2011) "Symbolic execution for software testing in practice: preliminary assessment". In Proc. of ICSE '11, ACM, 1066-1071. DOI: 10.1145/1985793.1985995
[CDE08] C. Cadar, D. Dunbar, D. Engler (2008) "KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs." In Proc. of OSDI'08, pp. 209-224.
[CH11] K. Claessen, J. Hughes (2011) "QuickCheck: a lightweight tool for random testing of Haskell programs". ACM SIGPLAN Notices 46:4, pp. 53-64. DOI: 10.1145/1988042.1988046
[CR07] Y. Cheon and C. E. Rubio-Medrano (2007) "Random Test Data Generation for Java Classes Annotated with JML Specifications." In Proc. of ICSERP, pages 385-392.
[CS04] C. Csallner and Y. Smaragdakis (2004) "JCrasher: An automatic robustness tester for Java," Software -- Practice ∧ Experience, vol. 34, no. 11, pp. 1025-1050. DOI: 10.1002/spe.602
[FA13] G. Fraser and A. Arcuri (2013) "Whole Test Suite Generation," IEEE Trans. on Software Engineering, vol. 39, iss. 2, pp. 276-291. DOI: 10.1109/TSE.2012.14
[G09] A. Gotlieb (2009) "EUCLIDE: A Constraint-Based Testing platform for critical C programs." In Proc. ICST'09, IEEE, pp. 151-160. DOI: 10.1109/ICST.2009.10
[GA14] S. Galler, B. Aichernig (2014) "Survey on test data generation tools" STTT 16:6, Springer, pp. 727-751. DOI: 10.1007/s10009-013-0272-3
[GKS05] P. Godefroid, N. Klarlund, and K. Sen. (2005) "DART: directed automated random testing." In Proc. PLDI '05. ACM, pp. 213-223. DOI: 10.1145/1065010.1065036
[GLM12] P. Godefroid, M. Y. Levin, and D. Molnar. (2012) "SAGE: Whitebox Fuzzing for Security Testing." ACM Queue 10:1. DOI: 10.1145/2090147.2094081
[JHGK09] K. Jayaraman, D. Harvison, V. Ganesh, A. Kiezun (2009) "jFuzz: A Concolic Whitebox Fuzzer for Java," NASA Formal Methods Workshop.
[K+11] K. Kahkonen et al. (2011) "LCT: An Open Source Concolic Testing Tool for Java Programs." In Proc. of BYTECODE'2011, pp. 75-80, Saarbrucken, Germany.
[LGR11] G. Li, I. Ghosh, S. P. Rajan (2011) "KLOVER: A Symbolic Execution and Automatic Test Generation Tool for C++ Programs". In CAV'11, pp. 609-615. DOI: 10.1007/978-3-642-22110-1_49
[M+09] B. Meyer et al. (2009) "Programs That Test Themselves," Computer, vol. 42, 2009, pp. 46-55. DOI: 10.1109/MC.2009.296
[M+15] Lei Ma et al. (2015) "GRT: Program-Analysis-Guided Random Testing ," in ASE'15, pp. 212-223. DOI: 10.1109/ASE.2015.49
[MMP15] S. Mirshokraie, A. Mesbah, and K. Pattabiraman (2015) "JSEFT: Automated JavaScript Unit Test Generation". In Proc. ICST'15. DOI: 10.1109/ICST.2015.7102595
[NRBM13] B. N. Nguyen, B. Robbins, I. Banerjee, A. Memon (2013) "GUITAR: an innovative tool for automated testing of GUI-driven software" Automated Software Engineering, Springer, pp. 1-41. DOI: 10.1007/s10515-013-0128-9
[OS06] N. Oster and F. Saglietti (2006) "Automatic Test Data Generation by Multi-objective Optimisation," In Proc SAFECOMP'06, Springer, pp. 426-438. DOI: 10.1007/11875567_32
[P+07] C. Pacheco et al. (2007) "Feedback-directed random test generation," In Proc. of ICSE'07, IEEE, pp. 75-84. DOI: 10.1109/ICSE.2007.37
[P+13] C. S. Pasareanu et al. (2013) "Symbolic PathFinder: integrating symbolic execution with model checking for Java bytecode analysis," Automated Software Engineering 20:3, pp 391-425. DOI: 10.1007/s10515-013-0122-2
[PTY09] B. Pasternak, S. Tyszberowicz, A. Yehudai (2009) "GenUTest: a unit test and mock aspect generation tool". STTT 11:4, pp. 273-290. DOI: 10.1007/s10009-009-0115-4
[PV09] C. S. Pasareanu, W. Visser (2009) "A survey of new trends in symbolic execution for software testing and analysis". STTT, 11(4): 339-353. DOI: 10.1007/s10009-009-0118-1
[PVB08] I.S.W.B. Prasetya, T.E.J. Vos, A.I. Baars (2008) "Trace-based Reflexive Testing of OO Programs with T2". ICST'08, pp. 151-160. DOI: 10.1109/ICST.2008.12
[QR11] X. Qu, B. Robinson (2011) "A Case Study of Concolic Testing Tools and their Limitations". In Proc. ESEM'11, IEEE, pp.117-126. DOI: 10.1109/ESEM.2011.20
[S+15] T. Su et al. "Combining symbolic execution and model checking for data flow testing". In Proc. ICSE 2015, pp. 654-665. DOI: 10.1109/ICSE.2015.81
[SA06] K. Sen and G. Agha (2006) "CUTE and jCUTE : Concolic Unit Testing and Explicit Path Model-Checking Tools," In Proc. CAV'06, Springer. DOI: 10.1007/11817963_38
[SC07] Y. Smaragdakis and C. Csallner (2007) "Combining static and dynamic reasoning for bug detection" In Proc. TAP'07, Springer, pp. 1-16. DOI: 10.1007/978-3-540-73770-4_1
[SKBG13] K. Sen, S. Kalasapur, T. Brutch, and S. Gibbs (2013) "Jalangi: A Selective Record-Replay and Dynamic Analysis Framework for JavaScript." In Proc. ESEC/FSE'13. DOI: 10.1145/2491411.2491447
[SPG15] A. Sakti, G. Pesant, Y.-G. Guéhéneuc (2015) "Instance Generator and Problem Representation to Improve Object Oriented Code Coverage," in Software Engineering, IEEE Transactions on, vol.41, no.3, pp.294-313. DOI: 10.1109/TSE.2014.2363479
[TH08] N. Tillmann and J. de Halleux (2008) "Pex -- White Box Test Generation for .NET," In Prof. TAP'08, Springer, pp. 134-153. DOI: 10.1007/978-3-540-79124-9_10
[WMMR05] N. Williams, B. Marre, P. Mouy, M. Roger. (2005) "PathCrawler: automatic generation of path tests by combining static and dynamic analysis." In Proc. EDCC-5, Springer, pp. 281-292. DOI: 10.1007/11408901_21
[Z+11] S. Zhang et al. (2011) "Combined static and dynamic automated test generation," In Proc. ISSTA'11, ACM, pp. 353-363. DOI: 10.1145/2001420.2001463
DOI:

Last modified: 2015. 12. 01.