Open Systems Laboratory at Illinois

Reducing the costs of bounded-exhaustive testing

By Vilas Jagannath, Yun Young Lee, Brett Daniel, and Darko Marinov. In FASE, volume 5503 of Lecture Notes in Computer Science, 171–185. Springer, 2009.

Full Text:
Download PDF
Publisher Link:


Bounded-exhaustive testing is an automated testing methodology that checks the code under test for all inputs within given bounds: first the user describes a set of test inputs and provides test oracles that check test outputs; then the tool generates all the inputs, executes them on the code under test, and checks the outputs; and finally the user inspects failing tests to submit bug reports. The costs of bounded-exhaustive testing include machine time for test generation and execution (which translates into human time waiting for these results) and human time for inspection of results. This paper proposes three techniques that reduce these costs. Sparse Test Generation skips some tests to reduce the time to the first failing test. Structural Test Merging generates a smaller number of larger test inputs (rather than a larger number of smaller test inputs) to reduce test generation and execution time. Oracle-based Test Clustering groups failing tests to reduce the inspection time. Results obtained from the bounded-exhaustive testing of the Eclipse refactoring engine show that these three techniques can substantially reduce the costs while mostly preserving fault-detection capability.


    author = "Jagannath, Vilas and Lee, Yun Young and Daniel, Brett
              and Marinov, Darko",
    editor = "Chechik, Marsha and Wirsing, Martin",
    title = "Reducing the Costs of Bounded-Exhaustive Testing",
    booktitle = "FASE",
    crossref = "conf/fase/2009",
    ee = "",
    keywords = "software engineering",
    pages = "171-185",
    year = "2009",

    editor = "Chechik, Marsha and Wirsing, Martin",
    title = "Fundamental Approaches to Software Engineering, 12th
             International Conference, FASE 2009, Held as Part of the Joint
             European Conferences on Theory and Practice of Software, ETAPS
             2009, York, UK, March 22-29, 2009. Proceedings",
    ee = "",
    isbn = "978-3-642-00592-3",
    publisher = "Springer",
    series = "Lecture Notes in Computer Science",
    volume = "5503",
    year = "2009",