Evaluating Shrinking (Experience Report)
Property-based testing frameworks rely on shrinking to turn noisy random
failures into counterexamples that developers can debug. Although bug-finding
performance is routinely measured, shrinking itself is rarely evaluated
quantitatively. We present an experience report on evaluating shrinking across
three Haskell frameworks: QuickCheck, Hedgehog, and Falsify. The comparison spans
four ETNA workloads and several generator families, including type-based,
API-based, and correct-by-construction generators. We measure both
effectiveness, using tree edit distance to a ground-truth minimum found by
exhaustive search, and cost, using shrink time and time per unit of shrinking
progress. Across these workloads, QuickCheck's structural shrinking is usually
faster and remains competitive on final counterexample quality; integrated
shrinking does not by itself guarantee a performance or effectiveness advantage.
We discuss what these results imply for future evaluations and designs of
shrinking algorithms.
Fri 28 AugDisplayed time zone: Eastern Time (US & Canada) change
11:00 - 12:30 | |||
11:00 30mTalk | Evaluating Shrinking (Experience Report) Haskell Alperen Keles University of Maryland at College Park, George Miao University of Maryland, College Park, Leonidas Lampropoulos University of Maryland at College Park DOI | ||
11:30 30mTalk | Xeus-Haskell: Interactive Haskell Computing in the Browser Haskell Masaya Taniguchi RIKEN AIP | ||
12:00 30mTalk | Turning Parser Errors into Suggestions for REPL-Driven DSLs (Functional Pearl) Haskell Matthías Páll Gissurarson Chalmers University of Technology, Sweden, Elisabet Lobo-Vesga Chalmers University of Technology, Sweden, Alejandro Russo Chalmers University of Technology; University of Gothenburg DOI | ||