Product Sampling for Product Lines: The Scalability Challenge
Tobias Pett, TU Braunschweig, Germany
Thomas Thüm, TU Braunschweig, Germany
Tobias Runge, TU Braunschweig, Germany
Sebastian Krieter, University of Magdeburg, Germany
Malte Lochau, TU Darmstadt, Germany
Ina Schaefer, TU Braunschweig, Germany
Quality assurance for product lines is often infeasible for each product separately. Instead, often only a subset of all products (i.e. a sample) is considered in testing such that at least the coverage of certain feature interactions is guaranteed. While pair-wise interaction sampling covers all interactions between two features, its generalization to t-wise interaction sampling ensures coverage for all interactions among t features. However, sampling for large product lines poses a challenge, as today’s algorithms tend to run out of memory, do not terminate, or produce samples, which are too large to be tested. To approach this challenge, we provide a set of large real-world feature-models with up-to 19 thousand features, which are supposed to be sampled. The performance of the sampling is evaluated based on the time and memory consumed to retrieve a sample and the sample size for a given coverage (i.e. t value). A well-performing sampling algorithm achieves full t-wise coverage, while minimizing all three of these properties.
Jeho Oh, Paul Gazzillo and Don Batory
SPLC 2019 Solution
Efficiently testing large configuration spaces of Software Product Lines (SPLs) needs a sampling algorithm that is both scalable and provides good t-wise coverage. The 2019 SPLC Sampling Challenge provides large real-world feature models and asks for a t-wise sampling algorithm that can work for those models. We evaluated t-wise coverage by uniform sampling (US) the configurations of one of the provided feature models. US means that every (legal) configuration is equally likely to be selected. US yields statistically representative samples of a configuration space and can be used as a baseline to compare other sampling algorithms. We used existing algorithm called Smarch to uniformly sample SPL configurations. While uniform sampling alone was not enough to produce 100% 1-wise and 2-wise coverage, we used standard probabilistic analysis to explain our experimental results and to conjecture how uniform sampling may enhance the scalability of existing t-wise sampling algorithms. Solution paper