Variability Challenges

Systems and Software Variability Challenges
The official website of the Systems and Software Product Line Conference challenge track

Community

The SPLC challenge track aims at providing the software product-line (SPL) community with a set of case studies that tackle relevant SPL problems and challenge the state of the art. The challenge track was established with SPLC 2018 in Gothenburg, initiated by Sarah Nadi, Timo Kehrer, and Thorsten Berger.
The challenge is organized in two phases every year. The first phase is the call for cases. Submitted cases will be reviewed by the challenge co-chairs to ensure that all required information is clearly described. Accepted cases will be part of the official conference proceedings. Authors of the accepted cases must attend the challenge track and participate in the discussion. The second phase is a call for solutions to both newly accepted as well as previously accepted cases. Submitted case solutions will be peer-reviewed by the challenge program committee. Accepted solutions will also be part of the official conference proceedings. Authors of accepted solutions must present their papers during the conference.

This year, we are placing a special emphasis in our call for cases on the need for actionable benchmarks and competition, akin to those prevalent in the machine learning and SAT communities, and as developed on platforms like Kaggle. We believe that introducing such benchmarks and competition will significantly enhance the ability to compare state-of-the-art solutions and new contributions in the field of software product-line engineering. This kind of proposal should not only foster a spirit of innovation and collaboration but also provides a concrete, measurable framework for assessing the effectiveness and efficiency of new methodologies or implementations. We encourage submissions that offer well-defined, challenging, and replicable benchmarks, thereby setting the stage for a more lively and results-oriented challenge track.
You are highly encouraged to contribute to this website. For example by collecting references to challenge solutions outside the SPLC challenge track. You can propose any improvement or correction by direct editing the github-based website and creating a pull request , creating a github issue, or just sending an email to the challenge track co-chairs.
Current SPLC

Challenge track co-chairs
Mathieu Acher
University of Rennes, FRA
Thomas Thüm
University of Ulm, GER

SPLC 2024

Call for Solution and Challenges is now open!

Submission instructions: https://2024.splc.net/

Please note that the call for solutions also includes the accepted challenges of previous years.

We highly encourage community members to provide solutions for the challenges of SPLC 2018-22. Solutions will be included in the conference proceedings, and presented during the conference.

Challenge track co-chairs
Angelo Gargantini University of Bergamo, IT
Stefania Gnesi Italian National Research Council, IT

Generating Pairwise Covering Arrays for Highly Configurable Software Systems

Jianping Song, Qiyuan Zhao, Yibei Li, Shaowei Cai and Chunming Hu
Solutions site
Highly configurable software systems play crucial roles in real-world applications, which urgently calls for useful testing methods. Combinatorial interaction testing (CIT) is an effective methodology for detecting those faults that are triggered by the interaction of any t options, where t is the testing strength. Pairwise testing, i.e., CIT with t = 2, is known to be the most practical and popular CIT technique, and the pairwise covering array generation (PCAG) problem is the most critical problem in pairwise testing. Due to the practical importance of PCAG, many PCAG algorithms have been proposed. Unfortunately, existing PCAG algorithms suffer from the severe scalability problem. To this end, the SPLC Scalability Challenge (i.e., Product Sampling for Product Lines: The Scalability Challenge) has been proposed since 2019, in order to motivate researchers to develop practical PCAG algorithms for overcoming this scalability problem. In this work, we present a practical PCAG algorithm dubbed SamplingCA-ASF. To the best of our knowledge, our experiments show that SamplingCA-ASF is the first algorithm that can generate PCAs for Automotive02 and Linux, the two hardest and largest-scale instances in the SPLC Scalability Challenge, within reasonable time. Our experimental results indicate that SamplingCA-ASF can effectively alleviate the scalability problem in pairwise testing.

Challenge track co-chairs
Laura Semini, University of Pisa, IT
Xavier Devroey, University of Namur, BE

A Benchmark for Active Learning of Variability-Intensive Systems

Shaghayegh Tavassoli, Carlos Diego Nascimento Damasceno, Mohammad Reza Mousavi and Ramtin Khosravi
Challenge and Solutions site
Behavioral models are the key enablers for behavioral analysis of Software Product Lines (SPL), including testing and model checking. Active model learning comes to the rescue when family behavioral models are non-existent or outdated. A key challenge on active model learning is to detect commonalities and variability efficiently and combine them into concise family models. Benchmarks and their associated metrics will play a key role in shaping the research agenda in this promising field and provide an effective means for comparing and identifying relative strengths and weaknesses in the forthcoming techniques. In this challenge, we seek benchmarks to evaluate the efficiency (e.g., learning time and memory footprint) and effectiveness (e.g., conciseness and accuracy of family models) of active model learning methods in the software product line context. These benchmark sets must contain the structural and behavioral variability models of at least one SPL. Each SPL in a benchmark must contain products that requires more than one round of model learning with respect to the basic active learning L* algorithm. Alternatively, tools supporting the synthesis of artificial benchmark models are also welcome.

Challenge track co-chairs
Paul Temple, University of Namur, BE
Iris Reinhartz-Berger, University of Haifa, IL

Variability Fault Localization: A Benchmark

Kien-Tuan Ngo, Thu-Trang Nguyen, Son Nguyen, Hieu Dinh Vo
Challenge and Solutions site
Software fault localization is one of the most expensive, tedious, and time-consuming activities in program debugging. This activity becomes even much more challenging in Software Product Line (SPL) systems due to the variability of failures in SPL systems. These unexpected behaviors are caused by variability faults which can only be exposed under some combinations of system features. Although localizing bugs in non-configurable code has been investigated in-depth, variability fault localization in SPL systems still remains mostly unexplored. To approach this challenge, we propose a benchmark for variability fault localization with a large set of 1,773 buggy versions of six SPL systems and baseline variability fault localization performance results. Our hope is to engage the community to propose new and better approaches to the problem of variability fault localization in SPL systems.

Managing Systems Evolving in Space and Time: Four Challenges for Maintenance, Evolution and Composition of Variants

Gabriela Karoline Michelon, David Obermann, Wesley K. G. Assunção, Lukas Linsbauer, Paul Grünbacher, Alexander Egyed
Challenge and Solutions site
Software companies need to provide a large set of features satisfying functional and non-functional requirements of diverse customers, thereby leading to variability in space. Feature location techniques have been proposed to support software maintenance and evolution in space. However, so far only one feature location technique also analyses the evolution in time of system variants, which is required for feature enhancements and bug fixing. Specifically, existing tools for managing a set of systems over time do not offer proper support for keeping track of feature revisions, updating existing variants, and creating new product configurations based on feature revisions. This paper presents four challenges concerning such capabilities for feature (revision) location and composition of new product configurations based on feature/s (revisions). We also provide a benchmark containing a ground truth and support for computing metrics. We hope that this will motivate researchers to provide and evaluate tool-supported approaches aiming at managing systems evolving in space and time. Further, we do not limit the evaluation of techniques to only this benchmark: we introduce and provide instructions on how to use a benchmark extractor for generating ground truth data for other systems. We expect that the feature (revision) location techniques maximize information retrieval in terms of precision, recall, and F-score, while keeping execution time and memory consumption low.

Challenge track co-chairs
José A. Galindo, University of Seville, Spain
Lukas Linsbauer, Technische Universität Braunschweig, Germany

Testing Configurable Software Systems: The Failure Observation Challenge

Fischer Ferreira, Markos Viggiato, MaurÍcio Souza and Eduardo Figueiredo.
Challenge and Solutions site
Configurable software systems can be adapted or configured according to a set of features to increase reuse and productivity. The testing process is essential because configurations that fail may potentially hurt user experience and degrade the reputation of a project. However, testing configurable systems is very challenging due to the number of configurations to run with each test, leading to a combinatorial explosion in the number of configurations and tests. Currently, several testing techniques and tools have been proposed to deal with this challenge, but their potential practical application remains mostly unexplored. To encourage the research area on testing configurable systems, researchers and practitioners should be able to try out their solutions in common datasets. In this paper, we propose a dataset with 30 configurable software systems and extensive test suits. Moreover, we report failures found in these systems and source code metrics to allow evaluating candidate solutions. We hope to engage the community and stimulate new and existing approaches to the problem of testing configurable systems.

Variability Management meets Microservices: Six Challenges of Re-Engineering Microservice-Based Webshops

Wesley K. G. Assunção, Jacob Krüger and Willian D. F. Mendonça
Challenge and Solutions site
A microservice implements a small unit of functionality that it provides through a network using lightweight protocols. So, microservices can be combined to fulfill tasks and implement features of a larger software system - resembling a variability mechanism in the context of a software product line (SPL). Microservices and SPLs have similar goals, namely facilitating reuse and customizing, but they are usually employed in different contexts. Any developer who has access to the network can provide a microservice for any task, while SPLs are usually intended to implement features of a single domain. Due to their different concepts, using microservices to implement an SPL or adopting SPL practices (e.g., variability management) for microservices is a challenging cross-area research problem. However, both techniques can complement each other, and thus tackling this problem promises benefits for organizations that employ either technique. In this paper, we reason on the importance of advancing in this direction, and sketch six concrete challenges to initiate research, namely (1) feature identification, (2) variability modeling, (3) variable microservice architectures, (4) interchangeability, (5) deep customization, and (6) re-engineering an SPL. We intend these challenges to serve as a starting point for future research in this cross-area research direction---avoiding that the concepts of one area are reinvented in the other.

A BDD for Linux? The Knowledge Compilation Challenge for Variability

Thomas Thüm
Challenge and Solutions site
What is the number of valid configurations for Linux? How to generate uniform random samples for Linux? Can we create a binary decision diagram for Linux? It seems that the product-line community tries hard to answer such questions for Linux and other configurable systems. However, attempts are often not published due to the publication bias; unsuccessful attempts are not published. As a consequence, researchers keep trying by potentially spending redundant effort.

Challenge track co-chairs
Jabier Martinez, Tecnalia, Spain
Timo Kehrer, Humboldt-Universität zu Berlin, Germany

Applying Product Line Engineering Concepts to Deep Neural Networks

Javad Ghofrani, Ehsan Kozegar, Anna Lena Fehlhaber, and Mohammad Divband Soorati
Challenge and Solutions site
Deep Neural Networks (DNNs) are increasingly being used as a machine learning solution thanks to the complexity of their architecture and hyperparameters-weights. A drawback is the excessive demand for massive computational power during the training process. Not only as a whole but parts of neural networks can also be in charge of certain functionalities. We present a novel challenge in an intersection between machine learning and variability management communities to reuse modules of DNNs without further training. Let us assume that we are given a DNN for image processing that recognizes cats and dogs. By extracting a part of the network, without additional training a new DNN should be divisible with the functionality of recognizing only cats. Existing research in variability management can offer a foundation for a product line of DNNs composing the reusable functionalities. An ideal solution can be evaluated based on its speed, granularity of determined functionalities, and the support for adding variability to the network. The challenge is decomposed in three subchallenges: feature extraction, feature abstraction, and the implementation of a product line of DNNs.

Product Sampling for Product Lines: The Scalability Challenge

Tobias Pett, Thomas Thüm, Tobias Runge, Sebastian Krieter, Malte Lochau, and Ina Schaefer
Challenge and Solutions site
Quality assurance for product lines is often infeasible for each product separately. Instead, often only a subset of all products (i.e. a sample) is considered in testing such that at least the coverage of certain feature interactions is guaranteed. While pair-wise interaction sampling covers all interactions between two features, its generalization to t-wise interaction sampling ensures coverage for all interactions among t features. However, sampling for large product lines poses a challenge, as today’s algorithms tend to run out of memory, do not terminate, or produce samples, which are too large to be tested. To approach this challenge, we provide a set of large real-world feature-models with up-to 19 thousand features, which are supposed to be sampled. The performance of the sampling is evaluated based on the time and memory consumed to retrieve a sample and the sample size for a given coverage (i.e. t value). A well-performing sampling algorithm achieves full t-wise coverage, while minimizing all three of these properties.

Challenge track co-chairs
Sarah Nadi, University of Alberta, Canada
Timo Kehrer, Humboldt-Universität zu Berlin, Germany

Apo-Games – A Case Study for Reverse Engineering Variability from Cloned Java Variants

Jacob Krüger, Wolfram Fenske, Thomas Thüm, Dirk Aporius, Gunter Saake, and Thomas Leich
Challenge and Solutions site
Software-product-line engineering is an approach to systematically manage reusable software features and has been widely adopted in practice. Still, in most cases, organizations start with a single product that they clone and modify when new customer requirements arise (a.k.a. clone-and-own). With an increasing number of variants, maintenance can become challenging and organizations may consider migrating towards a software product line, which is referred to as extractive approach. While this is the most common approach in practice, techniques to extract variability from cloned variants still fall short in several regards. In particular, this accounts for the low accuracy of automated analyses and refactoring, our limited understanding of the costs involved, and the high manual effort. A main reason for these limitations is the lack of realistic case studies. To tackle this problem, we provide a set of cloned variants. In this paper, we characterize these variants and challenge the research community to apply techniques for reverse engineering feature models, feature location, code smell analysis, architecture recovery, and the migration towards a software product line. By evaluating solutions with the developer of these variants, we aim to contribute to a larger body of knowledge on this real-world case study.

Feature Location Benchmark with ArgoUML SPL

Jabier Martinez, Nicolas Ordoñez, Xhevahire Tërnava, Tewfik Ziadi, Jairo Aponte, Eduardo Figueiredo, and Marco Tulio Valente
Challenge and Solutions site
Feature location is a traceability recovery activity to identify the implementation elements associated to a characteristic of a system. Besides its relevance for software maintenance of a single system, feature location in a collection of systems received a lot of attention as a first step to re-engineer system variants (created through clone-and-own) into a Software Product Line (SPL). In this context, the objective is to unambiguously identify the boundaries of a feature inside a family of systems to later create reusable assets from these implementation elements. Among all the case studies in the SPL literature, variants derived from ArgoUML SPL stands out as the most used one. However, the use of different settings, or the omission of relevant information (e.g., the exact configurations of the variants or the way the metrics are calculated), makes it difficult to reproduce or benchmark the different feature location techniques even if the same ArgoUML SPL is used. With the objective to foster the research area on feature location, we provide a set of common scenarios using ArgoUML SPL and a set of utils to obtain metrics based on the results of existing and novel feature location techniques.

Interoperability of Software Product Line Variants

Ferruccio Damiani, Reiner Hähnle, Eduard Kamburjan, and Michael Lienhardt
Challenge and Solutions site
Software Product Lines are an established mechanism to describe multiple variants of one software product. Current approaches however, do not offer a mechanism to support the use of multiple variants from one product line in the same application. We experienced the need for such a mechanism in an industry project with German Railways where we do not merely model a highly variable system, but a system with highly variable subsystems. We present the design challenges that arise when software product lines have to support the use of multiple variants in the same application, in particular: How to reference multiple variants, how to manage multiple variants to avoid name clashes, and how to keep multiple variants interoperable.

Localizing Configurations in Highly-Configurable Systems

Paul Gazzillo, Ugur Koc, Thanhvu Nguyen, and Shiyi Wei
Challenge and Solutions site
The complexity of configurable systems has grown immensely, and it is only getting more complex. Such systems are a challenge for software testing and maintenance, because bugs and other defects can and do appear in any configuration. One common requirement for many development tasks is to identify the configurations that lead to a given defect or some other program behavior. We distill this requirement down to a challenge question: given a program location in a source file, what are valid configurations that include the location? The key obstacle is scalability. When there are thousands of configuration options, enumerating all combinations is exponential and infeasible. We provide a set of target programs of increasing difficulty and variations on the challenge question so that submitters of all experience levels can try out solutions. Our hope is to engage the community and stimulate new and interesting approaches to the problem of analyzing configurations.