JSS An Empirical Eye-Tracker Study of Visualization Techniques for Coverage of Combinatorial Interaction Testing in Software Product Lines

Software Product Lines (SPLs) typically provide a large number of configurations to cater to a set of diverse requirements of specific markets. This large number of configurations renders unfeasible to test them all individually. Instead, Combinatorial Interaction Testing (CIT) compute a representative sample according to criteria of the interactions of feature in the configurations. We performed an empirical study using task performance an eye-tracker technologies to analyze the effectiveness of two visualization techniques at conveying the test coverage of ten case studies of varying complexity. Our evaluation considered accuracy, execution time, metacognitive monitoring, and visual attention. The study revealed clear advantages of a visualization technique over the other in three evaluation aspects, with a reverse effect depending on the strength of the coverage and distinct areas of visual attention.