An exploratory empirical eye-tracker study of visualization techniques for coverage of combinatorial interaction testing in software product lines

Software Product Lines (SPLs) typically provide a large number of configurations to cater to a set of diverse requirements of specific markets. This large number of configurations renders unfeasible to test them all individually. Instead, Combinatorial Interaction Testing (CIT) computes a representative sample according to criteria of the interactions of features in the configurations. We performed an empirical study using eye-tracker technologies to analyze the effectiveness of two basic visualization techniques at conveying test coverage information of ten case studies of varying complexity. Our evaluation considered response accuracy, time-on-task, metacognitive monitoring, and visual attention. The study revealed clear advantages of a visualization technique over the other in three evaluation aspects, with a reverse effect depending on the strength of the coverage and distinct areas of visual attention.