The aligned rank transform for nonparametric factorial analyses using only anova procedures

ABSTRACT

Nonparametric data from multi-factor experiments arise often in human-computer interaction (HCI). Examples may include error counts, Likert responses, and preference tallies. But because multiple factors are involved, common nonparametric tests (e.g., Friedman) are inadequate, as they are unable to examine interaction effects. While some statistical techniques exist to handle such data, these techniques are not widely available and are complex. To address these concerns, we present the Aligned Rank Transform (ART) for nonparametric factorial data analysis in HCI. The ART relies on a preprocessing step that "aligns" data before applying averaged ranks, after which point common ANOVA procedures can be used, making the ART accessible to anyone familiar with the F-test. Unlike most articles on the ART, which only address two factors, we generalize the ART to N factors. We also provide ARTool and ARTweb, desktop and Web-based programs for aligning and ranking data. Our re-examination of some published HCI results exhibits advantages of the ART.

References

1. Conover, W. J. and Iman, R. L. (1981). Rank transformations as a bridge between parametric and nonparametric statistics. The American Statistician 35 (3), 124--129.
2. Fawcett, R. F. and Salter, K. C. (1984). A Monte Carlo study of the F test and three tests based on ranks of treatment effects in randomized block designs. Communications in Statistics: Simulation and Computation 13 (2), 213--225.
3. Findlater, L., Moffatt, K., McGrenere, J. and Dawson, J. (2009). Ephemeral adaptation: The use of gradual onset to improve menu selection performance. Proc. CHI '09. New York: ACM Press, 1655--1664.
4. Higgins, J. J., Blair, R. C. and Tashtoush, S. (1990). The aligned rank transform procedure. Proc. Conf. on Applied Statistics in Agriculture. Kansas State, 185--195.
5. Higgins, J. J. and Tashtoush, S. (1994). An aligned rank transform test for interaction. Nonlinear World 1 (2), 201--211.
6. Hodges, J. L. and Lehmann, E. L. (1962). Rank methods for combination of independent experiments in the analysis of variance. Annals of Mathematical Statistics 33 (2), 482--497.
7. Kaptein, M., Nass, C. and Markopoulos, P. (2010). Powerful and consistent analysis of Likert-type rating scales. Proc. CHI '10. New York: ACM Press, 2391--2394.
8. Littell, R. C., Henry, P. R. and Ammerman, C. B. (1998). Statistical analysis of repeated measures data using SAS procedures. J. Animal Science 76 (4), 1216--1231.
9. MacKenzie, I. S. and Zhang, S. X. (1999). The design and evaluation of a high-performance soft keyboard. Proc. CHI '99. New York: ACM Press, 25--31.
10. Salter, K. C. and Fawcett, R. F. (1985). A robust and powerful rank test of treatment effects in balanced incomplete block designs. Communications in Statistics: Simulation and Computation 14 (4), 807--828.
11. Salter, K. C. and Fawcett, R. F. (1993). The art test of interaction: A robust and powerful rank test of interaction in factorial models. Communications in Statistics: Simulation and Computation 22 (1), 137--153.
12. Sawilowsky, S. S. (1990). Nonparametric tests of interaction in experimental design. Review of Educational Research 60 (1), 91--126.
13. Vermunt, J. K. (1997). Log-linear Models for Event Histories. Thousand Oaks, CA: Sage Publications.
14. Wobbrock, J. O., Wilson, A. D. and Li, Y. (2007). Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes. Proc. UIST '07. New York: ACM Press, 159--168.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

• Mathematics of computing

  • Mathematical analysis

    • Numerical analysis

      • Computation of transforms