Uncertainty Displays Using Quantile Dotplots or CDFs Improve Transit Decision-Making

ABSTRACT

Everyday predictive systems typically present point predictions, making it hard for people to account for uncertainty when making decisions. Evaluations of uncertainty displays for transit prediction have assessed people's ability to extract probabilities, but not the quality of their decisions. In a controlled, incentivized experiment, we had subjects decide when to catch a bus using displays with textual uncertainty, uncertainty visualizations, or no-uncertainty (control). Frequency-based visualizations previously shown to allow people to better extract probabilities (quantile dotplots) yielded better decisions. Decisions with quantile dotplots with 50 outcomes were(1) better on average, having expected payoffs 97% of optimal(95% CI: [95%,98%]), 5 percentage points more than control (95% CI: [2,8]); and (2) more consistent, having within-subject standard deviation of 3 percentage points (95% CI:[2,4]), 4 percentage points less than control (95% CI: [2,6]).Cumulative distribution function plots performed nearly as well, and both outperformed textual uncertainty, which was sensitive to the probability interval communicated. We discuss implications for real time transit predictions and possible generalization to other domains.

References

1. Jessica S. Ancker, Yalini Senathirajah, Rita Kukafka, and Justin B. Starren. 2006. Design Features of Graphs in Health Risk Communication: A Systematic Review. Journal of the American Medical Informatics Association 13, 6 (2006), 608--618.
2. William S. Cleveland and Robert McGill. 1984. Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods. J. Amer. Statist. Assoc. 79, 387 (1984), 531--554.
3. Francisco Cribari-Neto and Achim Zeileis. 2009. Beta regression in R. (2009).
4. Charles Holt D. Davis. 1996. Experimental economics : Douglas D. Davis and Charles A. Holt, (Princeton University Press, Princeton, 1993), pp xi+571. Journal of Economic Behavior & Organization 30, 3 (September 1996), 411--416. https: //ideas.repec.org/a/eee/jeborg/v30y1996i3p411--416.html
5. Beverley J. Evans. 1997. Dynamic display of spatial data-reliability: Does it benefit the map user? Computers & Geosciences 23, 4 (1997), 409--422. Exploratory Cartograpic Visualisation.
6. Eric D. Feigelson and G. Jogesh Babu. 1992. Statistical Challenges in Modern Astronomy. Springer-Verlag. 155--157 pages. http://www.springer.com/us/book/9781461392927
7. Brian Ferris, Kari Watkins, and Alan Borning. 2010. OneBusAway: Results from Providing Real-time Arrival Information for Public Transit. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 1807--1816.
8. Rocio Garcia-Retamero, Mirta Galesic, and Gerd Gigerenzer. 2010. Do Icon Arrays Help Reduce Denominator Neglect? Medical Decision Making 30, 6 (2010), 672--684. PMID: 20484088.
9. Gerd Gigerenzer and Ulrich Hoffrage. 1995. How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review 102 (1995), 684--704.
10. Dimitra Gkatzia, Oliver Lemon, and Verena Rieser. 2016. Natural Language Generation enhances human decision-making with uncertain information. CoRR abs/1606.03254 (2016). http://arxiv.org/abs/1606.03254
11. Miriam Greis, Thorsten Ohler, Niels Henze, and Albrecht Schmidt. 2015. Investigating Representation Alternatives for Communicating Uncertainty to Non-experts. Springer International Publishing, Cham, 256--263.
12. Jessica Hullman, Paul Resnick, and Eytan Adar. 2015. Hypothetical Outcome Plots Outperform Error Bars and Violin Plots for Inferences About Reliability of Variable Ordering. PLOS ONE 10, 11 (2015). http://idl.cs.washington.edu/papers/hops
13. Harald Ibrekk and M. Granger Morgan. 1987. Graphical Communication of Uncertain Quantities to Nontechnical People. Risk Analysis 7, 4 (1987), 519--529.
14. Edward W. Ishak and Steven K. Feiner. 2006. Content-aware Scrolling. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (UIST '06). ACM, New York, NY, USA, 155--158.
15. James J. Chudley Jesmond J. Allen. 2012. Smashing UX Design: Foundations for Designing Online User Experiences. Vol. 1st edition. Wiley.
16. Susan Joslyn and Jared LeClerc. 2013. Decisions With Uncertainty: The Glass Half Full. Current Directions in Psychological Science 22, 4 (2013), 308--315.
17. Susan Joslyn and Sonia Savelli. 2010. Communicating forecast uncertainty: public perception of weather forecast uncertainty. Meteorological Applications 17, 2 (2010), 180--195.
18. S. L. Joslyn and J. E. LeClerc. 2012. Uncertainty Forecasts Improve Weather-Related Decisions and Attenuate the Effects of Forecast Error. Journal of Experimental Psychology: Applied 18, 1 (2012), 126--140.
19. Malte F. Jung, David Sirkin, Turgut M. Gür, and Martin Steinert. 2015. Displayed Uncertainty Improves Driving Experience and Behavior: The Case of Range Anxiety in an Electric Car. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2201--2210.
20. Matthew Kay, Tara Kola, Jessica R. Hullman, and Sean A. Munson. 2016. When (Ish) is My Bus?: User-centered Visualizations of Uncertainty in Everyday, Mobile Predictive Systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5092--5103.
21. Matthew Kay, Dan Morris, mc schraefel, and Julie A. Kientz. 2013. There's No Such Thing As Gaining a Pound: Reconsidering the Bathroom Scale User Interface. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '13). ACM, New York, NY, USA, 401--410.
22. Matthew Kay, Gregory L. Nelson, and Eric B. Hekler. 2016. Researcher-Centered Design of Statistics: Why Bayesian Statistics Better Fit the Culture and Incentives of HCI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 4521--4532.
23. Michael Leitner and Barbara P. Buttenfield. 2000. Guidelines for the Display of Attribute Certainty. Cartography and Geographic Information Science, 27, 1 (January 2000), 3--14.
24. Limor Nadav-Greenberg and Susan L. Joslyn. 2009. Uncertainty Forecasts Improve Decision Making Among Nonexperts. Journal of Cognitive Engineering and Decision Making 3, 3 (2009), 209--227.
25. Robert A Rigby and D Mikis Stasinopoulos. 2006. Using the Box-Cox t distribution in GAMLSS to model skewness and kurtosis. Statistical Modelling 6, 3 (2006), 209--229.
26. Mark S. Roulston, Gary E. Bolton, Andrew N. Kleit, and Addison L. Sears-Collins. 2006. A laboratory study of the benefits of including uncertainty information in weather forecasts. Weather and Forecasting 21, 1 (2 2006), 116--122.
27. Sonia Savelli and Susan Joslyn. 2013. The Advantages of Predictive Interval Forecasts for Non-Expert Users and the Impact of Visualizations. Applied Cognitive Psychology 27, 4 (2013), 527--541.
28. John Scott. 2000. Understanding Contemporary Society: Theories of the Present. (2000).
29. Vlad V. Simianu, Margaret A. Grounds, Susan L. Joslyn, Jared E. LeClerc, Anne P. Ehlers, Nidhi Agrawal, Rafael Alfonso-Cristancho, Abraham D. Flaxman, and David R. Flum. 2016. Understanding clinical and non-clinical decisions under uncertainty: a scenario-based survey. BMC Medical Informatics and Decision Making 16, 1 (01 Dec 2016), 153.
30. Michael Smithson and Jay Verkuilen. 2006. A better lemon squeezer? Maximum-likelihood regression with beta-distributed dependent variables. Psychological methods 11, 1 (mar 2006), 54--71.
31. Amos Tversky and Daniel Kahneman. 1971. Belief in the law of small numbers. Psychological bulletin 76, 2 (1971), 105.
32. Amos Tversky and Daniel Kahneman. 1974. Judgment under Uncertainty: Heuristics and Biases. Science 185, 4157 (1974), 1124--1131. http://www.jstor.org/stable/1738360
33. T.S. Wallsten, R. Zwick, B. Forsyth, D.V. Budescu, A. Rappaport, and NORTH CAROLINA UNIV AT CHAPEL HILL. 1988. Measuring the Vague Meanings of Probability Terms. Defense Technical Information Center. https://books.google.com.fj/books?id=TE6TNwAACAAJ
34. Leland Wilkinson. 1999. Dot Plots. The American Statistician 53, 3 (August 1999), 276--281.
35. Paul Windschitl and Elke Weber. 1999. The Interpretation of 'Likely' Depends on the Context, But 70Perceived Certainty. 25 (12 1999), 1514--33.
36. Hao-Che Wu, Michael Lindell, and Carla Prater. 2015. Strike probability judgments and protective action recommendations in a dynamic hurricane tracking task. Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards 79, 1 (2015), 355--380. http://EconPapers.repec.org/RePEc: spr:nathaz:v:79:y:2015:i:1:p:355--380
37. Marcel Wunderlich, Kathrin Ballweg, Georg Fuchs, and Tatiana von Landesberger. 2017. Visualization of Delay Uncertainty and its Impact on Train Trip Planning: A Design Study. Computer Graphics Forum (2017).
38. Hang Zhang and Laurence Maloney. 2012. Ubiquitous Log Odds: A Common Representation of Probability and Frequency Distortion in Perception, Action, and Cognition. Frontiers in Neuroscience 6 (2012), 1.