Beat the Machine: Challenging Humans to Find a Predictive Model's “Unknown Unknowns”

Abstract

We present techniques for gathering data that expose errors of automatic predictive models. In certain common settings, traditional methods for evaluating predictive models tend to miss rare but important errors—most importantly, cases for which the model is confident of its prediction (but wrong). In this article, we present a system that, in a game-like setting, asks humans to identify cases that will cause the predictive model-based system to fail. Such techniques are valuable in discovering problematic cases that may not reveal themselves during the normal operation of the system and may include cases that are rare but catastrophic. We describe the design of the system, including design iterations that did not quite work. In particular, the system incentivizes humans to provide examples that are difficult for the model to handle by providing a reward proportional to the magnitude of the predictive model's error. The humans are asked to “Beat the Machine” and find cases where the automatic model (“the Machine”) is wrong. Experiments show that the humans using Beat the Machine identify more errors than do traditional techniques for discovering errors in predictive models, and, indeed, they identify many more errors where the machine is (wrongly) confident it is correct. Furthermore, those cases the humans identify seem to be not simply outliers, but coherent areas missed completely by the model. Beat the Machine identifies the “unknown unknowns.” Beat the Machine has been deployed at an industrial scale by several companies. The main impact has been that firms are changing their perspective on and practice of evaluating predictive models.

“There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know.”

--Donald Rumsfeld

References

1. Josh Attenberg, Panagiotis G. Ipeirotis, and Foster J. Provost. 2011. Beat the machine: Challenging workers to find the unknown unknowns. In Proceedings of the 3rd Human Computation Workshop (HCOMP'11).
2. Josh Attenberg and Foster Provost. 2010. Inactive learning? Difficulties employing active learning in practice. SIGKDD Explorations 12, 2 (2010), 36--41.
3. C. Chow. 1970. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory 16, 1 (Jan. 1970), 41--46.
4. C. K. Chow. 1957. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers EC-6, 4 (Dec. 1957), 247--254.
5. Pedro Domingos. 1999. Metacost: A general method for making classifiers cost-sensitive. In Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 155--164.
6. Charles Elkan. 2001. The foundations of cost-sensitive learning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence- Volume 2. 973--978.
7. David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 3--12.
8. C. Perlich, B. Dalessandro, T. Raeder, O. Stitelman, and F. Provost. 2014. Machine learning for targeted display advertising: Transfer learning in action. Machine Learning 95, 1 (2014), 103--127.
9. Foster Provost and Tom Fawcett. 2013. Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking. O'Reilly Media.
10. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Anna Jerebko, Charles Florin, Gerardo Hermosillo Valadez, Luca Bogoni, and Linda Moy. 2009. Supervised learning from multiple experts: Whom to trust when everyone lies a bit. In Proceedings of the 26th Annual International Conference on Machine Learning. 889--896.
11. R. Reiter. 1977. On Closed World Data Bases. Technical Report. University of British Columbia, Vancouver, BC, Canada.
12. Maytal Saar-Tsechansky and Foster Provost. 2004. Active sampling for class probability estimation and ranking. Machine Learning 54, 2 (2004), 153--178.
13. Robert E. Schapire. 1999. A brief introduction to boosting. In Proceedings of the 16th International Joint Conference on Artificial Intelligence - Volume 2.
14. Burr Settles. 2012. Active Learning. Vol. 6. Morgan & Claypool Publishers.
15. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 614--622.
16. Kurt VanLehn. 1998. Analogy events: How examples are used during problem solving. Cognitive Science 22, 3 (1998), 347--388.
17. Gary M. Weiss. 2010. The impact of small disjuncts on classifier learning. In Data Mining, Robert Stahlbock, Sven F. Crone, and Stefan Lessmann (Eds.). Annals of Information Systems, Vol. 8. Springer, 193--226.
18. Patrick H. Winston. 1970. Learning Structural Descriptions From Examples. Technical Report. Massachusetts Institute of Technology.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Distributed artificial intelligence

  • Machine learning

• Social and professional topics

  • Professional topics

    • Computing and business

    • Computing education

      • Computing education programs

        • Computer science education

        • Information science education

    • Management of computing and information systems

      • File systems management

      • System management