Abstract
Modern search engines aggregate results from different verticals: webpages, news, images, video, shopping, knowledge cards, local maps, and so on. Unlike “ten blue links,” these search results are heterogeneous in nature and not even arranged in a list on the page. This revolution directly challenges the conventional “ranked list” formulation in ad hoc search. Therefore, finding proper presentation for a gallery of heterogeneous results is critical for modern search engines.
We propose a novel framework that learns the optimal page presentation to render heterogeneous results onto search result page (SERP). Page presentation is broadly defined as the strategy to present a set of items on SERP, much more expressive than a ranked list. It can specify item positions, image sizes, text fonts, and any other styles as long as variations are within business and design constraints. The learned presentation is content aware, i.e., tailored to specific queries and returned results. Simulation experiments show that the framework automatically learns eye-catchy presentations for relevant results. Experiments on real data show that simple instantiations of the framework already outperform leading algorithm in federated search result presentation. It means the framework can learn its own result presentation strategy purely from data, without even knowing the “probability ranking principle.”
- Aman Agarwal, Soumya Basu, Tobias Schnabel, and Thorsten Joachims. 2017. Effective evaluation using logged bandit feedback from multiple loggers. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 687--696. Google Scholar
Digital Library
- Eugene Agichtein, Eric Brill, and Susan Dumais. 2006. Improving web search ranking by incorporating user behavior information. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 19--26. Google Scholar
Digital Library
- Jaime Arguello and Fernando Diaz. 2013. Vertical selection and aggregation. In Relevance Ranking for Vertical Search Engines, Bo Long and Yi Chang (Eds.). Elsevier.Google Scholar
- Jaime Arguello, Fernando Diaz, and Jamie Callan. 2011. Learning to aggregate vertical results into web search results. In Proceedings of the 20th ACM Conference on Information and Knowledge Management. 201--210. Google Scholar
Digital Library
- Jaime Arguello, Fernando Diaz, Jamie Callan, and Ben Carterette. 2011. A methodology for evaluating aggregated search results. In Advances in Information Retrieval. Springer, 141--152. Google Scholar
Digital Library
- Jaime Arguello, Fernando Diaz, Jamie Callan, and Jean-Francois Crespo. 2009. Sources of evidence for vertical selection. In Proceedings of the 32nd ACM SIGIR Conference on Research and Development in Information Retrieval. 315--322. Google Scholar
Digital Library
- Jaime Arguello, Fernando Diaz, and Jean-François Paiement. 2010. Vertical selection in the presence of unlabeled verticals. In Proceedings of the 33rd ACM SIGIR Conference on Research and Development in Information Retrieval. 691--698. Google Scholar
Digital Library
- Peter Bailey, Nick Craswell, Ryen W. White, Liwei Chen, Ashwin Satyanarayana, and Seyed M. M. Tahaghoghi. 2010. Evaluating whole-page relevance. In Proceedings of the 33rd ACM SIGIR Conference on Research and Development in Information Retrieval. 767--768. Google Scholar
Digital Library
- G. H. Bakir, T. Hofmann, B. Schölkopf, A. J. Smola, B. Taskar, and S. V. N. Vishwanathan. 2007. Predicting Structured Data. The MIT Press. Google Scholar
Digital Library
- Nicholas J. Belkin. 2008. Some (what) grand challenges for information retrieval. In ACM SIGIR Forum, Vol. 42. ACM, 47--54. Google Scholar
Digital Library
- Christopher J. C. Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning 11, 81 (2010), 23--581.Google Scholar
- Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st ACM SIGIR Conference on Research and Development in Information Retrieval. 335--336. Google Scholar
Digital Library
- Danqi Chen, Weizhu Chen, Haixun Wang, Zheng Chen, and Qiang Yang. 2012. Beyond ten blue links: Enabling user click modeling in federated web search. In Proceedings of the 5th ACM Conference on Web Search and Data Mining. 463--472. Google Scholar
Digital Library
- Tianqi Chen. eXtreme Gradient Boosting Library. Retrieved from https://github.com/dmlc/xgboost.Google Scholar
- Flavio Chierichetti, Ravi Kumar, and Prabhakar Raghavan. 2011. Optimizing two-dimensional search results presentation. In Proceedings of the 4th ACM Conference on Web Search and Data Mining. 257--266. Google Scholar
Digital Library
- Aleksandr Chuklin and Maarten de Rijke. 2016. Incorporating clicks, attention and satisfaction into a search engine result page evaluation model. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM, 175--184. Google Scholar
Digital Library
- Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. 2015. Click models for web search. Synth. Lect. Inf. Concept. Retriev. Serv. 7, 3 (2015), 1--115.Google Scholar
- Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An experimental comparison of click position-bias models. In Proceedings of the 2008 Conference on Web Search and Data Mining. 87--94. Google Scholar
Digital Library
- Ovidiu Dan and Brian D. Davison. 2016. Measuring and predicting search engine user satisfaction. ACM Comput. Surv. 49, 1 (2016), 18. Google Scholar
Digital Library
- Thomas Degris, Martha White, and Richard Sutton. 2012. Off-policy actor-critic. In Proceedings of the International Conference on Machine Learning. Google Scholar
Digital Library
- Thomas Demeester, Dolf Trieschnigg, Dong Nguyen, Ke Zhou, and Djoerd Hiemstra. 2014. Overview of the TREC 2014 Federated Web Search Track. Technical Report. Ghent University.Google Scholar
- Nikhil R. Devanur, Zhiyi Huang, Nitish Korula, Vahab S. Mirrokni, and Qiqi Yan. 2013. Whole-page optimization and submodular welfare maximization with online bidders. In Proceedings of the 14th ACM Conference on Electronic Commerce. 305--322. Google Scholar
Digital Library
- Fernando Diaz, Ryen W. White, Georg Buscher, and Dan Liebling. 2013. Robust models of mouse movement on dynamic web search results pages. In Proceedings of the 22nd ACM Conference on Information and Knowledge Management. 1451--1460. Google Scholar
Digital Library
- Pinar Donmez, Krysta M. Svore, and Christopher J. C. Burges. 2009. On the local optimality of LambdaRank. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 460--467. Google Scholar
Digital Library
- Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais, and Thomas White. 2005. Evaluating implicit measures to improve web search. ACM Trans. Inf. Syst. 23, 2 (2005), 147--168. Google Scholar
Digital Library
- Jerome H. Friedman. 2002. Stochastic gradient boosting. Comput. Stat. Data Anal. 38, 4 (2002), 367--378. Google Scholar
Digital Library
- Hotchkiss Gord. 2007. Eye Tracking On Universal And Personalized Search. Report No 21 9 2007. Retrieved from http://searchengineland.com/eye-tracking-on-universal-and-personalized-search-12233. Technical report, Enquiro Research.Google Scholar
- Laura A. Granka, Thorsten Joachims, and Geri Gay. 2004. Eye-tracking analysis of user behavior in WWW search. In Proceedings of the 27th ACM SIGIR Conference on Research and Development in Information Retrieval. 478--479. Google Scholar
Digital Library
- Ahmed Hassan, Rosie Jones, and Kristina Lisa Klinkner. 2010. Beyond DCG: User behavior as a predictor of a successful search. In Proceedings of the 3rd ACM International Conference on Web Search and Data Mining. ACM, 221--230. Google Scholar
Digital Library
- Marti Hearst. 2019. The evaluation of search user interfaces. In Search User Interfaces. Cambridge University Press.Google Scholar
Digital Library
- Shihao Ji, Ke Zhou, Ciya Liao, Zhaohui Zheng, Gui-Rong Xue, Olivier Chapelle, Gordon Sun, and Hongyuan Zha. 2009. Global ranking by exploiting user clicks. In Proceedings of the 32nd ACM SIGIR Conference on Research and Development in Information Retrieval. 35--42. Google Scholar
Digital Library
- Luo Jie, Sudarshan Lamkhede, Rochit Sapra, Evans Hsu, Helen Song, and Yi Chang. 2013. A unified search federation system based on online user feedback. In Proceedings of the 19th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM, 1195--1203. Google Scholar
Digital Library
- Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 133--142. Google Scholar
Digital Library
- Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th ACM SIGIR Conference on Research and Development in Information Retrieval. 154--161. Google Scholar
Digital Library
- Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased learning-to-rank with biased feedback. In Proceedings of the 10th ACM International Conference on Web Search and Data Mining. ACM, 781--789. Google Scholar
Digital Library
- Diane Kelly and others. 2009. Methods for evaluating interactive information retrieval systems with users. Found. Trends Inf. Retriev. 3, 1--2 (2009), 1--224. Google Scholar
Digital Library
- Harold W. Kuhn. 1955. The hungarian method for the assignment problem. Nav. Res. Logist. Quart. 2, 1--2 (1955), 83--97.Google Scholar
Cross Ref
- Dmitry Lagun and Eugene Agichtein. 2014. Effects of task and domain on searcher attention. In Proceedings of the 37th ACM SIGIR Conference on Research and Development in Information Retrieval. 1087--1090. Google Scholar
Digital Library
- John Langford. Vowpal Wabbit. Retrieved from http://hunch.net/vw.Google Scholar
- Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th Conference on World Wide Web. 661--670. Google Scholar
Digital Library
- Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. 2011. Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining. ACM, 297--306. Google Scholar
Digital Library
- Tie-Yan Liu. 2009. Learning to rank for information retrieval. Found. Trends Inf. Retriev. 3, 3 (2009), 225--331. Google Scholar
Digital Library
- Yiqun Liu, Ye Chen, Jinhui Tang, Jiashen Sun, Min Zhang, Shaoping Ma, and Xuan Zhu. 2015. Different users, different opinions: Predicting search satisfaction with mouse movement information. In Proceedings of the 38th ACM SIGIR Conference on Research and Development in Information Retrieval. Google Scholar
Digital Library
- Zeyang Liu, Yiqun Liu, Ke Zhou, Min Zhang, and Shaoping Ma. 2015. Influence of vertical result in web search examination. In Proceedings of the 38th ACM SIGIR Conference on Research and Development in Information Retrieval. Google Scholar
Digital Library
- Bo Long and Yi Chang. 2014. Relevance Ranking for Vertical Search Engines. Elsevier. Google Scholar
Digital Library
- Jiyun Luo, Sicong Zhang, and Hui Yang. 2014. Win-win search: Dual-agent stochastic game in session search. In Proceedings of the 37th International ACM SIGIR Conference on Research 8 Development in Information Retrieval. ACM, 587--596. Google Scholar
Digital Library
- Pavel Metrikov, Fernando Diaz, Sébastien Lahaie, and Justin Rao. 2014. Whole page optimization: How page elements interact with the position auction. In Proceedings of the 15th ACM Conference on Economics and Computation. 583--600. Google Scholar
Digital Library
- Permutohedron. Retrieved from https://en.wikipedia.org/wiki/Permutohedron.Google Scholar
- Ashok Kumar Ponnuswami, Kumaresh Pattabiraman, Qiang Wu, Ran Gilad-Bachrach, and Tapas Kanungo. 2011. On composition of a federated web search result page: Using online users to provide pairwise preference for heterogeneous verticals. In Proceedings of the 4th ACM Conference on Web Search and Data Mining. 715--724. Google Scholar
Digital Library
- Position bias. Google ads in second position get more attention. Retrieved from http://miratech.com/blog/eye-tracking-google.html.Google Scholar
- Tao Qin, Tie-Yan Liu, Xu-Dong Zhang, De-Sheng Wang, and Hang Li. 2009. Global ranking using continuous conditional random fields. In Advances in Neural Information Processing Systems. 1281--1288. Google Scholar
Digital Library
- Filip Radlinski and Thorsten Joachims. 2005. Query chains: Learning to rank from implicit feedback. In Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery in Data Mining. ACM, 239--248. Google Scholar
Digital Library
- Filip Radlinski and Thorsten Joachims. 2006. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In Proceedings of the National Conference on Artificial Intelligence, Vol. 21. MIT Press, Cambridge, NA, 1406. Google Scholar
Digital Library
- S. E. Robertson. 1977. The probability ranking principle in IR. In J. Document. 294--304.Google Scholar
- Gerard Salton and Christopher Buckley. 1988. Term-weighting approaches in automatic text retrieval. Inf. Process. Manage. 24, 5 (1988), 513--523. Google Scholar
Digital Library
- Anne Schuth, Katja Hofmann, and Filip Radlinski. 2015. Predicting search satisfaction metrics with interleaved comparisons. In Proceedings of the 38th ACM SIGIR Conference on Research and Development in Information Retrieval. Google Scholar
Digital Library
- Anne Schuth, Harrie Oosterhuis, Shimon Whiteson, and Maarten de Rijke. 2016. Multileave gradient descent for fast online learning to rank. In Proceedings of the 9th ACM International Conference on Web Search and Data Mining. ACM, 457--466. Google Scholar
Digital Library
- David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. 2014. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning (ICML’14). 387--395. Google Scholar
Digital Library
- Richard S. Sutton and Andrew G. Barto. 1998. Reinforcement Learning: An Introduction. MIT Press, Cambridge, Ma, Chapters 5 and 9. Google Scholar
Digital Library
- Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems. 1057--1063. Google Scholar
Digital Library
- Krysta M. Svore, Maksims N. Volkovs, and Christopher J. C. Burges. 2011. Learning to rank with multiple objective functions. In Proceedings of the 20th International Conference on World Wide Web. ACM, 367--376. Google Scholar
Digital Library
- Adith Swaminathan and Thorsten Joachims. 2015. Counterfactual risk minimization: Learning from logged bandit feedback. In Proceedings of the International Conference on Machine Learning. 814--823. Google Scholar
Digital Library
- Vertical bias. Eye Tracking Study: Google Results with Videos. Retrieved from https://www.looktracker.com/blog/eye-tracking-case-study/google-results-with-videos-eye-tracking-study.Google Scholar
- Yue Wang, Dawei Yin, Luo Jie, Pengyuan Wang, Makoto Yamada, Yi Chang, and Qiaozhu Mei. 2016. Beyond ranking: Optimizing whole-page presentation. In Proceedings of the 9th ACM International Conference on Web Search and Data Mining. ACM, 103--112. Google Scholar
Digital Library
- Emine Yilmaz, Milad Shokouhi, Nick Craswell, and Stephen Robertson. 2010. Expected browsing utility for web search evaluation. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management. ACM, 1561--1564. Google Scholar
Digital Library
- Yisong Yue and Thorsten Joachims. 2009. Interactively optimizing information retrieval systems as a dueling bandits problem. In Proceedings of the 26th Annual International Conference on Machine Learning. ACM, 1201--1208. Google Scholar
Digital Library
- Yisong Yue, Rajan Patel, and Hein Roehrig. 2010. Beyond position bias: Examining result attractiveness as a source of presentation bias in clickthrough data. In Proceedings of the 19th Conference on World Wide Web. 1011--1018. Google Scholar
Digital Library
- Chengxiang Zhai and John Lafferty. 2001. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of the 24th ACM SIGIR Conference on Research and Development in Information Retrieval. 334--342. Google Scholar
Digital Library
- Yinan Zhang and Chengxiang Zhai. 2015. Information retrieval as card playing: A formal model for optimizing interactive retrieval interface. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 685--694. Google Scholar
Digital Library
- Masrour Zoghi, Tomáš Tunys, Lihong Li, Damien Jose, Junyan Chen, Chun Ming Chin, and Maarten de Rijke. 2016. Click-based hot fixes for underperforming torso queries. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. ACM, 195--204. Google Scholar
Digital Library
Index Terms
Optimizing Whole-Page Presentation for Web Search
Recommendations
Optimizing web search using web click-through data
CIKM '04: Proceedings of the thirteenth ACM international conference on Information and knowledge managementThe performance of web search engines may often deteriorate due to the diversity and noisy information contained within web pages. User click-through data can be used to introduce more accurate description (metadata) for web pages, and to improve the ...
The Whole-Page Optimization via Dynamic Ad Allocation
WWW '18: Companion Proceedings of the The Web Conference 2018Modern search engines present result pages composed of two most prominent types of information: sponsored and organic search results. The whole-page results must satisfy user's information inquiry while sponsored ad alongside the search results has ...
Listwise vs Pagewise: Towards Better Ranking Strategies for Heterogeneous Search Results
WSDM '19: Proceedings of the Twelfth ACM International Conference on Web Search and Data MiningAs heterogeneous verticals account for more and more in search engines, users' preference of search results is largely affected by their presentations. Apart from texts, multimedia information such as images and videos has been widely adopted as it ...






Comments