Adaptation Algorithm and Theory Based on Generalized Discrepancy

ABSTRACT

We present a new algorithm for domain adaptation improving upon the discrepancy minimization algorithm (DM), which was previously shown to outperform a number of popular algorithms designed for this task. Unlike most previous approaches adopted for domain adaptation, our algorithm does not consist of a fixed reweighting of the losses over the training sample. Instead, it uses a reweighting that depends on the hypothesis considered and is based on the minimization of a new measure of generalized discrepancy. We give a detailed description of our algorithm and show that it can be formulated as a convex optimization problem. We also present a detailed theoretical analysis of its learning guarantees, which helps us select its parameters. Finally, we report the results of experiments demonstrating that it improves upon the DM algorithm in several tasks.

References

1. S. Ben-David and R. Urner. On the hardness of domain adaptation and the utility of unlabeled target samples. In Proceedings of ALT, pages 139--153, 2012.
2. S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira. Analysis of representations for domain adaptation. In Proceedings of NIPS, pages 137--144, 2006.
3. S. Ben-David, T. Lu, T. Luu, and D. Pál. Impossibility theorems for domain adaptation. JMLR - Proceedings Track, 9:129--136, 2010.
4. S. Bickel, M. Brückner, and T. Scheffer. Discriminative learning for differing training and test distributions. In Proceedings of ICML, pages 81--88, 2007.
5. J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. Wortman. Learning bounds for domain adaptation. In Proceedings of NIPS, 2007a.
6. J. Blitzer, M. Dredze, and F. Pereira. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of ACL, 2007b.
7. S. Boyd and L. Vandenberghe. Convex optimization. Cambridge University Press, Cambridge, 2004.
8. C. Cortes and M. Mohri. Domain adaptation in regression. In Proceedings of ALT, 2011.
9. C. Cortes and M. Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 9474, 2013.
10. C. Cortes, Y. Mansour, and M. Mohri. Learning bounds for importance weighting. In Proceedings of NIPS, pages 442--450, 2010.
11. C. Cortes, M. Mohri, and A. Muñoz. Adaptation algorithm and theory based on generalized discrepancy. ArXiv:1405.1503, May 2014.
12. H. Daumé III. Frustratingly easy domain adaptation. In Proceedings of ACL, Prague, Czech Republic, 2007.
13. M. Dredze, J. Blitzer, P. P. Talukdar, K. Ganchev, J. Graça, and F. Pereira. Frustratingly hard domain adaptation for dependency parsing. In EMNLP-CoNLL, 2007.
14. K. Fischer, B. Gärtner, and M. Kutz. Fast smallest-enclosing-ball computation in high dimensions. In Algorithms-ESA 2003, pages 630--641. Springer, 2003.
15. P. Germain, A. Habrard, F. Laviolette, and E. Morvant. A PAC-Bayesian approach for domain adaptation with specialization to linear classifiers. In Proceedings of ICML, 2013.
16. J. Hoffman, T. Darrell, and K. Saenko. Continuous manifold based adaptation for evolving visual domains. In Computer Vision and Pattern Recognition (CVPR), 2014.
17. J. Huang, A. J. Smola, A. Gretton, K. M. Borgwardt, and B. Schölkopf. Correcting sample selection bias by unlabeled data. In Proceedings of NIPS, volume 19, pages 601--608, 2006.
18. J. Jiang and C. Zhai. Instance Weighting for Domain Adaptation in NLP. In Proceedings of ACL, pages 264--271, 2007.
19. P. Kumar, J. S. B. Mitchell, and E. A. Yildirim. Computing core-sets and approximate smallest enclosing hyperspheres in high dimensions. In ALENEX, Lecture Notes Comput. Sci, pages 45--55, 2003.
20. C. J. Leggetter and P. C. Woodland. Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models. Computer Speech & Language, 9(2):171--185, 1995.
21. Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In Proceedings of COLT. Omnipress, 2009a.
22. Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation with multiple sources. In Proceedings of NIPS. MIT Press, 2009b.
23. A. M. Martínez. Recognizing imprecisely localized, partially occluded, and expression variant faces from a single sample per class. IEEE Trans. Pattern Anal., 24(6), 2002.
24. M. Mohri and A. Muñoz. New analysis and algorithm for learning with drifting distributions. In Proceedings of ALT. Springer, 2012.
25. S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199--210, 2011.
26. C. E. Rasmussen, R. M. Neal, G. Hinton, D. van Camp, M. R. Z. Ghahramani, R. Kustra, and R. Tibshirani. The delve project. http://www.cs.toronto.edu/~delve/data/datasets.html, 1996. version 1.0.
27. S. Schönherr. Quadratic Programming in Geometric Optimization: Theory, Implementation, and applications. PhD thesis, Swiss Federal Institute of Technology, 2002.
28. M. Sugiyama, S. Nakajima, H. Kashima, P. von Bünau, and M. Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In Proceedings of NIPS, pages 1433--1440, 2007.
29. T. Tommasi, T. Tuytelaars, and B. Caputo. A testbed for cross-dataset analysis. CoRR, abs/1402.5923, 2014. URL http://arxiv.org/abs/1402.5923.
30. E. Welzl. Smallest enclosing disks (balls and ellipsoids). In New results and new trends in computer science (Graz, 1991), volume 555 of Lecture Notes in Comput. Sci., pages 359--370. Springer, Berlin, 1991.
31. J. Wen, C. Yu, and R. Greiner. Robust learning under uncertain test distributions: Relating covariate shift to model misspecification. In Proceedings of ICML, pages 631--639, 2014.
32. E. A. Yildirim. Two algorithms for the minimum enclosing ball problem. SIAM Journal on Optimization, 19(3):1368--1391, 2008.
33. C. Zhang, L. Zhang, and J. Ye. Generalization bounds for domain adaptation. In Proceedings of NIPS, pages 1790--1798. MIT Press, 2012.
34. K. Zhang, B. Schölkopf, K. Muandet, and Z. Wang. Domain adaptation under target and conditional shift. In Proceedings of ICML 2013, pages 819-- 827, 2013.

Index Terms

• Computing methodologies

  • Machine learning

• Mathematics of computing

  • Mathematical analysis

    • Mathematical optimization