Snorkel MeTaL: Weak Supervision for Multi-Task Learning

ABSTRACT

Many real-world machine learning problems are challenging to tackle for two reasons: (i) they involve multiple sub-tasks at different levels of granularity; and (ii) they require large volumes of labeled training data. We propose Snorkel MeTaL, an end-to-end system for multi-task learning that leverages weak supervision provided at multiple levels of granularity by domain expert users. In MeTaL, a user specifies a problem consisting of multiple, hierarchically-related sub-tasks---for example, classifying a document at multiple levels of granularity---and then provides labeling functions for each sub-task as weak supervision. MeTaL learns a re-weighted model of these labeling functions, and uses the combined signal to train a hierarchical multi-task network which is automatically compiled from the structure of the sub-tasks. Using MeTaL on a radiology report triage task and a fine-grained news classification task, we achieve average gains of 11.2 accuracy points over a baseline supervised approach and 9.5 accuracy points over the predictions of the user-provided labeling functions.

References

1. S. H. Bach, B. He, A. J. Ratner, and C. Ré. Learning the structure of generative models without labeled data. In ICML, 2017.
2. R. Caruana. Multitask learning: A knowledge-based source of inductive bias. In ICML, pages 41--48, 1993.
3. T. Mitchell et. al. Never-ending learning. In AAAI, 2015.
4. A. Karpathy. Software 2.0. medium.com/@karpathy/software-2-0-a64152b37c35.
5. H. Kazuma, X. Caiming, T. Yoshimasa, and R. Socher. A joint many-task model: Growing a neural network for multiple NLP tasks. CoRR, abs/1611.01587, 2016.
6. D. D. Lewis, Y. Yang, T. G. Rose, and F. Li. Rcv1: A new benchmark collection for text categorization research. JMLR, 5(Apr):361--397, 2004.
7. M. Mintz, S. Bills, R. Snow, and D. Jurafsky. Distant supervision for relation extraction without labeled data. In Proc ACL, pages 1003--1011, 2009.
8. Volodymyr Mnih and Geoffrey E Hinton. Learning to label aerial images from noisy data. In ICML, pages 567--574, 2012.
9. National Institutes of Health. Open-i. 2017. URL https://openi.nlm.nih.gov/.
10. A. et. al. Paszke. Automatic differentiation in pytorch. In NIPS-W, 2017.
11. E. Platanios, H. Poon, T. M. Mitchell, and E. J. Horvitz. Estimating accuracy from unlabeled data: A probabilistic logic approach. In NIPS, pages 4364--4373, 2017.
12. A.J. Ratner, C.M. De Sa, S. Wu, D. Selsam, and C. Ré. Data programming: Creating large training sets, quickly. In Adv Neural Inf Process Syst, pages 3567--3575, 2016.
13. A.J. Ratner, S.H. Bach, H. Ehrenberg, J. Fries, S. Wu, and C. Ré. Snorkel: Rapid training data creation with weak supervision. In VLDB, 2018.
14. T. Rekatsinas, M. Joglekar, H. Garcia-Molina, A. Parameswaran, and C. Ré. Slim-fast: Guaranteed results for data fusion and source reliability. In SIGMOD, 2017.
15. S. Ruder. An overview of multi-task learning in deep neural networks. CoRR, abs/1706.05098, 2017. URL http://arxiv.org/abs/1706.05098.
16. A. Søgaard and Y. Goldberg. Deep multi-task learning with low level tasks supervised at lower layers. In Proc ACL, volume 2, pages 231--235, 2016.
17. T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang. Learning from massive noisy labeled data for image classification. In CVPR, pages 2691--2699, 2015.

Index Terms

• Computing methodologies

  • Machine learning