EasyFont: A Style Learning-Based System to Easily Build Your Large-Scale Handwriting Fonts

Abstract

Generating personal handwriting fonts with large amounts of characters is a boring and time-consuming task. For example, the official standard GB18030-2000 for commercial font products consists of 27,533 Chinese characters. Consistently and correctly writing out such huge amounts of characters is usually an impossible mission for ordinary people. To solve this problem, we propose a system, EasyFont, to automatically synthesize personal handwriting for all (e.g., Chinese) characters in the font library by learning style from a small number (as few as 1%) of carefully-selected samples written by an ordinary person. Major technical contributions of our system are twofold. First, we design an effective stroke extraction algorithm that constructs best-suited reference data from a trained font skeleton manifold and then establishes correspondence between target and reference characters via a non-rigid point set registration approach. Second, we develop a set of novel techniques to learn and recover users’ overall handwriting styles and detailed handwriting behaviors. Experiments including Turing tests with 97 participants demonstrate that the proposed system generates high-quality synthesis results, which are indistinguishable from original handwritings. Using our system, for the first time, the practical handwriting font library in a user’s personal style with arbitrarily large numbers of Chinese characters can be generated automatically. It can also be observed from our experiments that recently-popularized deep learning based end-to-end methods are not able to properly handle this task, which implies the necessity of expert knowledge and handcrafted rules for many applications.

References

1. M. Arjovsky, S. Chintala, and L. Bottou. 2017. Wasserstein GAN. arXiv preprint arXiv:1701.07875 (2017).
2. S. Baluja. 2016. Learning typographic style. CoRR abs/1603.04000 (2016). http://arxiv.org/abs/1603.04000.
3. W. Baxter and N. Govindaraju. 2010. Simple data-driven modeling of brushes. In Proc. ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. 135--142.
4. E. Bernhardsson. 2016. Analyzing 50k fonts using deep neural networks. Retrieved from https://erikbern.com/2016/01/21/analyzing-50k-fonts-using-deep-neural-networks/.
5. N. D. F. Campbell and J. Kautz. 2014. Learning a manifold of fonts. ACM Transactions on Graphics 33, 4 (2014), 91.
6. X. Chen, Z. Lian, Y. Tang, and J. Xiao. 2017. An automatic stroke extraction method using manifold learning. In Proc. Eurographics 2017 Short Paper.
7. Já. Dolinsky and H. Takagi. 2009. Analysis and modeling of naturalness in handwritten characters. IEEE Transactions on Neural Networks 20, 10 (2009), 1540--1553.
8. J. Dong, M. Xu, and Y. Pan. 2008. Statistic model-based simulation on calligraphy creation. Chinese Journal of Computers 31, 7 (2008), 1276--1282 (In Chinese).
9. J. L. Elman. 1990. Finding structure in time. Cognitive Science 14, 2 (1990), 179--211.
10. J. Fan. 1990a. Intelligent Chinese character design and an experimental system ICCDS. JCIP 4, 3 (1990), 1--11 (In Chinese).
11. J. Fan. 1990b. A method of computerizing the calligraphical rules basing on CC structure code. JCIP 4, 4 (1990), 43--52 (In Chinese).
12. FontCreator. 2017. High logic. Retrieved from http://www.high-logic.com/.
13. FontLab. 2017. Fontlab. Retrieved from http://www.fontlab.com/.
14. Founder. 2017. Founder group. Retrieved from http://www.foundertype.com/.
15. L. A. Gatys, A. S. Ecker, M. Bethge, S. Hertzmann, and E. Shechtman. 2017. Controlling perceptual factors in neural style transfer. In Proc. CVPR 2017.
16. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative adversarial networks. In Proc. NIPS 2014.
17. T. S. F. Haines, M. Aodha, and G. J. Brostow. 2016. My text in your handwriting. ACM Transactions on Graphics (TOG) 35, 3 (2016), 26.
18. K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep residual learning for image recognition. In Proc. CVPR 2016. 770--778.
19. G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504--507.
20. S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation 9, 8 (1997), 1735--1780.
21. P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros. 2017. Image-to-image translation with conditional adversarial nets. In Proc. CVPR 2017.
22. B. K. Jang and R. T. Chin. 1990. Analysis of thinning algorithms using mathematical morphology. IEEE Transactions on Pattern Analysis and Machine Intelligence 12, 6 (1990), 541--551.
23. L. Jin and X. Gao. 2004. Study of several handwritten Chinese character directional feature extraction approaches. Application Research of Computers 21, 11 (2004), 38--40.
24. H. Khosravi and E. Kabir. 2010. Farsi font recognition based on Sobel--Roberts features. Pattern Recognition Letters 31, 1 (2010), 75--82.
25. A. Krizhevsky, I. Sutskever, and G. E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25. 1097--1105.
26. P. K. Lai, D. Y. Yeung, and M. C. Pong. 1996. A heuristic search approach to Chinese glyph generation using hierarchical character composition. Computer Processing of Oriental Languages 10, 3 (1996), 307--323.
27. B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science 350, 6266 (2015), 1332--1338.
28. N. Lawrence. 2005. Probabilistic non-linear principal component analysis with Gaussian process latent variable models. Journal of Machine Learning Research 6, Nov (2005), 1783--1816.
29. W. Li, Y. Song, and C. Zhou. 2014. Computationally evaluating and synthesizing Chinese calligraphy. Neurocomputing 135, 5 (2014), 299--305.
30. Z. Lian and J. Xiao. 2012. Automatic shape morphing for Chinese characters. In Proc. SIGGRAPH Asia 2012 TB. 2.
31. Z. Lian, B. Zhao, and J. Xiao. 2016. Automatic generation of large-scale handwriting fonts via style learning. In Proc. SIGGRAPH Asia 2016 TB. 12.
32. J. Lin, C. Wang, C. Ting, and R. Chang. 2014. Font generation of personal handwritten Chinese characters. In Proc. IGIP 2014.
33. Z. Lin and L. Wan. 2007. Style-preserving English handwriting synthesis. Pattern Recognition 40, 7 (2007), 2097--2109.
34. J. Long, E. Shelhamer, and T. Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proc. CVPR 2015. 3431--3440.
35. J. Lu, C. Barnes, S. DiVerdi, and A. Finkelstein. 2013. RealBrush: Painting with examples of physical media. In Proc. ACM SIGGRAPH 2013.
36. J. Lu, C. Barnes, C. Wan, P. Asente, R. Mech, and A. Finkelstein. 2014. DecoBrush: Drawing structured decorative patterns by example. In Proc. ACM SIGGRAPH 2014.
37. J. Lu, F. Yu, A. Finkelstein, and S. DiVerdi. 2012. HelpingHand: Example-based stroke stylization. In Proc. ACM SIGGRAPH 2012.
38. A. Myronenko and X. Song. 2010. Point set registration: Coherent point drift. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 12 (2010), 2262--2275.
39. W. Pan, Z. Lian, R. Sun, Y. Tang, and J. Xiao. 2014. FlexiFont: A flexible system to generate personal font libraries. In Proc. DocEng 2014. 17--20.
40. H. Q. Phan, H. Fu, and A. B. Chan. 2015. FlexyFont: Learning transferring rules for flexible typeface synthesis. Computer Graphics Forum 34, 7 (2015), 245--256.
41. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986. Learning representations by back-propagating errors. Nature 323, 6088 (1986), 533--536.
42. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. 2015. ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115, 3 (2015), 211--252.
43. D. Silver, A. Huang, and C. J. Maddison. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529, 7587 (2016), 484--489.
44. K. Simonyan and A. Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014).
45. A. A. Soltani, H. Huang, J. Wu, T. Kulkarni, and J. Tenenbaum. 2017. Synthesizing 3D shapes via modeling multi-view depth maps and silhouettes with deep generative networks. In Proc. CVPR 2017.
46. S. Strassmann. 1986. Hairy brushes. In Proc. ACM SIGGRAPH 1986, Vol. 20. 225--232.
47. Z. Sun, L. Jin, Z. Xie, Z. Feng, and S. Zhang. 2016. Convolutional multi-directional recurrent network for offline handwritten text recognition. In 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR). 240--245.
48. R. Suveeranont and T. Igarashi. 2010. Example-based automatic font generation. In Proc. Smart Graphics. 127--138.
49. Y. Tian. 2016. ReWrite. Retrieved from https://github.com/kaonashi-tyc/Rewrite/.
50. Y. Tian. 2017. ReWrite. Retrieved from https://github.com/kaonashi-tyc/zi2zi/.
51. Y. Wang, H. Wang, C. Pan, and L. Fang. 2008. Style preserving Chinese character synthesis based on hierarchical representation of character. In Proc. ICASSP 2008. 1097--1100.
52. Z. Wang and Y. Pang. 1991. A computer calligraphy system CCCS. Journal of Computer Aided Design and Computer Graphics 3, 1 (1991), 35--40 (In Chinese).
53. S. T. Wong, H. Leung, and H. H. S. Ip. 2008. Model-based analysis of Chinese calligraphy images. Computer Vision and Image Understanding 109, 1 (2008), 69--85.
54. W. Xia and L. Jin. 2009. A Kai style calligraphic beautification method for handwriting chinese character. In Proc. ICDAR 2009. 798--802.
55. S. Xu, H. Jiang, T. Jin, F. Lau, and Y. Pan. 2008. Automatic facsimile of Chinese calligraphic writings. In Computer Graphics Forum, Vol. 27. 1879--1886.
56. S. Xu, H. Jiang, F. C. M. Lau, and Y. Pan. 2007. An intelligent system for Chinese calligraphy. In Proc. The National Conference on Artificial Intelligence, Vol. 22. 1578.
57. S. Xu, T. Jin, H. Jiang, and F. C. M. Lau. 2009. Automatic generation of personal chinese handwriting by capturing the characteristics of personal handwriting. In Proc. IAAI 2009.
58. S. Xu, F. Lau, F. Tang, and Y. Pan. 2003. Advanced design for a realistic virtual brush. In Computer Graphics Forum, Vol. 22. 533--542.
59. S. Xu, F. C. M. Lau, W. K. Cheung, and Y. Pan. 2005. Automatic generation of artistic Chinese calligraphy. IEEE Intelligent Systems 20, 3 (2005), 32--39.
60. T. Yi, Z. Lian, Y. Tang, and J. Xiao. 2014. A data-driven personalized digital ink for Chinese characters. In Proc. MultiMedia Modeling 2014. 254--265.
61. K. Yu, J. Wu, and Y. Zhuang. 2009. Style-consistency calligraphy synthesis system in digital library. In Proc. the 9th ACM/IEEE-CS Joint Conference on Digital Libraries. 145--152.
62. Z. Zhang, C. Zhang, W. Shen, C. Yao, W. Liu, and X. Bai. 2016. Multi-oriented text detection with fully convolutional networks. In Proc. CVPR 2016. 4159--4167.
63. B. Zhou, W. Wang, and Z. Chen. 2011. Easy generation of personal chinese handwritten fonts. In Proc. ICME 2011. 1--6.
64. C. L. Zitnick. 2013. Handwriting beautification using token means. In Proc. ACM SIGGRAPH 2013.

Index Terms

• Computing methodologies

  • Computer graphics

    • Shape modeling

  • Machine learning

    • Machine learning approaches

      • Neural networks