skip to main content
research-article

Dissimilarity-Based Regularized Learning of Charts

Published:12 November 2021Publication History
Skip Abstract Section

Abstract

Chart images exhibit significant variabilities that make each image different from others even though they belong to the same class or categories. Classification of charts is a major challenge because each chart class has variations in features, structure, and noises. However, due to the lack of affiliation between the dissimilar features and the structure of the chart, it is a challenging task to model these variations for automatic chart recognition. In this article, we present a novel dissimilarity-based learning model for similar structured but diverse chart classification. Our approach jointly learns the features of both dissimilar and similar regions. The model is trained by an improved loss function, which is fused by a structural variation-aware dissimilarity index and incorporated with regularization parameters, making the model more prone toward dissimilar regions. The dissimilarity index enhances the discriminative power of the learned features not only from dissimilar regions but also from similar regions. Extensive comparative evaluations demonstrate that our approach significantly outperforms other benchmark methods, including both traditional and deep learning models, over publicly available datasets.

REFERENCES

  1. [1] Abouelenien Mohamed and Yuan Xiaohui. 2013. Boosting for learning from multiclass data sets via a regularized loss function. In Proceedings of the 2013 IEEE International Conference on Granular Computing (GrC’13). IEEE, Los Alamitos, CA, 49.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Amara Jihen, Kaur Pawandeep, Owonibi Michael, and Bouaziz Bassem. 2017. Convolutional neural network based chart image classification. In Proceedings of the 25th International Conference in Central Europe on Computer Graphics, Visualization, and Computer Vision.Google ScholarGoogle Scholar
  3. [3] Beddoe Jennifer. 2014. Study.com—Bar Graph Definition, Types and Examples. Retrieved September 16, 2021 from https://study.com/academy/lesson/bar-graph-definition-types-examples.html.Google ScholarGoogle Scholar
  4. [4] Cao Jie, Qiu Yinping, Chang Dongliang, Li Xiaoxu, and Ma Zhanyu. 2019. Dynamic attention loss for small-sample image classification. In Proceedings of the 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC’19). IEEE, Los Alamitos, CA, 7579.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Chagas Paulo, Akiyama Rafael, Meiguins Aruanda, Santos Carlos, Saraiva Filipe, Meiguins Bianchi, and Morais Jefferson. 2018. Evaluation of convolutional neural network architectures for chart image classification. In Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN’18). IEEE, Los Alamitos, CA, 18.Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Chen Min and Golan Amos. 2015. What may visualization processes optimize? IEEE Transactions on Visualization and Computer Graphics 22, 12 (2015), 26192632. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Cheng Beibei, Stanley R. Joe, Antani Sameer, and Thoma George R.. 2013. Graphical figure classification using data fusion for integrating text and image features. In Proceedings of the 2013 12th International Conference on Document Analysis and Recognition. IEEE, Los Alamitos, CA, 693697. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Cheng Gong, Han Junwei, Zhou Peicheng, and Xu Dong. 2018. Learning rotation-invariant and Fisher discriminative convolutional neural networks for object detection. IEEE Transactions on Image Processing 28, 1 (2018), 265278. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Choi Jinho, Jung Sanghun, Park Deok Gun, Choo Jaegul, and Elmqvist Niklas. 2019. Visualizing for the non-visual: Enabling the visually impaired to use visualization. Computer Graphics Forum (2019), 249260.Google ScholarGoogle Scholar
  10. [10] Dalal Navneet and Triggs Bill. 2005. Histograms of oriented gradients for human detection. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 1. IEEE, Los Alamitos, CA, 886893. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Davila Kenny, Kota Bhargava Urala, Setlur Srirangaraj, Govindaraju Venu, Tensmeyer Christopher, Shekhar Sumit, and Chaudhry Ritwick. 2019. ICDAR 2019 competition on harvesting raw tables from infographics (CHART-infographics). In Proceedings of the 2019 International Conference on Document Analysis and Recognition (ICDAR’19). IEEE, Los Alamitos, CA, 15941599.Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Demirkaya Ahmet, Chen Jiasi, and Oymak Samet. 2020. Exploring the role of loss functions in multiclass classification. In Proceedings of the 2020 54th Annual Conference on Information Sciences and Systems (CISS’20). IEEE, Los Alamitos, CA, 15.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Haghighat Mohammad, Abdel-Mottaleb Mohamed, and Alhalabi Wadee. 2016. Discriminant correlation analysis: Real-time feature level fusion for multimodal biometric recognition. IEEE Transactions on Information Forensics and Security 11, 9 (2016), 19841996. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Hinton Geoffrey. 2018. Neural Networks for Machine Learning Online Course. Retrieved September 16, 2021 from https://www.coursera.org/learn/neural-networks/home/welcome.Google ScholarGoogle Scholar
  15. [15] Hong Chaoqun, Yu Jun, You Jane, Chen Xuhui, and Tao Dapeng. 2015. Multi-view ensemble manifold regularization for 3D object recognition. Information Sciences 320 (2015), 395405. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. [16] Huang Weihua and Tan Chew Lim. 2007. A system for understanding imaged infographics and its applications. In Proceedings of the 2007 ACM Symposium on Document Engineering. 918. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Huang Weihua, Tan Chew Lim, and Leow Wee Kheng. 2004. Elliptic arc vectorization for 3D pie chart recognition. In Proceedings of the 2004 International Conference on Image Processing (ICIP’04), Vol. 5. IEEE, Los Alamitos, CA, 28892892.Google ScholarGoogle Scholar
  18. [18] Huang Weihua, Zong Siqi, and Tan Chew Lim. 2007. Chart image classification using multiple-instance learning. In Proceedings of the 2007 IEEE Workshop on Applications of Computer Vision (WACV’07). IEEE, Los Alamitos, CA, 2727. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Jiang Bo and Lin Doudou. 2018. Graph Laplacian regularized graph convolutional networks for semi-supervised learning. arXiv:1809.09839.Google ScholarGoogle Scholar
  20. [20] Jung Daekyoung, Kim Wonjae, Song Hyunjoo, Hwang Jeong-In, Lee Bongshin, Kim Bohyoung, and Seo Jinwook. 2017. ChartSense: Interactive data extraction from chart images. In Proceedings of the 2017 Chi Conference on Human Factors in Computing Systems. 67066717. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Jurio Aranzazu, Bustince Humberto, Pagola Miguel, Couto Pedro, and Pedrycz Witold. 2014. New measures of homogeneity for image processing: An application to fingerprint segmentation. Soft Computing 18, 6 (2014), 10551066. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Kahou Samira Ebrahimi, Michalski Vincent, Atkinson Adam, Kádár Ákos, Trischler Adam, and Bengio Yoshua. 2017. Figureqa: An annotated figure dataset for visual reasoning. arXiv:1710.07300.Google ScholarGoogle Scholar
  23. [23] Karthikeyani V. and Nagarajan S.. 2012. Machine learning classification algorithms to recognize chart types in portable document format (PDF) files. International Journal of Computer Applications 39, 2 (2012), 15.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Kim Daehyun, Ramesh Balaji Polepalli, and Yu Hong. 2011. Automatic figure classification in bioscience literature. Journal of Biomedical Informatics 44, 5 (2011), 848858. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] Kotu Vijay and Deshpande Bala. 2018. Data Science: Concepts and Practice. Morgan Kaufmann.Google ScholarGoogle Scholar
  26. [26] Krizhevsky Alex, Sutskever Ilya, and Hinton Geoffrey E.. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems—Volume 1(NIPS’12). 1097–1105. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. [27] LeCun Yann, Bottou Léon, Bengio Yoshua, and Haffner Patrick. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11 (1998), 22782324.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Liu Weifeng, Ma Xueqi, Zhou Yicong, Tao Dapeng, and Cheng Jun. 2018. -Laplacian regularization for scene recognition. IEEE Transactions on Cybernetics 49, 8 (2018), 29272940.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Lowe David G.. 1999. Object recognition from local scale-invariant features. In Proceedings of the 7th IEEE International Conference on Computer Vision, Vol. 2. IEEE, Los Alamitos, CA, 11501157. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Mishchenko Ales and Vassilieva Natalia. 2011. Model-based chart image classification. In Proceedings of the International Symposium on Visual Computing. 476485. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Negotia John. 2016. Pie Chart and Donut Chart. https://code.tutsplus.comRetrieved November 14, 2016 from.Google ScholarGoogle Scholar
  32. [32] Nilsback M.-E. and Zisserman Andrew. 2006. A visual vocabulary for flower classification. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), Vol. 2. IEEE, Los Alamitos, CA, 14471454. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. [33] Ojala Timo, Pietikainen Matti, and Maenpaa Topi. 2002. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 7 (2002), 971987. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Oliva Aude and Torralba Antonio. 2001. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision 42, 3 (2001), 145175. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Pandey Ram Krishna, Ramakrishnan A. G., and Karmakar Souvik. 2019. Effects of modifying the input features and the loss function on improving emotion classification. In Proceedings of the 2019 IEEE Region 10 Conference (TENCON’19). IEEE, Los Alamitos, CA, 11591162.Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] Peng Yong, Wang Suhang, Long Xianzhong, and Lu Bao-Liang. 2015. Discriminative graph regularized extreme learning machine and its application to face recognition. Neurocomputing 149 (2015), 340353. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Poco Jorge and Heer Jeffrey. 2017. Reverse-engineering visualizations: Recovering visual encodings from chart images. Computer Graphics Forum 36 (2017), 353363. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. [38] Prasad V. Shiv Naga, Siddiquie Behjat, Golbeck Jennifer, and Davis Larry S.. 2007. Classifying computer generated charts. In Proceedings of the 2007 International Workshop on Content-Based Multimedia Indexing. IEEE, Los Alamitos, CA, 8592.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Redko Alla. 2014. Oracle Docs. Retrieved September 16, 2021 from docs.oracle.com.Google ScholarGoogle Scholar
  40. [40] Sandryhaila Aliaksei and Moura Jose M. F.. 2013. Classification via regularization on graphs. In Proceedings of the 2013 IEEE Global Conference on Signal and Information Processing. IEEE, Los Alamitos, CA, 495498.Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Savva Manolis, Kong Nicholas, Chhajta Arti, Fei-Fei Li, Agrawala Maneesh, and Heer Jeffrey. 2011. Revision: Automated classification, analysis and redesign of chart images. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. 393402. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. [42] Setty Shankar, Husain Moula, Beham Parisa, Gudavalli Jyothi, Kandasamy Menaka, Vaddi Radhesyam, Hemadri Vidyagouri, et al. 2013. Indian movie face database: A benchmark for face recognition under wide variations. In Proceedings of the 2013 4th National Conference on Computer Vision, Pattern Recognition, Image Processing, and Graphics (NCVPRIPG’13). IEEE, Los Alamitos, CA, 15.Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Shao Mingyan and Futrelle R.. 2005. Graphics recognition in PDF documents. In Proceedings of the 6th International Conference on Graphics Recognition (GREC’05).Google ScholarGoogle Scholar
  44. [44] Shukla Sudhindra and Samal Ashok. 2008. Recognition and quality assessment of data charts in mixed-mode documents. International Journal of Document Analysis and Recognition 11, 3 (2008), 111. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. [45] Siegel Noah, Horvitz Zachary, Levin Roie, Divvala Santosh, and Farhadi Ali. 2016. FigureSeer: Parsing result-figures in research papers. In Proceedings of the European Conference on Computer Vision. 664680.Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Simonyan Karen and Zisserman Andrew. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556.Google ScholarGoogle Scholar
  47. [47] Song Kaikai, Li Feng, Long Fei, Wang Junping, and Ling Qiang. 2018. Discriminative deep feature learning for semantic-based image retrieval. IEEE Access 6 (2018), 4426844280.Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Starr Ben. 2015. How to Design Area Charts. https://visage.co/data-visualization-101-area-charts/Retrieved January 13, 2015 from.Google ScholarGoogle Scholar
  49. [49] Szegedy Christian, Liu Wei, Jia Yangqing, Sermanet Pierre, Reed Scott, Anguelov Dragomir, Erhan Dumitru, Vanhoucke Vincent, and Rabinovich Andrew. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 19.Google ScholarGoogle ScholarCross RefCross Ref
  50. [50] Ta Vinh Thong, Lezoray Olivier, and Elmoataz Abderrahim. 2007. Graph based semi and unsupervised classification and segmentation of microscopic images. In Proceedings of the 2007 IEEE International Symposium on Signal Processing and Information Technology. IEEE, Los Alamitos, CA, 11601165.Google ScholarGoogle ScholarCross RefCross Ref
  51. [51] Tang Binbin, Liu Xiao, Lei Jie, Song Mingli, Tao Dapeng, Sun Shuifa, and Dong Fangmin. 2016. DeepChart: Combining deep convolutional networks and deep belief networks in chart classification. Signal Processing 124 (2016), 156161. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. [52] Wang Lin, Wang Chaoli, Sun Zhanquan, Cheng Shuqun, and Guo Lei. 2020. Class balanced loss for image classification. IEEE Access 8 (2020), 8114281153.Google ScholarGoogle ScholarCross RefCross Ref
  53. [53] Williams Shannon. n.d. Lucid Charts. Retrieved September 16, 2021 from https://www.lucidchart.com/blog/how-to-make-a-bubble-chart-in-excel.Google ScholarGoogle Scholar
  54. [54] Xu Yong, Zhong Zuofeng, Yang Jian, You Jane, and Zhang David. 2016. A new discriminative sparse representation method for robust face recognition via {2} regularization. IEEE Transactions on Neural Networks and Learning Systems 28, 10 (2016), 22332242.Google ScholarGoogle ScholarCross RefCross Ref
  55. [55] Ye Minxiang, Stankovic Vladimir, Stankovic Lina, and Cheung Gene. 2019. Deep graph regularized learning for binary classification. In Proceedings of the 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’19). IEEE, Los Alamitos, CA, 35373541.Google ScholarGoogle ScholarCross RefCross Ref
  56. [56] Yuan Yuan, Mou Lichao, and Lu Xiaoqiang. 2015. Scene recognition by manifold regularized deep learning architecture. IEEE Transactions on Neural Networks and Learning Systems 26, 10 (2015), 22222233.Google ScholarGoogle ScholarCross RefCross Ref
  57. [57] Zhang Shiliang, Lei Ming, Ma Bin, and Xie Lei. 2019. Robust audio-visual speech recognition using bimodal DFSMN with multi-condition training and dropout regularization. In Proceedings of the 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’19). IEEE, Los Alamitos, CA, 65706574.Google ScholarGoogle ScholarCross RefCross Ref
  58. [58] Zhou Y. P. and Tan Chew Lim. 2001. Learning-based scientific chart recognition. In Proceedings of the 4th IAPR International Workshop on Graphics Recognition (GREC’01). 482492.Google ScholarGoogle Scholar

Index Terms

  1. Dissimilarity-Based Regularized Learning of Charts

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 17, Issue 4
      November 2021
      529 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3492437
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 12 November 2021
      • Accepted: 1 March 2021
      • Revised: 1 January 2021
      • Received: 1 August 2020
      Published in tomm Volume 17, Issue 4

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!