Abstract
Web data record extraction aims at extracting a set of similar object records from a single webpage. These records have similar attributes or fields and are presented with a regular format in a coherent region of the page. To tackle this problem, most existing works analyze the DOM tree of an input page. One major limitation of these methods is that the lack of a global view in detecting data records from an input page results in a myopic decision. Their brute-force searching manner in detecting various types of records degrades the flexibility and robustness. We propose a Structure-Knowledge-Oriented Global Analysis (Skoga) framework which can perform robust detection of different-kinds of data records and record regions. The major component of the Skoga framework is a DOM structure-knowledge-driven detection model which can conduct a global analysis on the DOM structure to achieve effective detection. The DOM structure knowledge consists of background knowledge as well as statistical knowledge capturing different characteristics of data records and record regions, as exhibited in the DOM structure. The background knowledge encodes the semantics of labels indicating general constituents of data records and regions. The statistical knowledge is represented by some carefully designed features that capture different characteristics of a single node or a node group in the DOM. The feature weights are determined using a development dataset via a parameter estimation algorithm based on a structured output support vector machine. An optimization method based on the divide-and-conquer principle is developed making use of the DOM structure knowledge to quantitatively infer and recognize appropriate records and regions for a page. Extensive experiments have been conducted on four datasets. The experimental results demonstrate that our framework achieves higher accuracy compared with state-of-the-art methods.
- Arasu, A. and Garcia-Molina, H. 2003. Extracting structured data from web pages. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD) 337--348. Google Scholar
Digital Library
- Arocena, G. O. and Mendelzon, A. O. 1999. Weboql: Restructuring documents, databases, and webs. Theory Practice Object Syst. 5, 127--141. Google Scholar
Digital Library
- Baumgartner, R., Gottlob, G., and Herzog, M. 2009. Scalable Web data extraction for online market intelligence. Proc. VLDB Endow. 2, 1512--1523. Google Scholar
Digital Library
- Bing, L., Lam, W., and Gu, Y. 2011. Towards a unified solution: Data record region detection and segmentation. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM). 1265--1274. Google Scholar
Digital Library
- Bing, L., Lam, W., and Wong, T.-L. 2013. Wikipedia entity expansion and attribute extraction from the Web using semi-supervised learning. In Proceedings of the 6th ACM International Conference on Web Search and Data Mining (WSDM). 567--576. Google Scholar
Digital Library
- Buttler, D., Liu, L., and Pu, C. 2001. A fully automated object extraction system for the World Wide Web. In Proceedings of the the 21st International Conference on Distributed Computing Systems (ICDCS). 361--370. Google Scholar
Digital Library
- Cafarella, M. J., Halevy, A., and Madhavan, J. 2011. Structured data on the Web. Comm. ACM 54, 72--79. Google Scholar
Digital Library
- Cafarella, M. J., Halevy, A., Wang, D. Z., Wu, E., and Zhang, Y. 2008. Webtables: Exploring the power of tables on the Web. Proc. VLDB Endow. 1, 538--549. Google Scholar
Digital Library
- Cai, D., Yu, S., Wen, J.-R., and Ma, W.-Y. 2003. VIPS: A vision-based page segmentation algorithm. Tech. rep. MSR-TR-2003-79. Microsoft Research.Google Scholar
- Chang, C.-H., Kayed, M., Girgis, M. R., and Shaalan, K. F. 2006. A survey of Web information extraction systems. IEEE Trans. Knowl. Data Eng. 18, 1411--1428. Google Scholar
Digital Library
- Chang, C.-H. and Lui, S.-C. 2001. Iepad: Information extraction based on pattern discovery. In Proceedings of the 10th International Conference on World Wide Web (WWW). 681--688. Google Scholar
Digital Library
- Crescenzi, V., Mecca, G., and Merialdo, P. 2001. Roadrunner: Towards automatic data extraction from large Web sites. In Proceedings of the 27th International Conference on Very Large Data Bases (VLDB). 109--118. Google Scholar
Digital Library
- Elmeleegy, H., Madhavan, J., and Halevy, A. 2009. Harvesting relational tables from lists on the Web. Proc. VLDB Endow. 2, 1078--1089. Google Scholar
Digital Library
- Embley, D. W., Campbell, D. M., Jiang, Y. S., Liddle, S. W., Lonsdale, D. W., Ng, Y.-K., and Smith, R. D. 1999a. Conceptual-model-based data extraction from multiple-record Web pages. Data Knowl. Eng. 31, 227--251. Google Scholar
Digital Library
- Embley, D. W., Jiang, Y., and Ng, Y.-K. 1999b. Record-boundary discovery in Web documents. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD). 467--478. Google Scholar
Digital Library
- Gatterbauer, W., Bohunsky, P., Herzog, M., Krüpl, B., and Pollak, B. 2007. Towards domain-independent information extraction from Web tables. In Proceedings of the 16th International Conference on World Wide Web (WWW). 71--80. Google Scholar
Digital Library
- Hao, Q., Cai, R., Pang, Y., and Zhang, L. 2011. From one tree to a forest: A unified solution for structured Web data extraction. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). 775--784. Google Scholar
Digital Library
- He, B., Patel, M., Zhang, Z., and Chang, K. C.-C. 2007. Accessing the deep Web. Comm. ACM 50, 94--101. Google Scholar
Digital Library
- Hogue, A. and Karger, D. 2005. Thresher: Automating the unwrapping of semantic content from the World Wide Web. In Proceedings of the 14th International Conference on World Wide Web (WWW). 86--95. Google Scholar
Digital Library
- Hsu, C.-N. and Dung, M.-T. 1998. Generating finite-state transducers for semi-structured data extraction from the web. Info. Syst. 23, 521--538. Google Scholar
Digital Library
- Kayed, M. and Chang, C.-H. 2010. Fivatech: Page-level Web data extraction from template pages. IEEE Trans. Knowl. Data Eng. 22, 249--263. Google Scholar
Digital Library
- Kushmerick, N. 2000. Wrapper induction: Efficiency and expressiveness. Artificial Intell. 118, 15--68. Google Scholar
Digital Library
- Laender, A. H. F., Ribeiro-Neto, B., and da Silva, A. S. 2002. Debye - date extraction by example. Data Knowl. Eng. 40, 121--154. Google Scholar
Digital Library
- Liu, B., Grossman, R., and Zhai, Y. 2003. Mining data records in Web pages. In Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 601--606. Google Scholar
Digital Library
- Liu, B. and Zhai, Y. 2005. Net -- a system for extracting Web data from flat and nested data records. In Proceedings of the 6th International Conference on Web Information Systems Engineering (WISE). 487--495. Google Scholar
Digital Library
- Liu, L., Pu, C., and Han, W. 2000. Xwrap: An xml-enabled wrapper construction system for web information sources. In Proceedings of the 16th International Conference on Data Engineering (ICDE). 611--621. Google Scholar
Digital Library
- Liu, W., Meng, X., and Meng, W. 2010. Vide: A vision-based approach for deep Web data extraction. IEEE Trans. Knowl. Data Eng. 22, 447--460. Google Scholar
Digital Library
- Luo, X., Xu, Z., Yu, J., and Chen, X. 2011. Building association link network for semantic link on Web resources. IEEE Trans. Autom. Sci. Eng. 8, 3, 482--494.Google Scholar
Cross Ref
- Madhavan, J., Ko, D., Kot, L., Ganapathy, V., Rasmussen, A., and Halevy, A. 2008. Google's deep Web crawl. Proc. VLDB Endow. 1, 1241--1252. Google Scholar
Digital Library
- Miao, G., Tatemura, J., Hsiung, W.-P., Sawires, A., and Moser, L. E. 2009. Extracting data records from the Web using tag path clustering. In Proceedings of the 18th International Conference on World Wide Web (WWW). 981--990. Google Scholar
Digital Library
- Muslea, I., Minton, S., and Knoblock, C. A. 2001. Hierarchical wrapper induction for semistructured information sources. Auton. Agents Multi-Agent Syst. 4, 93--114. Google Scholar
Digital Library
- Ryan, M. S. and Nudd, G. R. 1993. The viterbi algorithm. Tech. rep. Department of Computer Science, University of Warnick. Google Scholar
Digital Library
- Simon, K. and Lausen, G. 2005. Viper: Augmenting automatic information extraction with visual perceptions. In Proceedings of the 14th ACM International Conference on Information and Knowledge Management (CIKM). 381--388. Google Scholar
Digital Library
- Sleiman, H. A. and Corchuelo, R. 2012. A survey on region extractors from Web documents. IEEE Trans. Knowl. Data Eng. 99. Google Scholar
Digital Library
- Song, X., Liu, J., Cao, Y., Lin, C.-Y., and Hon, H.-W. 2010. Automatic extraction of Web data records containing user-generated content. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM). 39--48. Google Scholar
Digital Library
- Su, W., Wang, J., and Lochovsky, F. H. 2009. Ode: Ontology-assisted data extraction. ACM Trans. Database Syst. 34, 12:1--12:35. Google Scholar
Digital Library
- Tsochantaridis, I., Joachims, T., Hofmann, T., and Altun, Y. 2005. Large margin methods for structured and interdependent output variables. J. Machine Learn. Res. 6, 1453--1484. Google Scholar
Digital Library
- Wang, J. and Lochovsky, F. H. 2003. Data extraction and label assignment for Web databases. In Proceedings of the 12th International Conference on World Wide Web (WWW). 187--196. Google Scholar
Digital Library
- Wong, T.-L. and Lam, W. 2010. Learning to adapt Web information extraction knowledge and discovering new attributes via a bayesian approach. IEEE Trans. Knowl. Data Eng. 22, 523--536. Google Scholar
Digital Library
- Wong, T.-L., Lam, W., and Chan, S.-K. 2006. Collaborative information extraction and mining from multiple Web documents. In Proceedings of the SIAM International Conference on Data Mining (SDM). 440--450.Google Scholar
- Wong, T.-L., Lam, W., and Chen, B. 2009. Mining employment market via text block detection and adaptive cross-domain information extraction. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). 283--290. Google Scholar
Digital Library
- Yamada, Y., Craswell, N., Nakatoh, T., and Hirokawa, S. 2004. Testbed for information extraction from deep Web. In Proceedings of the 13th International World Wide Web Conference on Alternate Track Papers & Posters (WWW Alt). 346--347. Google Scholar
Digital Library
- Yang, C., Cao, Y., Nie, Z., Zhou, J., and Wen, J.-R. 2010. Closing the loop in Webpage understanding. IEEE Trans. Knowl. Data Eng. 22, 639--650. Google Scholar
Digital Library
- Yang, J.-M., Cai, R., Wang, Y., Zhu, J., Zhang, L., and Ma, W.-Y. 2009. Incorporating site-level knowledge to extract structured data from Web forums. In Proceedings of the 18th International Conference on World Wide Web WWW. 181--190. Google Scholar
Digital Library
- Zhai, Y. and Liu, B. 2006. Structured data extraction from the Web based on partial tree alignment. IEEE Trans. Knowl. Data Eng. 18, 1614--1628. Google Scholar
Digital Library
- Zhai, Y. and Liu, B. 2007. Extracting Web data using instance-based learning. J. World Wide Web 10, 2, 113--132. Google Scholar
Digital Library
- Zhao, B., Yin, X., and Xing, E. P. 2011. Max margin learning on domain-independent Web information extraction. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM). 1305--1310. Google Scholar
Digital Library
- Zhao, H., Meng, W., Wu, Z., Raghavan, V., and Yu, C. 2005. Fully automatic wrapper generation for search engines. In Proceedings of the 14th International Conference on World Wide Web (WWW). 66--75. Google Scholar
Digital Library
- Zhao, H., Meng, W., and Yu, C. 2006. Automatic extraction of dynamic record sections from search engine result pages. In Proceedings of the 32nd International Conference on Very Large Data Bases (VLDB). 989--1000. Google Scholar
Digital Library
- Zheng, S., Song, R., Wen, J.-R., and Giles, C. L. 2009. Efficient record-level wrapper induction. In Proceeding of the 18th ACM Conference on Information and Knowledge Management (CIKM). 47--56. Google Scholar
Digital Library
- Zheng, S., Song, R., Wen, J.-R., and Wu, D. 2007. Joint optimization of wrapper generation and template detection. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 894--902. Google Scholar
Digital Library
- Zhu, J., Nie, Z., Wen, J.-R., Zhang, B., and Ma, W.-Y. 2006. Simultaneous record detection and attribute labeling in Web data extraction. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 494--503. Google Scholar
Digital Library
Index Terms
Robust detection of semi-structured web records using a DOM structure-knowledge-driven model
Recommendations
Extraction of flat and nested data records from web pages
AusDM '06: Proceedings of the fifth Australasian conference on Data mining and analystics - Volume 61This paper deals with studies the problem of identification and extraction of flat and nested data records from a given web page. With the explosive growth of information sources available on the World Wide Web, it has become increasingly difficult to ...
A model-driven approach to semi-structured database design
Recently XML has become a standard for data representation and the preferred method of encoding structured data for exchange over the Internet. Moreover it is frequently used as a logical format to store structured and semi-structured data in databases. ...
Mining data records in Web pages
KDD '03: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data miningA large amount of information on the Web is contained in regularly structured objects, which we call data records. Such data records are important because they often present the essential information of their host pages, e.g., lists of products or ...






Comments