Abstract
We demonstrate a powerful and easy-to-use tool called Dedoop (<u>De</u>duplication with Ha<u>doop</u>) for MapReduce-based entity resolution (ER) of large datasets. Dedoop supports a browser-based specification of complex ER workflows including blocking and matching steps as well as the optional use of machine learning for the automatic generation of match classifiers. Specified workflows are automatically translated into MapReduce jobs for parallel execution on different Hadoop clusters. To achieve high performance Dedoop supports several advanced load balancing strategies.
References
- Dedoop. http://dbs.uni-leipzig.de/dedoop.Google Scholar
- Baxter, Christen, and Churches. A Comparison of Fast Blocking Methods for Record Linkage. In Workshop Data Cleaning, Record Linkage, and Object Consolidation, pages 25--27, 2003.Google Scholar
- Kolb, Thor, and Rahm. Load Balancing for MapReduce-based Entity Resolution. In ICDE, pages 618--629, 2012. Google Scholar
- Kolb, Thor, and Rahm. Multi-pass Sorted Neighborhood Blocking with MapReduce. CSRD, 27(1):45--63, 2012. Google Scholar
- Köpcke and Rahm. Frameworks for Entity Matching: A Comparison. Data Knowl. Eng., 69(2):197--210, 2010. Google Scholar
- Köpcke, Thor, and Rahm. Evaluation of Entity Resolution Approaches on real-world Match Problems. PVLDB, 3(1):484--493, 2010. Google Scholar
Index Terms
(auto-classified)Dedoop: efficient deduplication with Hadoop





Comments