检索结果-内蒙古大学图书馆

duplicate detection in adverse drug reaction surveillance

DATA MINING AND KNOWLEDGE DISCOVERY 2007年第3期14卷 305-328页

作者： Noren, G. Niklas Orre, Roland Bate, Andrew Edwards, I. Ralph WHO Collaborating Ctr Int Drug Monitoring Uppsala Sweden Stockholm Univ S-10691 Stockholm Sweden Neurolog Sweden AB Stockholm Sweden

The WHO Collaborating Centre for International Drug Monitoring in Uppsala, Sweden, maintains and analyses the world's largest database of reports on suspected adverse drug reaction (ADR) incidents that occur after drugs are on the market. The presence of duplicate case reports is an important data quality problem and their detection remains a formidable challenge, especially in the WHO drug safety database where reports are anonymised before submission. In this paper, we propose a duplicate detection method based on the hit-miss model for statistical record linkage described by Copas and Hilton, which handles the limited amount of training data well and is well suited for the available data (categorical and numerical rather than free text). We propose two extensions of the standard hit-miss model: a hit-miss mixture model for errors in numerical record fields and a new method to handle correlated record fields, and we demonstrate the effectiveness both at identifying the most likely duplicate for a given case report (94.7% accuracy) and at discriminating true duplicates from random matches (63% recall with 71% precision). The proposed method allows for more efficient data cleaning in post-marketing drug safety data sets, and perhaps other knowledge discovery applications as well.

关键词： data cleaning duplicate detection hit-miss model

来源：评论

学校读者我要写书评

暂无评论

duplicate detection for Identifying Social Spam in Microblogs

Duplicate Detection for Identifying Social Spam in Microblog...

引用

IEEE International Congress on Big Data

作者： Zhang, Qunyan Ma, Haixin Qian, Weining Zhou, Aoying E China Normal Univ Ctr Cloud Comp & Big Data Inst Software Engn Shanghai 200062 Peoples R China

ISBN: (纸本)9780769550060

As an important kind of social media, microblog has become an important source of opinion mining and collective behavior study. However, social spams may affect the analytical results greatly. This paper focuses on the problem of identifying potential social spammers who copy pieces of information from others. An improved locality-sensitive hashing based method is used for detecting duplicated tweets. Intensive empirical study over a real-life microblog dataset crawled from Sina Weibo, one of the most popular microblogging services, is conducted. The characteristics of potential spammers and their behaviors are analyzed.

关键词： duplicate detection social spam microblog locality-sensitive hash MapReduce

来源：评论

学校读者我要写书评

暂无评论

Progressive duplicate detection

引用

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 2015年第5期27卷 1316-1329页

作者： Papenbrock, Thorsten Heise, Arvid Naumann, Felix Hasso Plattner Inst Dept Informat Syst D-14482 Potsdam Germany

duplicate detection is the process of identifying multiple representations of same real world entities. Today, duplicate detection methods need to process ever larger datasets in ever shorter time: maintaining the quality of a dataset becomes increasingly difficult. We present two novel, progressive duplicate detection algorithms that significantly increase the efficiency of finding duplicates if the execution time is limited: They maximize the gain of the overall process within the time available by reporting most results much earlier than traditional approaches. Comprehensive experiments show that our progressive algorithms can double the efficiency over time of traditional duplicate detection and significantly improve upon related work.

关键词： duplicate detection entity resolution pay-as-you-go progressiveness data cleaning

来源：评论

学校读者我要写书评

暂无评论

duplicate detection in web shops using LSH to reduce the number of computations 16

Duplicate detection in web shops using LSH to reduce the num...

引用

31st Annual ACM Symposium on Applied Computing, SAC 2016

作者： Van Dam, Iris Nijenhuis, Nikki Van Ginkel, Gerhard Vandic, Damir Kuipers, Wim Frasincar, Flavius Erasmus University Rotterdam P.O. Box 1738 DR Rotterdam3000 Netherlands

ISBN: (纸本)9781450337397

The amount of online shops is growing daily and many Web shops focus on the same product types, like consumer electronics. Since Web shops use different product representations, it is hard to compare products among different Web shops. duplicate detection methods aim to solve this problem by identifying the same products in different Web shops. In this paper, we focus on reducing the computation time of a state-of-the-art duplicate detection algorithm. First, we construct uniform vector representations for the products. We use these vectors as input for a Locality Sensitive Hashing (LSH) algorithm, which pre-selects potential duplicates. Finally, duplicate products are found by applying the Multi-component Similarity Method (MSM). Compared to original MSM, the number of needed computations can be reduced by 95% with only a minor decrease by 9% in the F1-measure.

关键词： duplicate detection locality-sensitive hashing web shop products

来源：评论

学校读者我要写书评

暂无评论

Scalable Iterative Graph duplicate detection

引用

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 2012年第11期24卷 2094-2108页

作者： Herschel, Melanie Naumann, Felix Szott, Sascha Taubert, Maik Univ Tubingen Wilhelm Schickard Inst Informat D-72076 Tubingen Germany Hasso Plattner Inst Softwaresyst Tech D-14482 Potsdam Germany Konrad Zuse Zentrum Informat Tech Berlin D-14195 Berlin Germany Biotron SE & Co KG D-12359 Berlin Germany

duplicate detection determines different representations of real-world objects in a database. Recent research has considered the use of relationships among object representations to improve duplicate detection. In the general case where relationships form a graph, research has mainly focused on duplicate detection quality/effectiveness. Scalability has been neglected so far, even though it is crucial for large real-world duplicate detection tasks. We scale-up duplicate detection in graph data (DDG) to large amounts of data and pairwise comparisons, using the support of a relational database management system. To this end, we first present a framework that generalizes the DDG process. We then present algorithms to scale DDG in space (amount of data processed with bounded main memory) and in time. Finally, we extend our framework to allow batched and parallel DDG, thus further improving efficiency. Experiments on data of up to two orders of magnitude larger than data considered so far in DDG show that our methods achieve the goal of scaling DDG to large volumes of data.

关键词： duplicate detection data cleaning data integration record linkage entity resolution scalability parallelization

来源：评论

学校读者我要写书评

暂无评论

Semi-Automatic Classification and duplicate detection From Human Loss News Corpus

引用

IEEE ACCESS 2020年 8卷 97737-97747页

作者： Abid, Adnan Ali, Waqas Farooq, Muhammad Shoaib Farooq, Uzma Khan, Nabeel Sabir Abid, Kamran Univ Management & Technol Dept Comp Sci Lahore 54770 Pakistan Univ Management & Technol Dept Software Engn Lahore 54770 Pakistan Univ Punjab Dept Elect Engn Lahore 54590 Pakistan

Automatic news repository collection systems involve a news crawler that extracts news from different news portals, subsequently, these news need to be processed to figure out the category of a news article e.g. sports, politics, showbiz etc. In this process there are two main challenges first one is to place a news article under the right category of news, while the second one is to detect a duplicate news, i.e. when the news are being extracted from multiple sources, it is highly probable to get the same news from many different portals, resulting into duplicate news;failing to which may result into inconsistent statistics obtained after pre-processing the news text. This problem becomes more pertinent when we deal with human loss news involving crime, accident etc. related news articles. As the system may count the same news many times resulting into misleading statistics. In order to address these problems, this research presents the following contributions. Firstly, a news corpus comprising of human loss news of different categories has been developed by gathering data from different sources of well-known and authentic news websites. The corpus also includes a number of duplicate news. Secondly, a comparison of different classification approaches has been conducted to empirically find out the best suitable text classifier for the categorization of different sub-categories of human loss news. Lastly, methods have been proposed and compared to detect duplicate news from the corpus by involving different pre-processing techniques and widely used similarity measures, cosine similarity, and Jaccard & x2019;s coefficient. The results show that conventional text classifiers are still relevant and perform well in text classification tasks as MNB has given 89.5 & x0025;accurate results. While, Jaccard coefficient exhibits much better results than Cosine similarity for duplicate news detection with different pre-processing variations with an average accuracy of 83.16 & x

关键词： Text categorization Security Classification algorithms Indexes Urban areas Semantics Support vector machines Information retrieval news classification news similarity duplicate detection security profiling

来源：评论

学校读者我要写书评

暂无评论

DENATURE: duplicate detection and type identification in open source bug repositories

引用

INTERNATIONAL JOURNAL OF SYSTEM ASSURANCE ENGINEERING AND MANAGEMENT 2023年第SUPPL 1期14卷 S275-S292页

作者： Chauhan, Ruby Sharma, Shakshi Goyal, Anjali NorthCap Univ Sect 23 A Gurugram 122017 Haryana India Univ Tartu Tartu Estonia Sharda Univ Sch Engn & Technol Dept Comp Sci & Engn Greater Noida India

Software projects reckon on the bug tracking systems to guide software maintenance activities. The critical information about the nature of the crash is carried by the bug reports which are submitted to bug repositories. This information is in free form text format and is submitted by users or developers. A large amount of bug reports gets collected in bug repositories. Out of these submitted bugs, many reports are mere identical of the already existing bugs. Furthermore, not all non-duplicate bugs are reproducible in nature. This paper introduces DENATURE, a two step framework for detecting duplication and identifying bug type. The proposed framework will help to minimize time and developer's effort utilized in resolution of bug reports which will further improvise overall software quality. Information retrieval techniques are used for finding duplicate bugs and machine learning classification techniques are used for identifying the type of bug report. Through experiments, we found that the proposed framework obtained prediction accuracy up to 88.81%.

关键词： Bug tracking system Bug reports duplicate detection Bug type identification Similarity measures Classification Information retrieval techniques

来源：评论

学校读者我要写书评

暂无评论

Efficient and Effective duplicate detection in Hierarchical Data

引用

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 2013年第5期25卷 1028-1041页

作者： Leitao, Luis Calado, Pavel Herschel, Melanie Inst Super Tecn P-2744016 Porto Salvo Portugal Univ Paris 11 LRI F-91409 Orsay France INRIA Saclay Saclay France

Although there is a long line of work on identifying duplicates in relational data, only a few solutions focus on duplicate detection in more complex hierarchical structures, like XML data. In this paper, we present a novel method for XML duplicate detection, called XMLDup. XMLDup uses a Bayesian network to determine the probability of two XML elements being duplicates, considering not only the information within the elements, but also the way that information is structured. In addition, to improve the efficiency of the network evaluation, a novel pruning strategy, capable of significant gains over the unoptimized version of the algorithm, is presented. Through experiments, we show that our algorithm is able to achieve high precision and recall scores in several data sets. XMLDup is also able to outperform another state-of-the-art duplicate detection solution, both in terms of efficiency and of effectiveness.

关键词： duplicate detection record linkage entity resolution XML Bayesian networks data cleaning optimization

来源：评论

学校读者我要写书评

暂无评论

Data Preparation for duplicate detection

引用

ACM JOURNAL OF DATA AND INFORMATION QUALITY 2020年第3期12卷 1–24页

作者： Koumarelas, Ioannis Jiang, Lan Naumann, Felix Univ Potsdam Hasso Plattner Inst Prof Dr Helmert Str 2-3 D-14482 Potsdam Germany

Data errors represent a major issue in most application workflows. Before any important task can take place, a certain data quality has to be guaranteed by eliminating a number of different errors that may appear in data. Typically, most of these errors are fixed with data preparation methods, such as whitespace removal. However, the particular error of duplicate records, where multiple records refer to the same entity, is usually eliminated independently with specialized techniques. Our work is the first to bring these two areas together by applying data preparation operations under a systematic approach prior to performing duplicate detection. Our process workflow can be summarized as follows: It begins with the user providing as input a sample of the gold standard, the actual dataset, and optionally some constraints to domain-specific data preparations, such as address normalization. The preparation selection operates in two consecutive phases. First, to vastly reduce the search space of ineffective data preparations, decisions are made based on the improvement or worsening of pair similarities. Second, using the remaining data preparations an iterative leave-one-out classification process removes preparations one by one and determines the redundant preparations based on the achieved area under the precision-recall curve (AUC-PR). Using this workflow, we manage to improve the results of duplicate detection up to 19% in AUC-PR.

关键词： Data preparation data wrangling record linkage duplicate detection similarity measures

来源：评论

学校读者我要写书评

暂无评论

Identification of FRBR works within bibliographic databases: An experiment with UNIMARC and duplicate detection techniques

引用

10th International Conference on Asian Digital Libraries

作者： Freire, Nuno Borbinha, Jose Calado, Pavel INESC ID P-1000029 Lisbon Portugal

ISBN: (纸本)9783540770930

Many experiments and studies have been conducted on the application of FRBR as an implementation model for bibliographic databases, in order to improve the services of resource discovery and transmit better perception of the information spaces represented in catalogues. One of these applications is the attempt to identify the FRBR work instances shared by several bibliographic records. In our work we evaluate the applicability to this problem of techniques based on string similarity, used in duplicate detection procedures mainly by the database research community. We describe the particularities of the application of these techniques to bibliographic data, and empirically compare the results obtained with these techniques to those obtained by current techniques, which are based on exact matching. Experiments performed on the Portuguese national union catalogue show a significant improvement over currently used approaches.

关键词： Functional Requirements for Bibliographic Records FRBR bibliographic databases string similarity duplicate detection

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：