Simulating Information Retrieval Test Collections

丛 书 名：Synthesis lectures on information concepts, retrieval, and services,

作     者：David Hawking Bodo Billerbeck Paul Thomas Nick Craswell 

I S B N：(纸本) 9781681739571;9781681739595 

出 版 社：Morgan & Claypool 

出 版 年：2020年

主 题 词：bisac category: science \/ history \/ history \/ asia \/ china \/ science \/ physics \/ general 

学科分类：1205[管理学-图书情报与档案管理] 12[管理学] 120501[管理学-图书馆学] 081203[工学-计算机应用技术] 08[工学] 0835[工学-软件工程] 120502[管理学-情报学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

摘      要：Simulated test collections may find application in situations where real datasets cannot easily be accessed due to confidentiality concerns or practical inconvenience. They can potentially support Information Retrieval (IR) experimentation, tuning, validation, performance prediction, and hardware sizing. Naturally, the accuracy and usefulness of results obtained from a simulation depend upon the fidelity and generality of the models which underpin it. The fidelity of emulation of a real corpus is likely to be limited by the requirement that confidential information in the real corpus should not be able to be extracted from the emulated version. We present a range of methods exploring trade-offs between emulation fidelity and degree of preservation of privacy. We present three different simple types of text generator which work at a micro level: Markov models, neural net models, and substitution ciphers. We also describe macro level methods where we can engineer macro properties of a corpus, giving a range of models for each of the salient properties: document length distribution, word frequency distribution (for independent and non-independent cases), word length and textual representation, and corpus growth. We present results of emulating existing corpora and for scaling up corpora by two orders of magnitude. We show that simulated collections generated with relatively simple methods are suitable for some purposes and can be generated very quickly. Indeed it may sometimes be feasible to embed a simple lightweight corpus generator into an indexer for the purpose of efficiency studies. Naturally, a corpus of artificial text cannot support IR experimentation in the absence of a set of compatible queries. We discuss and experiment with published methods for query generation and query log emulation. We present a proof-of-the-pudding study in which we observe the predictive accuracy of efficiency and effectiveness results obtained on emulated versions of TREC corpora. T