Let’s quickly test this on 12TB of data

MapReduce for Information Retrieval Evaluation

by Djoerd Hiemstra and Claudia Hauff

We propose to use MapReduce to quickly test new retrieval approaches on a cluster of machines by sequentially scanning all documents. We present a small case study in which we use a cluster of 15 low cost machines to search a web crawl of 0.5 billion pages showing that sequential scanning is a viable approach to running large-scale information retrieval experiments with little effort. The code is available to other researchers at: http://mirex.sourceforge.net.

The paper will be presented at the CLEF 2010 Conference on Multilingual and Multimodal Information Access Evaluation on 20-23 September 2010 in Padua, Italy