This paper provides an in-depth description of MapReduce algorithm and Nutch Distributed File System in Nutch web search engine. Nutch is an open-source Web search engine that can be used at global, local, and even personal scale. To engineer a search engine is a challenging task. Search engines index tens to hundreds of millions of web pages involving a comparable number of distinct terms. They answer tens of millions of queries every day. Despite the importance of large-scale search engines on the web, very little academic research has been done on them. Furthermore, due to rapid advance in technology and web proliferation, creating a web search engine today is very different from ten years ago.

1. White, Tom. 2006. “Introduction to Nutch, Part 1: Crawling”. Retrieved from http://today.java.net/pub/a/today/2006/01/10/introduction-to-nutch-1.html.
2. Smart, John Ferguson. 2006. “Integrate Advanced Search Functionalities Into Your Apps”. Retrieved from
3. Integrate advanced search functionalities into your apps at: http://www.javaworld.com/javaworld/jw-09-2006/jw-0925-lucene.html .
4. Open source at: http://en.wikipedia.org/wiki/Open_Source
5. Welcome to Apache Nutch at: http://lucene.apache.org/nutch/
6. Welcome to Apache Nutch at: http://www.nutch.org/
7. M. Cafarella and D. Cutting. Building Nutch: open source search. 2004.
8. MapReduce and Simplified Data Processing on large Clusters, journal of Jeffrey Dean and Sanjay Ghemawat, Google, Inc.

MapReduce Algorithm Nutch Distributed Nutch