Search Engine Development with Lucene API
Budget: $30 – $250 USD
In phase I, you will design and implement the indexing and searching components of the system using the open source IR tools from Apache and relying on a corpus for generating content. You will also evaluate this search engine for the quality of the results in terms of the following metrics: precision, recall, and ranking.
Scoring and Retrieval Model
Lucene combines Boolean model (BM) of Information Retrieval with Vector Space Model (VSM) of Information Retrieval - documents "approved" by BM are scored by VSM. Lucene scoring uses a combination of the Vector Space Model (VSM) of Information Retrieval and the Boolean model to determine how relevant a given Document is to a User's query. In general, the idea behind the VSM is the more times a query term appears in a document relative to the number of times the term appears in all the documents in the collection, the more relevant that document is to the query. It uses the Boolean model to first narrow down the documents that need to be scored based on the use of boolean logic in the Query specification. Lucene also adds some capabilities and refinements onto this model to support boolean and fuzzy searching, but it essentially remains a VSM based system at the heart. For some valuable references on VSM and IR in general refer to the Lucene Wiki IR references.
Project Corpus
I. Web Search
Corpus for this part of the project is the Cranfield document collection, a set of 1398 abstracts from aerodynamics journal articles. This historically significant dataset was first used in the field of IR for accurate measurement of retrieval performance. A description of the corpus follows:
* cran.all: The Cranfield collection corpus file with many fields. The title (T) and the content of the abstract (W) are the two fields that you may use in this project.
* query.text: A set of 225 queries that can be used to test the retrieval performance of your system.
* qrels.text: Query relevance judgements for each query. Each query is accompanied with a set of documents that are relevant in separate lines.
* README.txt: Fine grained description of each column in the dataset files
I – Simple Search Engine:
Lucene is a high-performance Java based search engine library. You will use the Lucene API to index the documents in Cranfield dataset to answer some standard queries. Here we describe possible modulesyou need to complete in Phase I.
Indexer
The indexer will process all the documents in the corpus to create a searchable index. Note that documents in the Cranfield collection are contained in a single file and should be separated into multiple files before indexing. Internally, Lucene stores each document as a set of field-value pairs. Indexer should convert each document file to a Lucene document with fields such as title and abstract. In general, the index can be in-memory or on disk. For this project, you are required to write the index to disk which will be utilized by Index Searcher later. You can try various analyzers to pre-process a document, to tokenize, to remove stop words, and tostem, before indexing. After indexing, you will query the index to retrieve documents based on the querystring supplied by a user. Query string has to undergo a similar treatment as document during indexing (so remember to use the same analyzers for both indexing documents and processing queries).
Index Searcher
Index Searcher takes a user query as input and applies the same set of preprocessing operations appliedto the documents during indexing. You can use Query Parser for pre-processing and Index Searcher for finding hits for a particular query. The hits are returned along with the details such as number of hits and details of each hit. Hits should be displayed to the user in a way that allows the user to access theresult set documents. The details of “linking mechanism” and the teaser text to be displayed is left to your imagination.
Enhanced Search Results
* You can selectively boost the importance of fields in a document using Field.setBoost. E.g., youcan weigh title more depending on your knowledge of the dataset.
* You can explore other enhancements or experiment with different analyzers, stop word lists, etc.
Results Evaluation and Ranking
Discuss the evaluation metrics to be used to quantify the quality of the search results, and determine the quality of results (e.g., precision, recall, ranking) for various user queries (at least 15 queries) from the Cranfield collection, and report the evaluation metrics for both the baseline search engine and the enhanced search engine.
Milestones for Phase I:
* Prepare the corpus according to the input requirements of the Lucene API.
* Implement Indexer to index all the documents in the corpus and store the index on disk (of thelocal machine).
* Implement Query Parser to process the incoming queries.
* Implement Index Searcher to satisfy the user query and retrieve the relevant documents. This is the baseline search engine for our purposes.
* Enhance the search results and the search engine performance by modifying Lucene API parameters. This is the enhanced search engine for our purposes.
* You can selectively boost the importance of fields in a document using Field.setBoost. E.g., youcan weigh title more depending on your knowledge of the dataset.
* You can explore other enhancements or experiment with different analyzers, stop word lists, etc.
* Report the evaluation metrics for both the baseline search engine and the enhanced search engine
Scoring and Retrieval Model
Lucene combines Boolean model (BM) of Information Retrieval with Vector Space Model (VSM) of Information Retrieval - documents "approved" by BM are scored by VSM. Lucene scoring uses a combination of the Vector Space Model (VSM) of Information Retrieval and the Boolean model to determine how relevant a given Document is to a User's query. In general, the idea behind the VSM is the more times a query term appears in a document relative to the number of times the term appears in all the documents in the collection, the more relevant that document is to the query. It uses the Boolean model to first narrow down the documents that need to be scored based on the use of boolean logic in the Query specification. Lucene also adds some capabilities and refinements onto this model to support boolean and fuzzy searching, but it essentially remains a VSM based system at the heart. For some valuable references on VSM and IR in general refer to the Lucene Wiki IR references.
Project Corpus
I. Web Search
Corpus for this part of the project is the Cranfield document collection, a set of 1398 abstracts from aerodynamics journal articles. This historically significant dataset was first used in the field of IR for accurate measurement of retrieval performance. A description of the corpus follows:
* cran.all: The Cranfield collection corpus file with many fields. The title (T) and the content of the abstract (W) are the two fields that you may use in this project.
* query.text: A set of 225 queries that can be used to test the retrieval performance of your system.
* qrels.text: Query relevance judgements for each query. Each query is accompanied with a set of documents that are relevant in separate lines.
* README.txt: Fine grained description of each column in the dataset files
I – Simple Search Engine:
Lucene is a high-performance Java based search engine library. You will use the Lucene API to index the documents in Cranfield dataset to answer some standard queries. Here we describe possible modulesyou need to complete in Phase I.
Indexer
The indexer will process all the documents in the corpus to create a searchable index. Note that documents in the Cranfield collection are contained in a single file and should be separated into multiple files before indexing. Internally, Lucene stores each document as a set of field-value pairs. Indexer should convert each document file to a Lucene document with fields such as title and abstract. In general, the index can be in-memory or on disk. For this project, you are required to write the index to disk which will be utilized by Index Searcher later. You can try various analyzers to pre-process a document, to tokenize, to remove stop words, and tostem, before indexing. After indexing, you will query the index to retrieve documents based on the querystring supplied by a user. Query string has to undergo a similar treatment as document during indexing (so remember to use the same analyzers for both indexing documents and processing queries).
Index Searcher
Index Searcher takes a user query as input and applies the same set of preprocessing operations appliedto the documents during indexing. You can use Query Parser for pre-processing and Index Searcher for finding hits for a particular query. The hits are returned along with the details such as number of hits and details of each hit. Hits should be displayed to the user in a way that allows the user to access theresult set documents. The details of “linking mechanism” and the teaser text to be displayed is left to your imagination.
Enhanced Search Results
* You can selectively boost the importance of fields in a document using Field.setBoost. E.g., youcan weigh title more depending on your knowledge of the dataset.
* You can explore other enhancements or experiment with different analyzers, stop word lists, etc.
Results Evaluation and Ranking
Discuss the evaluation metrics to be used to quantify the quality of the search results, and determine the quality of results (e.g., precision, recall, ranking) for various user queries (at least 15 queries) from the Cranfield collection, and report the evaluation metrics for both the baseline search engine and the enhanced search engine.
Milestones for Phase I:
* Prepare the corpus according to the input requirements of the Lucene API.
* Implement Indexer to index all the documents in the corpus and store the index on disk (of thelocal machine).
* Implement Query Parser to process the incoming queries.
* Implement Index Searcher to satisfy the user query and retrieve the relevant documents. This is the baseline search engine for our purposes.
* Enhance the search results and the search engine performance by modifying Lucene API parameters. This is the enhanced search engine for our purposes.
* You can selectively boost the importance of fields in a document using Field.setBoost. E.g., youcan weigh title more depending on your knowledge of the dataset.
* You can explore other enhancements or experiment with different analyzers, stop word lists, etc.
* Report the evaluation metrics for both the baseline search engine and the enhanced search engine