Elasticsearch program in python for movies.
Budget: $30 – $250 CAD
The dataset contains descriptions of 34,886 movies from around the world. The plot summary descriptions are scraped from Wikipedia. This freely available dataset is provided to the global research community to apply recent advances in information retrieval and other AI techniques to generate models that can return a movie title based on an input plot description or return movie titles with plots similar to the user query.
• Indexing: The first step for you will be to obtain the dataset. Once you have done so choose a sample of 1000 articles as your corpus (the simplest thing is to use the first 1000 documents). This will need to be imported to Elasticsearch later (after you defined your processing pipeline).
• Tokenization and Case folding: The next step should be to transform the input text into a normal form. For this task you are required to use Elasticsearch’s build-in analyzers or other libraries (as learned in Lab 2) to tokenize the document and perform case folding to the tokens.
• Selecting Keywords: One aim of your system is to identify the words and phrases in the text that are most useful for indexing purposes. For this task you are required to do include stopword removal and (n-gram extraction or named entity recognition). As well as apply tf.idf as part of your selection and weighting step. (Hint: the stopword removal, n-gram extraction can be done with Elasticsearch’s build-in tokenizer and tf.idf scores can also be configured using Elasticsearch similarity module.)
• Stemming or Morphological Analysis: Writing word stems to the database rather than words allows to treat various inflected forms of a word in the same way, e.g. bus and busses refer to exactly the same thing even though they are different words.
• Searching: Once you have indexed the collection you want to be able to search it. You can do that on the command line (like in Lab 1), but it would be easier to do it Kibana’s dev tool. The task is to create 3 textural queries that the user might come up and write the corresponding Elasticsearch queries.
• Working with Elasticsearch API: Finally the last part is to see if you could make everything work with the Elasticsearch API.
• Indexing: The first step for you will be to obtain the dataset. Once you have done so choose a sample of 1000 articles as your corpus (the simplest thing is to use the first 1000 documents). This will need to be imported to Elasticsearch later (after you defined your processing pipeline).
• Tokenization and Case folding: The next step should be to transform the input text into a normal form. For this task you are required to use Elasticsearch’s build-in analyzers or other libraries (as learned in Lab 2) to tokenize the document and perform case folding to the tokens.
• Selecting Keywords: One aim of your system is to identify the words and phrases in the text that are most useful for indexing purposes. For this task you are required to do include stopword removal and (n-gram extraction or named entity recognition). As well as apply tf.idf as part of your selection and weighting step. (Hint: the stopword removal, n-gram extraction can be done with Elasticsearch’s build-in tokenizer and tf.idf scores can also be configured using Elasticsearch similarity module.)
• Stemming or Morphological Analysis: Writing word stems to the database rather than words allows to treat various inflected forms of a word in the same way, e.g. bus and busses refer to exactly the same thing even though they are different words.
• Searching: Once you have indexed the collection you want to be able to search it. You can do that on the command line (like in Lab 1), but it would be easier to do it Kibana’s dev tool. The task is to create 3 textural queries that the user might come up and write the corresponding Elasticsearch queries.
• Working with Elasticsearch API: Finally the last part is to see if you could make everything work with the Elasticsearch API.