Textbook Content Extraction & LLM-based Question Answering System
Budget: ₹600 – ₹1,500 INR
I'm in need of an experienced developer to help with my project of extracting content from 3 digital textbooks which have at least 300 pages each. This extracted content should be used to create a vector database using MILVUS with RAPTOR indexing. The programming language required for this task is Python.
Task Description:
Textbook Selection and Content Extraction:
Select any three digital textbooks, each with more than 300 pages, for content extraction.
Extract the content from the selected textbooks thoroughly, ensuring that all relevant text is captured.
Data Chunking and RAPTOR Indexing:
Chunk the extracted content into short, contiguous texts of approximately 100 tokens each, preserving sentence boundaries.
The chunked texts should be embedded using SBERT (Sentence-BERT) to create vector representations.
Implement the RAPTOR indexing method, which involves the following steps:
Clustering the embedded chunks using Gaussian Mixture Models (GMMs) with soft clustering, allowing nodes to belong to multiple clusters.
Summarizing the clusters using an LLM (e.g., GPT-3.5-turbo) to create concise representations of the grouped texts.
Re-embedding the summarized texts and recursively applying the clustering and summarization process until a hierarchical tree structure is formed.
The resulting RAPTOR index should capture both high-level and low-level details of the textbooks, enabling efficient retrieval at different levels of granularity.
Store the RAPTOR index in a MILVUS vector database, along with relevant metadata such as the textbook title and page number.
Key requirements:
- Content extraction from 3 different digital textbooks, each with more than 300 pages.
- Development of a vector database using MILVUS with RAPTOR indexing.
- Creation of a question-answering system using an LLM (Language Model).
Retrieval Techniques:
Implement query expansion techniques to enhance the retrieval process. This may include techniques such as synonym expansion, stemming, or leveraging external knowledge bases to expand the query terms.
Hybrid retrieval methods could be employed, combining BM25 (Best Match 25) and BERT/bi-encoder based retrieval methods such as DPR (Dense Passage Retrieval) and SPIDER (Semantic Passage Retrieval).
Experiment with different retrieval strategies and evaluate their effectiveness in retrieving relevant content from the RAPTOR index.
The retrieved data should be re-ranked based on relevance and similarity to the query using appropriate ranking algorithms.
Question Answering:
The retrieved and re-ranked data should be passed to an LLM (Language Model) of your choice for question answering.
The LLM should be able to generate accurate and relevant answers based on the retrieved data from the RAPTOR index.
Resources:
RAPTOR Paper: RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
MILVUS Documentation: https://milvus.io/docs/
Relevant Python libraries: NLTK, Gensim, Transformers, PyPDF2 (for PDF extraction), Pyserini (for BM25), Sentence-Transformers (for bi-encoder retrieval), etc.
Task Description:
Textbook Selection and Content Extraction:
Select any three digital textbooks, each with more than 300 pages, for content extraction.
Extract the content from the selected textbooks thoroughly, ensuring that all relevant text is captured.
Data Chunking and RAPTOR Indexing:
Chunk the extracted content into short, contiguous texts of approximately 100 tokens each, preserving sentence boundaries.
The chunked texts should be embedded using SBERT (Sentence-BERT) to create vector representations.
Implement the RAPTOR indexing method, which involves the following steps:
Clustering the embedded chunks using Gaussian Mixture Models (GMMs) with soft clustering, allowing nodes to belong to multiple clusters.
Summarizing the clusters using an LLM (e.g., GPT-3.5-turbo) to create concise representations of the grouped texts.
Re-embedding the summarized texts and recursively applying the clustering and summarization process until a hierarchical tree structure is formed.
The resulting RAPTOR index should capture both high-level and low-level details of the textbooks, enabling efficient retrieval at different levels of granularity.
Store the RAPTOR index in a MILVUS vector database, along with relevant metadata such as the textbook title and page number.
Key requirements:
- Content extraction from 3 different digital textbooks, each with more than 300 pages.
- Development of a vector database using MILVUS with RAPTOR indexing.
- Creation of a question-answering system using an LLM (Language Model).
Retrieval Techniques:
Implement query expansion techniques to enhance the retrieval process. This may include techniques such as synonym expansion, stemming, or leveraging external knowledge bases to expand the query terms.
Hybrid retrieval methods could be employed, combining BM25 (Best Match 25) and BERT/bi-encoder based retrieval methods such as DPR (Dense Passage Retrieval) and SPIDER (Semantic Passage Retrieval).
Experiment with different retrieval strategies and evaluate their effectiveness in retrieving relevant content from the RAPTOR index.
The retrieved data should be re-ranked based on relevance and similarity to the query using appropriate ranking algorithms.
Question Answering:
The retrieved and re-ranked data should be passed to an LLM (Language Model) of your choice for question answering.
The LLM should be able to generate accurate and relevant answers based on the retrieved data from the RAPTOR index.
Resources:
RAPTOR Paper: RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
MILVUS Documentation: https://milvus.io/docs/
Relevant Python libraries: NLTK, Gensim, Transformers, PyPDF2 (for PDF extraction), Pyserini (for BM25), Sentence-Transformers (for bi-encoder retrieval), etc.