RAG Pipeline Enhancement for Q&A Forum
Budget: $15 – $25 USD
Overview
We are seeking an experienced RAG (Retrieval-Augmented Generation) Pipeline Specialist to expand and integrate our existing knowledge base for a community Q&A forum. We currently use a vector database (leveraging OpenAI embeddings) to provide answers based on a large set of PDF documents. The goal is to seamlessly integrate new web-scraped content and additional PDF documents into this unified knowledge base.
This is a critical project requiring expertise in Python, web scraping (Scrapy), text processing, vector databases, and the OpenAI API.
Project Goals
The primary objective is to execute a unified data ingestion process to create a single, comprehensive, and searchable index for our LLM-powered Q&A forum.
1. Web Data Integration (USPS Manual)
Source: A specific, deep, multi-page website (e.g., https://about.usps.com/manuals/elm/html/welcome.htm).
Task: Develop and run a Scrapy Spider to recursively crawl and extract text content and metadata (URL, Title, etc.) from the entire manual structure.
2. New PDF Integration
Source: A directory of new PDF files.
Task: Process the new PDF files, extracting text reliably, and maintaining proper document and page metadata.
3. Unified Embedding and Indexing
Task: Run the extracted data from both sources through the exact same chunking and embedding process used for our existing data to ensure semantic consistency.
Final Action: Upsert all new vectors and metadata into our existing single vector database index (mention your specific vector database here, e.g., Pinecone, Chroma, etc.).
Required Expertise and Deliverables
Technical Skills
Expert Python programming, including familiarity with common RAG frameworks (LangChain, LlamaIndex, etc.).
Expert Scrapy or advanced web scraping techniques, including handling recursion and politeness settings (robots.txt, DOWNLOAD_DELAY).
Deep understanding of RAG principles: Text chunking strategy (e.g., Recursive Character Splitting), consistency in chunk size/overlap, and metadata usage.
Proficiency with OpenAI API: Specifically the Embeddings API (using the same model we currently use, e.g., text-embedding-3-small).
Experience with Vector Databases: (Specify your database here: [E.g., Pinecone, Chroma, Weaviate]).
Deliverables
Scrapy Spider Code: A fully functional, production-ready Python file (e.g., usps_manual_crawler.py) used to scrape the target website.
Data Ingestion Script: A documented Python script that handles the end-to-end process: PDF loading, text extraction, consistent chunking, embedding generation, and unified upsert into our Vector Database.
Deployment Instructions: Clear instructions on how to run the pipeline, ideally within a provided Python virtual environment.
Verification Report: Confirmation that the new data has been successfully indexed and is retrievable alongside the existing knowledge base.
We are seeking an experienced RAG (Retrieval-Augmented Generation) Pipeline Specialist to expand and integrate our existing knowledge base for a community Q&A forum. We currently use a vector database (leveraging OpenAI embeddings) to provide answers based on a large set of PDF documents. The goal is to seamlessly integrate new web-scraped content and additional PDF documents into this unified knowledge base.
This is a critical project requiring expertise in Python, web scraping (Scrapy), text processing, vector databases, and the OpenAI API.
Project Goals
The primary objective is to execute a unified data ingestion process to create a single, comprehensive, and searchable index for our LLM-powered Q&A forum.
1. Web Data Integration (USPS Manual)
Source: A specific, deep, multi-page website (e.g., https://about.usps.com/manuals/elm/html/welcome.htm).
Task: Develop and run a Scrapy Spider to recursively crawl and extract text content and metadata (URL, Title, etc.) from the entire manual structure.
2. New PDF Integration
Source: A directory of new PDF files.
Task: Process the new PDF files, extracting text reliably, and maintaining proper document and page metadata.
3. Unified Embedding and Indexing
Task: Run the extracted data from both sources through the exact same chunking and embedding process used for our existing data to ensure semantic consistency.
Final Action: Upsert all new vectors and metadata into our existing single vector database index (mention your specific vector database here, e.g., Pinecone, Chroma, etc.).
Required Expertise and Deliverables
Technical Skills
Expert Python programming, including familiarity with common RAG frameworks (LangChain, LlamaIndex, etc.).
Expert Scrapy or advanced web scraping techniques, including handling recursion and politeness settings (robots.txt, DOWNLOAD_DELAY).
Deep understanding of RAG principles: Text chunking strategy (e.g., Recursive Character Splitting), consistency in chunk size/overlap, and metadata usage.
Proficiency with OpenAI API: Specifically the Embeddings API (using the same model we currently use, e.g., text-embedding-3-small).
Experience with Vector Databases: (Specify your database here: [E.g., Pinecone, Chroma, Weaviate]).
Deliverables
Scrapy Spider Code: A fully functional, production-ready Python file (e.g., usps_manual_crawler.py) used to scrape the target website.
Data Ingestion Script: A documented Python script that handles the end-to-end process: PDF loading, text extraction, consistent chunking, embedding generation, and unified upsert into our Vector Database.
Deployment Instructions: Clear instructions on how to run the pipeline, ideally within a provided Python virtual environment.
Verification Report: Confirmation that the new data has been successfully indexed and is retrievable alongside the existing knowledge base.