Vector DB Integration for AI
Budget: €12 – €18 EUR
I have a collection of plain-text company documents—well under 1 GB in total—that need to be transformed into a searchable vector store so our internal GPT-style chatbot can retrieve answers directly from them. You may work with Pinecone, Weaviate, or propose an equivalent service as long as it supports state-of-the-art embeddings and low-latency similarity search.
The job breaks down into three clear stages:
1. Parsing & Chunking
• Read the raw .txt files from a secure S3 bucket (or a local hand-off, if you prefer) and split them into sensible, context-preserving chunks.
2. Embedding & Indexing
• Generate embeddings with OpenAI, Cohere, or another proven model.
• Insert those vectors, together with the original text and metadata, into the chosen database.
3. Validation & Handover
• Expose a simple API endpoint or notebook showing that a sample query returns the relevant passages.
• Deliver a short README detailing environment variables, setup steps, and how my chatbot can hit the vector store.
Acceptance criteria
• End-to-end pipeline runs in under 15 minutes on the full dataset.
• A demo query reliably surfaces the correct source paragraph.
• All code is reproducible in a clean Docker container or requirements.txt environment.
Security is essential; the documents are company-confidential, so the workflow must keep data on our cloud account or an agreed private workspace.
If this sounds straightforward, tell me which database you prefer, why, and roughly how long the initial ingestion will take.
The job breaks down into three clear stages:
1. Parsing & Chunking
• Read the raw .txt files from a secure S3 bucket (or a local hand-off, if you prefer) and split them into sensible, context-preserving chunks.
2. Embedding & Indexing
• Generate embeddings with OpenAI, Cohere, or another proven model.
• Insert those vectors, together with the original text and metadata, into the chosen database.
3. Validation & Handover
• Expose a simple API endpoint or notebook showing that a sample query returns the relevant passages.
• Deliver a short README detailing environment variables, setup steps, and how my chatbot can hit the vector store.
Acceptance criteria
• End-to-end pipeline runs in under 15 minutes on the full dataset.
• A demo query reliably surfaces the correct source paragraph.
• All code is reproducible in a clean Docker container or requirements.txt environment.
Security is essential; the documents are company-confidential, so the workflow must keep data on our cloud account or an agreed private workspace.
If this sounds straightforward, tell me which database you prefer, why, and roughly how long the initial ingestion will take.