Web-Scraping Info Chatbot
Budget: $10 – $30 USD
I’m looking to have an AI chatbot built whose sole purpose is to answer user questions by pulling facts from publicly available websites in real time.
Core requirements
• Develop the chatbot engine—natural-language understanding, dialogue management, and response generation.
• Implement a crawler or API-based scraper that fetches content from selected public websites, cleans it, and stores it in a searchable format (vector store, database, or other approach you recommend).
• Design an information-retrieval pipeline that ranks and cites the most relevant passages so answers can be traced back to their sources.
• Provide a simple web interface or API endpoint so the bot can be embedded later in my site or app.
Key considerations
– Accuracy is critical; I want the bot to return verifiable answers, not guesses.
– Latency should stay comfortably below a few seconds per query.
– The solution must respect robots.txt and any site usage policies.
– Preferred tech stack is Python with libraries such as LangChain, FastAPI, or similar, but I’m open to what you do best.
Deliverables
1. Source code with clear docstrings and in-line comments.
2. Setup guide and requirements.txt (or equivalent) so I can run it locally.
3. Short video or written walkthrough demonstrating the chatbot answering sample questions.
4. Optional: suggestions for scaling the scraper and adding more sites in the future.
If you have prior experience building retrieval-augmented chatbots or working with web scraping/NLP pipelines, I’d love to see examples. Let’s create a helpful, information-rich assistant together.
Core requirements
• Develop the chatbot engine—natural-language understanding, dialogue management, and response generation.
• Implement a crawler or API-based scraper that fetches content from selected public websites, cleans it, and stores it in a searchable format (vector store, database, or other approach you recommend).
• Design an information-retrieval pipeline that ranks and cites the most relevant passages so answers can be traced back to their sources.
• Provide a simple web interface or API endpoint so the bot can be embedded later in my site or app.
Key considerations
– Accuracy is critical; I want the bot to return verifiable answers, not guesses.
– Latency should stay comfortably below a few seconds per query.
– The solution must respect robots.txt and any site usage policies.
– Preferred tech stack is Python with libraries such as LangChain, FastAPI, or similar, but I’m open to what you do best.
Deliverables
1. Source code with clear docstrings and in-line comments.
2. Setup guide and requirements.txt (or equivalent) so I can run it locally.
3. Short video or written walkthrough demonstrating the chatbot answering sample questions.
4. Optional: suggestions for scaling the scraper and adding more sites in the future.
If you have prior experience building retrieval-augmented chatbots or working with web scraping/NLP pipelines, I’d love to see examples. Let’s create a helpful, information-rich assistant together.