Build a Website Like Firecrawl.dev with Crawling, AI Summary, API & Dashboard
Budget: ₹600 – ₹1,500 INR
I am looking for a full-stack developer (or a team) to build an exact replica of https://firecrawl.dev with all features and functionality. The final website should include:
⸻
Key Features Required:
1. Website Crawler
• Crawl any given URL
• Capture text, meta info, links, HTML
• Store page data in structured format
2. Web Scraper
• Scrape visible content + hidden data (e.g. meta tags, OpenGraph, schema)
• Extract data in JSON format
3. AI Summarizer
• Use GPT or any LLM to summarize page content
• Key Points, SEO tags, summaries, etc.
• Ability to customize prompt for LLM
4. API Access
• Provide REST API to:
• Submit crawl requests
• Fetch structured data
• Get AI summaries
• API key and user auth support
5. Dashboard / UI
• Clean frontend UI (like Firecrawl) using React/Next.js
• Input URLs and get structured response
• Show crawl history, results, logs
• Option to export data
6. Auth and Credits System
• Login/Register via email or OAuth (Google, GitHub)
• Usage-based pricing or credit system (optional for MVP)
7. Tech Stack Preferred (suggested, not mandatory)
• Frontend: React.js or Next.js
• Backend: Node.js + Express / Python (FastAPI)
• Database: MongoDB / PostgreSQL
• AI: OpenAI API / Claude / Llama2
• Hosting: Vercel, Supabase, or custom cloud
⸻
Deliverables:
• Fully functional frontend and backend
• REST API documentation
• Admin dashboard (basic)
• Source code with deployment instructions
• Assistance with initial deployment (if needed)
It should have all of it own functionality no api of any web scraper
⸻
Key Features Required:
1. Website Crawler
• Crawl any given URL
• Capture text, meta info, links, HTML
• Store page data in structured format
2. Web Scraper
• Scrape visible content + hidden data (e.g. meta tags, OpenGraph, schema)
• Extract data in JSON format
3. AI Summarizer
• Use GPT or any LLM to summarize page content
• Key Points, SEO tags, summaries, etc.
• Ability to customize prompt for LLM
4. API Access
• Provide REST API to:
• Submit crawl requests
• Fetch structured data
• Get AI summaries
• API key and user auth support
5. Dashboard / UI
• Clean frontend UI (like Firecrawl) using React/Next.js
• Input URLs and get structured response
• Show crawl history, results, logs
• Option to export data
6. Auth and Credits System
• Login/Register via email or OAuth (Google, GitHub)
• Usage-based pricing or credit system (optional for MVP)
7. Tech Stack Preferred (suggested, not mandatory)
• Frontend: React.js or Next.js
• Backend: Node.js + Express / Python (FastAPI)
• Database: MongoDB / PostgreSQL
• AI: OpenAI API / Claude / Llama2
• Hosting: Vercel, Supabase, or custom cloud
⸻
Deliverables:
• Fully functional frontend and backend
• REST API documentation
• Admin dashboard (basic)
• Source code with deployment instructions
• Assistance with initial deployment (if needed)
It should have all of it own functionality no api of any web scraper
Related categories:
PHP
Python
Web Scraping
HTML
Node.js
JSON
Web Development
Data Scraping
Full Stack Development
AI Development