News Scraper & Summarizer module
Budget: ₹600 – ₹1,500 INR
I need an end-to-end pipeline that automatically fetches full-text articles from Times of India, BBC and CNN, stores the raw and normalized data on S3, then produces concise LLM-generated summaries — all without creating duplicates when the job is rerun.
Architecture & flow
• An AWS Lambda “fetch” function triggers per source on a schedule, each source implemented as its own small adapter so that future outlets can be plugged in without touching the core pipeline.
• The canonical URL (or another deterministic key) must become article_id so every article retains a stable identity across reruns.
• Raw HTML lands in s3://news/raw/--date--, then a normalizer writes JSON to s3://news/normalized/.
• A second Lambda reads the normalized JSON, calls the chosen LLM for summary output, and saves a separate JSON file in s3://news/summaries/ plus a daily manifest.
• Broad categories (politics, sports, business) can derive from native site sections, rules, or an LLM — whichever is easiest to keep accurate — but finer topics such as “elections” or “IPL” must be tagged strictly through rule-based logic so users can filter by their saved interests.
Data contracts
version: "1.0" for normalized articles
– metadata (title, author, publish_time, source, article_id)
– body_html, body_text
– categories[], topics[]
version: "1.0" for summaries
– headline, short_text, bullet_points[], model_name, prompt_version
Deliverables
1. Terraform or CloudFormation template setting up the two Lambda functions, S3 buckets and IAM roles.
2. Source adapters for Times of India, BBC and CNN with accompanying unit tests.
3. Normalizer and summarizer code (Python preferred, open to other runtimes).
4. Rule set for topic extraction, documented and test-covered.
5. Sample manifests and both JSON schemas.
6. README explaining local testing, deployment and how to add a new source.
Acceptance criteria
• Running the job twice on the same day does not create duplicate article_ids.
• Summaries respect the JSON schema and remain under 100 words.
• Topics are populated solely by the supplied rules.
• Adding a fourth source requires only a new adapter and config entry, no core code changes.
If anything is unclear, let me know — I’d like to kick this off as soon as we agree on the approach.
Architecture & flow
• An AWS Lambda “fetch” function triggers per source on a schedule, each source implemented as its own small adapter so that future outlets can be plugged in without touching the core pipeline.
• The canonical URL (or another deterministic key) must become article_id so every article retains a stable identity across reruns.
• Raw HTML lands in s3://news/raw/--date--, then a normalizer writes JSON to s3://news/normalized/.
• A second Lambda reads the normalized JSON, calls the chosen LLM for summary output, and saves a separate JSON file in s3://news/summaries/ plus a daily manifest.
• Broad categories (politics, sports, business) can derive from native site sections, rules, or an LLM — whichever is easiest to keep accurate — but finer topics such as “elections” or “IPL” must be tagged strictly through rule-based logic so users can filter by their saved interests.
Data contracts
version: "1.0" for normalized articles
– metadata (title, author, publish_time, source, article_id)
– body_html, body_text
– categories[], topics[]
version: "1.0" for summaries
– headline, short_text, bullet_points[], model_name, prompt_version
Deliverables
1. Terraform or CloudFormation template setting up the two Lambda functions, S3 buckets and IAM roles.
2. Source adapters for Times of India, BBC and CNN with accompanying unit tests.
3. Normalizer and summarizer code (Python preferred, open to other runtimes).
4. Rule set for topic extraction, documented and test-covered.
5. Sample manifests and both JSON schemas.
6. README explaining local testing, deployment and how to add a new source.
Acceptance criteria
• Running the job twice on the same day does not create duplicate article_ids.
• Summaries respect the JSON schema and remain under 100 words.
• Topics are populated solely by the supplied rules.
• Adding a fourth source requires only a new adapter and config entry, no core code changes.
If anything is unclear, let me know — I’d like to kick this off as soon as we agree on the approach.