Export all tweets, replies and image-related threads from a Twitter/X account (verifiable archive format)

Summary

I need a freelancer to programmatically download all tweets and all replies from a specific public Twitter/X account. For any tweet that includes or responds to an image, the freelancer must also download the entire conversation thread (all ancestor and descendant tweets) and all media files.
The output must be verifiable: each tweet must have a canonical link to the original post on X, full raw JSON data, and cryptographic hashes to prove authenticity.

Objectives / Scope of Work

Retrieve every tweet posted by the target account (full history).

Retrieve every reply made by that account.

For any tweet containing or replying to an image:

Download the image files (original quality).

Collect the entire related conversation thread (ancestors and descendants).

Save structured, verifiable data with tweet metadata, text, media info, and canonical URLs.

Provide all deliverables, a reproducible extraction script, and clear documentation.

Deliverables
1. Data Files

a. tweets.jsonl — newline-delimited JSON containing the raw tweet object for each tweet.
Each JSON object must include at least:

id, conversation_id

author_id, author_username, author_name

created_at (ISO-8601, UTC)

text

in_reply_to_status_id, in_reply_to_user_id

public_metrics (retweets, replies, likes, quotes)

entities (hashtags, mentions, urls)

attachments.media_keys

media list with URLs, dimensions, file names, and SHA256 hashes

permalink: https://x.com/i/web/status/<tweet_id>

captured_at (UTC timestamp of extraction)

raw_json: full unmodified API response

source_endpoint: API endpoint used

b. tweets.csv — human-readable table for quick browsing:
Columns: tweet_id, created_at, author_username, text, conversation_id, in_reply_to_id, has_media, media_count, permalink.

2. Media

Folder: media/

Files named <tweet_id>_<index>.<ext>

Manifest file media_manifest.csv with:

tweet_id, original_url, local_path, width, height, media_key, sha256.

3. Thread Mapping

threads.json — mapping each conversation_id → ordered list of tweet_ids representing the full reconstructed thread.

4. Verification Data

hashes.jsonl: list of tweet_id → SHA256 hash of the raw JSON object.

media_hashes.csv: file path → SHA256 hash.

PROVENANCE.md: record of extraction environment (date, library versions, endpoints, total tweet/media counts).

(Optional but recommended):

Timestamp hashes via OpenTimestamps or similar notarization.

For each tweet or thread, archive snapshots on Internet Archive (archive_url).

5. Code & Documentation

Python script or Jupyter notebook (extract_twitter.py or .ipynb) with clear comments.

README.md explaining:

Setup, dependencies, and environment variables.

API key configuration.

Step-by-step instructions.

How to resume extraction.

How to verify hashes and open permalinks.

Optional verification utility (verify_dataset.py) that checks hashes and opens permalinks.

Technical Requirements

Use Twitter/X API v2 (official) with full expansions (author_id, referenced_tweets.id, attachments.media_keys, etc.).

Implement pagination, rate-limit handling, and checkpoint recovery.

Reconstruct threads using conversation_id and recursive lookups.

Download media in original resolution.

Save all timestamps in UTC, ISO-8601.

Preserve raw text (no trimming or escaping changes).

Optionally include oEmbed HTML or screenshots for offline viewing.

Verification & Auditability

Every tweet must include a canonical permalink (https://x.com/i/web/status/<tweet_id>).

Raw JSON data must be identical to API output.

SHA256 hashes must verify data integrity.

Optionally notarize hash set (OpenTimestamps).

Archive.org snapshot URLs should be saved when possible for independent verification.

Legal & Ethical Compliance

Only process public accounts.

No scraping of protected or private data.

Follow Twitter/X Developer Terms and Platform Rules.

Do not republish or redistribute the data without explicit permission.

Respect rate limits and API conditions.

Milestones & Payment

Milestone 1 – Proof of Concept:
Extract 100 most recent tweets + media + threads. Deliver partial dataset and verify structure.

Milestone 2 – Full Extraction:
Deliver complete dataset and media archive.