Web Scraper & Supabase Import Automation
Budget: $30 – $250 USD
Automated Profile Scraper & Hourly Supabase Import
Project Type: Data Scraping, Automation, Supabase Integration
Delivery Timeline: 2 days max
1. Project Overview
I am building an escort directory called Rosey using Next.js and Supabase (Postgres). I need
a fully automated scraper and data pipeline that extracts all profiles from a target escort
directory (escortsaffair.com or adultsearch.com) and imports them directly into Supabase. This is a fresh data import , the scraped
data becomes the canonical dataset for the platform.
After initial setup, the system must run autonomously every hour to reflect new profiles,
profile updates, and removed profiles from the source site.
2. Core Requirements
- Scrape all profiles from the target directory ( escortsaffair.com or adultsearch.com)
- Extract rich, structured profile data
- Insert / upsert profiles and images directly into Supabase
- Run automatically every hour
- Be idempotent (safe to rerun, no duplicates)
- Skip broken profiles without crashing
3. Technology
Any technology may be used (Node.js, Python, Playwright, Puppeteer, etc.) as long as:
- It integrates cleanly with Supabase
- It runs via a single command (e.g. npm run scrape)
- It supports scheduled execution (cron, GitHub Actions, server scheduler)
- All secrets are supplied via environment variables:
//////////////////////////////
SUPABASE_URL
SUPABASE_SERVICE_ROLE_KEY
4. Data to Extract (Summary)
- Name, username (derived from URL), source_url
- Bio / description
- City, state, country + slugs
- Home locations (array)
- Demographics (gender, age, body type, etc. when available)
- Rates (base currency and base hourly rate)
- Contact info (email, phone)
- Social links (array)
- Images (external URLs only, largest versions preferred)Profile schema currently looks like this
create table public.profiles (
id uuid not null default gen_random_uuid (),
user_id uuid null,
working_name text not null,
username text not null,
display_name text null,
slug text null,
tagline text null,
about text null,
age integer null,
displayed_age integer null,
gender text null,
pronouns text[] not null default array[]::text[],
identity_tags text[] not null default array[]::text[],
city text null,
state text null,
country text null,
city_slug text null,
country_slug text null,
home_locations text[] not null default array[]::text[],
latitude numeric null,
longitude numeric null,
languages text[] not null default array[]::text[],
categories text[] not null default array[]::text[],
caters_to text[] not null default array[]::text[],
available_days text[] not null default array[]::text[],
base_currency text not null default '''USD''::text'::text,
base_hourly_rate numeric not null default 0,
is_active boolean not null default true,
onboarding_completed boolean not null default false,
appear_on_other_profiles boolean not null default true,
temporary_hide_days integer not null default 14,
status public.profile_status not null default 'draft'::profile_status,
verification public.verification_level not null default 'none'::verification_level,
verification_notes text null,
verified_at timestamp with time zone null,
instagram text null,
created_at timestamp with time zone not null default now(),
updated_at timestamp with time zone not null default now(),
body_type text null,
ethnicity_category text null,
gender_presentation text null,
profile_type text null,
trans_only boolean not null default false,
trans_status text null,
is_verified boolean not null default false,
approval_status text null,
verification_photo_verified boolean not null default false,
id_verified boolean not null default false,
min_photos_verified boolean not null default false,
profile_fields_verified boolean not null default false,
is_fully_verified boolean not null default false,
height_cm integer null,
eye_color text null,
hair_color text null,
is_420_friendly boolean not null default false,
contact_email text null,
contact_phone text null,
claim_status text not null default 'unclaimed'::text,
claim_notes text null,
socials text[] not null default array[]::text[],
source_url text null,
state_slug text null,
constraint profiles_pkey primary key (id),
constraint profiles_slug_key unique (slug),
constraint profiles_user_id_key unique (user_id),
constraint profiles_username_key unique (username),
constraint profiles_temporary_hide_days_check check ((temporary_hide_days >= 0)),
constraint profiles_age_check check (
(
(age is null)
or (
(age >= 18)
and (age <= 120)
)
)
),
constraint profiles_username_not_blank check (
(
length(
TRIM(
both
from
username
)
) > 0
)
),
constraint profiles_base_hourly_rate_check check ((base_hourly_rate >= (0)::numeric)),
constraint profiles_displayed_age_check check (
(
(displayed_age is null)
or (
(displayed_age >= 18)
and (displayed_age <= 120)
)
)
),
constraint profiles_slug_lowercase check (
(
(slug is null)
or (slug = lower(slug))
)
)
) TABLESPACE pg_default;
create index IF not exists profiles_user_id_idx on public.profiles using btree (user_id)
TABLESPACE pg_default;
create index IF not exists profiles_city_country_idx on public.profiles using btree (country,
city) TABLESPACE pg_default;
create index IF not exists profiles_status_idx on public.profiles using btree (status)
TABLESPACE pg_default;
create index IF not exists profiles_is_active_idx on public.profiles using btree (is_active)
TABLESPACE pg_default;
create unique INDEX IF not exists profiles_user_id_unique on public.profiles using btree
(user_id) TABLESPACE pg_default
where
(user_id is not null);
create unique INDEX IF not exists profiles_source_url_unique on public.profiles using btree
(source_url) TABLESPACE pg_default;
create trigger profiles_set_updated_at BEFORE
update on profiles for EACH row
execute FUNCTION set_updated_at ();
create trigger trg_create_wallet_on_claimafter
update OF user_id on profiles for EACH row
execute FUNCTION create_wallet_on_claim ();
create trigger trg_profiles_claim_status BEFORE INSERT
or
update on profiles for EACH row
execute FUNCTION set_claim_status_from_user_id ();
5. Supabase Integration
- Insert / upsert into profiles table
- Insert / upsert into images table
- Use source_url as the unique identifier
- Running hourly must not create duplicates
- If a profile disappears from the source, mark it inactive (do not delete)
6. Required Monitoring UI
A simple UI is required for monitoring and control of the scraper. This UI does not need
advanced styling or authentication.
The UI must allow:
- Triggering a manual scrape run
- Viewing last run time and status (success / error)
- Viewing counts (profiles scraped, inserted, updated)
- Viewing recent logs or errors
The UI can be local-only (e.g. http://localhost:3000 or similar).
7. Automation
- Designed to run automatically every hour
- No manual steps after setup
- Suitable for long-running unattended operation
8. Error Handling
- Profile-level failures must be skipped and logged
- Errors must not terminate the full scraping run9. Deliverables
1. Full source code
2. README with setup and scheduling instructions
3. Working monitoring UI
4. Sample Supabase-imported data
Project Type: Data Scraping, Automation, Supabase Integration
Delivery Timeline: 2 days max
1. Project Overview
I am building an escort directory called Rosey using Next.js and Supabase (Postgres). I need
a fully automated scraper and data pipeline that extracts all profiles from a target escort
directory (escortsaffair.com or adultsearch.com) and imports them directly into Supabase. This is a fresh data import , the scraped
data becomes the canonical dataset for the platform.
After initial setup, the system must run autonomously every hour to reflect new profiles,
profile updates, and removed profiles from the source site.
2. Core Requirements
- Scrape all profiles from the target directory ( escortsaffair.com or adultsearch.com)
- Extract rich, structured profile data
- Insert / upsert profiles and images directly into Supabase
- Run automatically every hour
- Be idempotent (safe to rerun, no duplicates)
- Skip broken profiles without crashing
3. Technology
Any technology may be used (Node.js, Python, Playwright, Puppeteer, etc.) as long as:
- It integrates cleanly with Supabase
- It runs via a single command (e.g. npm run scrape)
- It supports scheduled execution (cron, GitHub Actions, server scheduler)
- All secrets are supplied via environment variables:
//////////////////////////////
SUPABASE_URL
SUPABASE_SERVICE_ROLE_KEY
4. Data to Extract (Summary)
- Name, username (derived from URL), source_url
- Bio / description
- City, state, country + slugs
- Home locations (array)
- Demographics (gender, age, body type, etc. when available)
- Rates (base currency and base hourly rate)
- Contact info (email, phone)
- Social links (array)
- Images (external URLs only, largest versions preferred)Profile schema currently looks like this
create table public.profiles (
id uuid not null default gen_random_uuid (),
user_id uuid null,
working_name text not null,
username text not null,
display_name text null,
slug text null,
tagline text null,
about text null,
age integer null,
displayed_age integer null,
gender text null,
pronouns text[] not null default array[]::text[],
identity_tags text[] not null default array[]::text[],
city text null,
state text null,
country text null,
city_slug text null,
country_slug text null,
home_locations text[] not null default array[]::text[],
latitude numeric null,
longitude numeric null,
languages text[] not null default array[]::text[],
categories text[] not null default array[]::text[],
caters_to text[] not null default array[]::text[],
available_days text[] not null default array[]::text[],
base_currency text not null default '''USD''::text'::text,
base_hourly_rate numeric not null default 0,
is_active boolean not null default true,
onboarding_completed boolean not null default false,
appear_on_other_profiles boolean not null default true,
temporary_hide_days integer not null default 14,
status public.profile_status not null default 'draft'::profile_status,
verification public.verification_level not null default 'none'::verification_level,
verification_notes text null,
verified_at timestamp with time zone null,
instagram text null,
created_at timestamp with time zone not null default now(),
updated_at timestamp with time zone not null default now(),
body_type text null,
ethnicity_category text null,
gender_presentation text null,
profile_type text null,
trans_only boolean not null default false,
trans_status text null,
is_verified boolean not null default false,
approval_status text null,
verification_photo_verified boolean not null default false,
id_verified boolean not null default false,
min_photos_verified boolean not null default false,
profile_fields_verified boolean not null default false,
is_fully_verified boolean not null default false,
height_cm integer null,
eye_color text null,
hair_color text null,
is_420_friendly boolean not null default false,
contact_email text null,
contact_phone text null,
claim_status text not null default 'unclaimed'::text,
claim_notes text null,
socials text[] not null default array[]::text[],
source_url text null,
state_slug text null,
constraint profiles_pkey primary key (id),
constraint profiles_slug_key unique (slug),
constraint profiles_user_id_key unique (user_id),
constraint profiles_username_key unique (username),
constraint profiles_temporary_hide_days_check check ((temporary_hide_days >= 0)),
constraint profiles_age_check check (
(
(age is null)
or (
(age >= 18)
and (age <= 120)
)
)
),
constraint profiles_username_not_blank check (
(
length(
TRIM(
both
from
username
)
) > 0
)
),
constraint profiles_base_hourly_rate_check check ((base_hourly_rate >= (0)::numeric)),
constraint profiles_displayed_age_check check (
(
(displayed_age is null)
or (
(displayed_age >= 18)
and (displayed_age <= 120)
)
)
),
constraint profiles_slug_lowercase check (
(
(slug is null)
or (slug = lower(slug))
)
)
) TABLESPACE pg_default;
create index IF not exists profiles_user_id_idx on public.profiles using btree (user_id)
TABLESPACE pg_default;
create index IF not exists profiles_city_country_idx on public.profiles using btree (country,
city) TABLESPACE pg_default;
create index IF not exists profiles_status_idx on public.profiles using btree (status)
TABLESPACE pg_default;
create index IF not exists profiles_is_active_idx on public.profiles using btree (is_active)
TABLESPACE pg_default;
create unique INDEX IF not exists profiles_user_id_unique on public.profiles using btree
(user_id) TABLESPACE pg_default
where
(user_id is not null);
create unique INDEX IF not exists profiles_source_url_unique on public.profiles using btree
(source_url) TABLESPACE pg_default;
create trigger profiles_set_updated_at BEFORE
update on profiles for EACH row
execute FUNCTION set_updated_at ();
create trigger trg_create_wallet_on_claimafter
update OF user_id on profiles for EACH row
execute FUNCTION create_wallet_on_claim ();
create trigger trg_profiles_claim_status BEFORE INSERT
or
update on profiles for EACH row
execute FUNCTION set_claim_status_from_user_id ();
5. Supabase Integration
- Insert / upsert into profiles table
- Insert / upsert into images table
- Use source_url as the unique identifier
- Running hourly must not create duplicates
- If a profile disappears from the source, mark it inactive (do not delete)
6. Required Monitoring UI
A simple UI is required for monitoring and control of the scraper. This UI does not need
advanced styling or authentication.
The UI must allow:
- Triggering a manual scrape run
- Viewing last run time and status (success / error)
- Viewing counts (profiles scraped, inserted, updated)
- Viewing recent logs or errors
The UI can be local-only (e.g. http://localhost:3000 or similar).
7. Automation
- Designed to run automatically every hour
- No manual steps after setup
- Suitable for long-running unattended operation
8. Error Handling
- Profile-level failures must be skipped and logged
- Errors must not terminate the full scraping run9. Deliverables
1. Full source code
2. README with setup and scheduling instructions
3. Working monitoring UI
4. Sample Supabase-imported data