AI Video Conversational Training Platform
Budget: $3,000 – $5,000 USD
We seek a senior full-stack team to build a modular, enterprise conversational training platform with a real-time AI avatar and pre-session identity verification.
First campaign: Cybersecurity awareness. The platform must let us spin up future campaigns (e.g., Ethics, Compliance) by configuring prompts/scenarios in an Admin portal and pointing to another LLM (Gemini/Gemma) — no backend rewrite.
****User Roles (Azure AD SSO + AD Groups)
Administrator (inherits all Analyst permissions)
Create/clone/edit/publish campaigns; versioning (draft/published).
Configure areas/roles per campaign (e.g., IT, Finance, Legal, Sales).
Generate a unique URL per area, or auto-assign area from the user’s AD group at login.
Control scenario weighting: 40% general / 60% area-specific.
Configure prompts, scenarios, difficulty levels, cooldowns, rubrics, thresholds.
Preview/Test mode to validate flows before publishing.
Share campaign URLs with target audiences.
Full reporting: view results, play recordings, export.
Report Analyst
Dashboards by campaign/cohort/area/user; risk indicators, score distributions.
Playback of audio/video recordings; export to CSV/BigQuery.
No access to edit campaigns.
End User
Azure AD SSO.
Capture ID photo (government ID) and a selfie → system performs 1:1 face match and liveness (blink/head-turn).
Conversational training with the AI avatar (voice-to-voice), scenarios selected by area (60% area-specific, 40% general), adaptive difficulty, and instant score + feedback.
****Functional Modules
Auth & Role Management: Azure AD SSO (OIDC/SAML); roles from AD groups; area mapping from AD or via area-specific URL.
Identity Verification (NO KYC): user uploads ID photo + selfie; 1:1 face match; basic liveness; session presence (pause/flag if face lost).
Conversational Session: WebRTC; AI avatar (D-ID Real-Time API or HeyGen) lip-synced; Gemini 2.5 Flash Live (preferred) for low-latency speech-to-speech (or Google STT → Gemini → Google TTS fallback); orchestrator for scenario rotation & dynamic difficulty (1–5) with reprompts.
Scoring & Feedback: rubric (40% signal detection, 40% correct action, 20% process/time), penalties for unsafe actions; immediate feedback + micro-reinforcements.
Admin Portal: CRUD for campaigns/prompts/scenarios/rubrics; configure areas and create area URLs; thresholds (face-match, response time) and weights (40/60); publish/retire.
Reports & Playback: dashboards; GCS video playback (signed URLs); CSV/BigQuery export.
Recording: composite user + avatar streams; retention 30–60 days.
****Google-First Technical Considerations (Required)
Core GCP Services
GKE (Kubernetes): microservices (FastAPI), RTC server, TURN, and web app hosting.
Vertex AI:
LLM: Gemini 2.5 Flash Live (voice), or Gemini text models via Vertex AI for fallback pipeline.
Custom Face-Match: deploy a custom face-embedding model (ArcFace/InsightFace) as a Vertex AI Prediction endpoint (containerized) to verify ID photo ↔ selfie (cosine similarity threshold configurable). Note: Google Vision/MediaPipe detect faces but do not provide identity matching; hence a custom model on Vertex AI (or GKE) is required.
Cloud SQL (Postgres): relational data; JSONB for prompts/scenarios/rubrics.
Cloud Storage (GCS): encrypted storage of session recordings (lifecycle 30–60 days); signed URLs for admin/analyst playback.
BigQuery: analytics sink (optional export pipeline).
Artifact Registry: container images.
Secret Manager: store API keys (avatar provider, etc.).
Cloud Logging/Monitoring/Trace: observability & SLOs.
Cloud Armor (WAF): protect ingress; rate-limit and IP allow/deny rules.
Workload Identity: no long-lived service keys inside pods.
(Optional) Cloud DNS + HTTPS external LB for web/REST endpoints.
WebRTC/RTC on GCP
RTC server: LiveKit (preferred) or mediasoup deployed on GKE.
STUN/TURN: coturn HA on GKE; enable TLS/443 TURN fallback (many enterprises block UDP).
Networking: dedicated node pool for RTC workloads; PodDisruptionBudgets; autoscaling; static public IPs for TURN; QoS/affinity to minimize jitter; ensure egress rules.
Recording/Egress: composite avatar + user streams server-side; write to GCS via service account with least privilege.
Security & Compliance
End-to-end TLS; JWT validation at gateway; RBAC for Admin/Analyst.
Biometrics policy: store only embeddings and only per session (configurable purge); or anonymize; clear consent screen for camera/recording.
CMEK (Cloud KMS) optional for GCS/Cloud SQL encryption.
Audit logging: who accessed which recording; who published/edited campaigns.
Configurability (Non-Negotiable)
All external endpoints/keys are configurable via env vars/ConfigMaps:
Gemini model names; Vertex AI endpoint for face-match; Google STT/TTS; avatar provider (D-ID/HeyGen) API keys; thresholds (similarity, liveness); weights (40/60); difficulty ceilings per area.
Ability to swap face-match from Vertex AI endpoint → self-hosted GKE service (ONNXRuntime) without code changes beyond config.
**Technical Stack
Frontend: Angular 17+, Angular Material (corporate theming), WebRTC, MediaPipe Face Landmarker (liveness).
Backend: Python FastAPI (async), SQLAlchemy 2 + Alembic, Pydantic v2; services: auth-gateway, identity-verifier, orchestrator, llm-proxy, scoring-engine, recording-service, admin-api.
RTC: LiveKit (preferred) or mediasoup on GKE; coturn for STUN/TURN; composite & egress to GCS.
Data: Cloud SQL (Postgres JSONB), GCS, optional BigQuery export.
Deployment: Docker containers, Helm/Kustomize manifests; GKE Standard; HPA; PodDisruptionBudgets; readiness/liveness probes.
**Deliverables
Angular app (Material UI) with designs/mockups submitted and approved before coding; Azure AD SSO; WebRTC + liveness UI; avatar integrated.
FastAPI microservices (containerized) with OpenAPI docs.
LiveKit/mediasoup + coturn manifests for GKE; recording/egress to GCS.
Admin portal (campaigns, prompts, scenarios, rubrics, areas, area URLs, versioning, test/preview).
Report module with GCS playback (signed URLs) and CSV/BigQuery export.
Scoring engine (40/40/20 + penalties) and feedback generator.
Initial content: 20 scenarios (10 general + 10 area-specific).
Architecture diagram(s); deployment guide; config matrix (all env vars/endpoints/thresholds).
First campaign: Cybersecurity awareness. The platform must let us spin up future campaigns (e.g., Ethics, Compliance) by configuring prompts/scenarios in an Admin portal and pointing to another LLM (Gemini/Gemma) — no backend rewrite.
****User Roles (Azure AD SSO + AD Groups)
Administrator (inherits all Analyst permissions)
Create/clone/edit/publish campaigns; versioning (draft/published).
Configure areas/roles per campaign (e.g., IT, Finance, Legal, Sales).
Generate a unique URL per area, or auto-assign area from the user’s AD group at login.
Control scenario weighting: 40% general / 60% area-specific.
Configure prompts, scenarios, difficulty levels, cooldowns, rubrics, thresholds.
Preview/Test mode to validate flows before publishing.
Share campaign URLs with target audiences.
Full reporting: view results, play recordings, export.
Report Analyst
Dashboards by campaign/cohort/area/user; risk indicators, score distributions.
Playback of audio/video recordings; export to CSV/BigQuery.
No access to edit campaigns.
End User
Azure AD SSO.
Capture ID photo (government ID) and a selfie → system performs 1:1 face match and liveness (blink/head-turn).
Conversational training with the AI avatar (voice-to-voice), scenarios selected by area (60% area-specific, 40% general), adaptive difficulty, and instant score + feedback.
****Functional Modules
Auth & Role Management: Azure AD SSO (OIDC/SAML); roles from AD groups; area mapping from AD or via area-specific URL.
Identity Verification (NO KYC): user uploads ID photo + selfie; 1:1 face match; basic liveness; session presence (pause/flag if face lost).
Conversational Session: WebRTC; AI avatar (D-ID Real-Time API or HeyGen) lip-synced; Gemini 2.5 Flash Live (preferred) for low-latency speech-to-speech (or Google STT → Gemini → Google TTS fallback); orchestrator for scenario rotation & dynamic difficulty (1–5) with reprompts.
Scoring & Feedback: rubric (40% signal detection, 40% correct action, 20% process/time), penalties for unsafe actions; immediate feedback + micro-reinforcements.
Admin Portal: CRUD for campaigns/prompts/scenarios/rubrics; configure areas and create area URLs; thresholds (face-match, response time) and weights (40/60); publish/retire.
Reports & Playback: dashboards; GCS video playback (signed URLs); CSV/BigQuery export.
Recording: composite user + avatar streams; retention 30–60 days.
****Google-First Technical Considerations (Required)
Core GCP Services
GKE (Kubernetes): microservices (FastAPI), RTC server, TURN, and web app hosting.
Vertex AI:
LLM: Gemini 2.5 Flash Live (voice), or Gemini text models via Vertex AI for fallback pipeline.
Custom Face-Match: deploy a custom face-embedding model (ArcFace/InsightFace) as a Vertex AI Prediction endpoint (containerized) to verify ID photo ↔ selfie (cosine similarity threshold configurable). Note: Google Vision/MediaPipe detect faces but do not provide identity matching; hence a custom model on Vertex AI (or GKE) is required.
Cloud SQL (Postgres): relational data; JSONB for prompts/scenarios/rubrics.
Cloud Storage (GCS): encrypted storage of session recordings (lifecycle 30–60 days); signed URLs for admin/analyst playback.
BigQuery: analytics sink (optional export pipeline).
Artifact Registry: container images.
Secret Manager: store API keys (avatar provider, etc.).
Cloud Logging/Monitoring/Trace: observability & SLOs.
Cloud Armor (WAF): protect ingress; rate-limit and IP allow/deny rules.
Workload Identity: no long-lived service keys inside pods.
(Optional) Cloud DNS + HTTPS external LB for web/REST endpoints.
WebRTC/RTC on GCP
RTC server: LiveKit (preferred) or mediasoup deployed on GKE.
STUN/TURN: coturn HA on GKE; enable TLS/443 TURN fallback (many enterprises block UDP).
Networking: dedicated node pool for RTC workloads; PodDisruptionBudgets; autoscaling; static public IPs for TURN; QoS/affinity to minimize jitter; ensure egress rules.
Recording/Egress: composite avatar + user streams server-side; write to GCS via service account with least privilege.
Security & Compliance
End-to-end TLS; JWT validation at gateway; RBAC for Admin/Analyst.
Biometrics policy: store only embeddings and only per session (configurable purge); or anonymize; clear consent screen for camera/recording.
CMEK (Cloud KMS) optional for GCS/Cloud SQL encryption.
Audit logging: who accessed which recording; who published/edited campaigns.
Configurability (Non-Negotiable)
All external endpoints/keys are configurable via env vars/ConfigMaps:
Gemini model names; Vertex AI endpoint for face-match; Google STT/TTS; avatar provider (D-ID/HeyGen) API keys; thresholds (similarity, liveness); weights (40/60); difficulty ceilings per area.
Ability to swap face-match from Vertex AI endpoint → self-hosted GKE service (ONNXRuntime) without code changes beyond config.
**Technical Stack
Frontend: Angular 17+, Angular Material (corporate theming), WebRTC, MediaPipe Face Landmarker (liveness).
Backend: Python FastAPI (async), SQLAlchemy 2 + Alembic, Pydantic v2; services: auth-gateway, identity-verifier, orchestrator, llm-proxy, scoring-engine, recording-service, admin-api.
RTC: LiveKit (preferred) or mediasoup on GKE; coturn for STUN/TURN; composite & egress to GCS.
Data: Cloud SQL (Postgres JSONB), GCS, optional BigQuery export.
Deployment: Docker containers, Helm/Kustomize manifests; GKE Standard; HPA; PodDisruptionBudgets; readiness/liveness probes.
**Deliverables
Angular app (Material UI) with designs/mockups submitted and approved before coding; Azure AD SSO; WebRTC + liveness UI; avatar integrated.
FastAPI microservices (containerized) with OpenAPI docs.
LiveKit/mediasoup + coturn manifests for GKE; recording/egress to GCS.
Admin portal (campaigns, prompts, scenarios, rubrics, areas, area URLs, versioning, test/preview).
Report module with GCS playback (signed URLs) and CSV/BigQuery export.
Scoring engine (40/40/20 + penalties) and feedback generator.
Initial content: 20 scenarios (10 general + 10 area-specific).
Architecture diagram(s); deployment guide; config matrix (all env vars/endpoints/thresholds).
Related categories:
PHP
Website Design
Video Services
Azure
HTML
Google Cloud Platform
Angular
WebRTC
FastAPI
AI Development