Enhance AR Nail Try-On Application -- 2

Project Brief: Add Nail Segmentation Model
Overview

We currently have a working AR nail try-on prototype that uses MediaPipe HandLandmarker to detect finger tips and estimate nail positions.
Right now, the app renders artificial nails by drawing shapes on top of detected fingertip coordinates, which is not accurate enough for a production app.

We need to integrate a nail segmentation model that can precisely identify the nail region in the camera feed, so the nail polish color can be applied pixel-perfectly.

Goals

Replace the fingertip-based approximation with a real-time nail segmentation model.

For each detected hand:

Generate a segmentation mask of all visible nails.

Apply the selected color and effects (opacity, gloss, etc.) only inside the nail region.

Maintain real-time performance (minimum 20 FPS on mid-range phones).

Current Tech Stack

Frontend: HTML, CSS, Vanilla JS

AR Pipeline:

Camera feed via getUserMedia

Hand detection via MediaPipe HandLandmarker

Rendering: Canvas (2D context)

UI controls: Nail color, opacity, gloss, nail size, blur, etc.

What Needs to Be Built
1. Nail Segmentation Model Integration

Acceptable options:

TensorFlow.js, ONNX, or MediaPipe Tasks Vision custom model.

If using TF.js or ONNX:

Load the model directly in the browser.

Support WebGL acceleration for performance.

Input: Camera frame (HTMLVideoElement or frame buffer).

Output:

A segmentation mask where each pixel is nail vs. non-nail (1 for nail, 0 for background/skin).

2. Mask Rendering

Convert the mask into a Canvas overlay.

Apply the selected color and opacity ONLY to nail pixels:

ctx.globalCompositeOperation = 'source-in';


Blend gloss and feathering options for a natural finish.

How It Should Work

Camera frame → HandLandmarker for hand bounding boxes (optional for speed).

Crop ROI (region of interest) around detected hands.

Feed cropped image into nail segmentation model.

Receive mask → upscale to full frame size.

Overlay color & effects on nails, blending smoothly with the skin.

Performance Requirements

Target 20–30 FPS on:

iPhone XR or later

Mid-range Android phones (Snapdragon 7xx)

Optimize by:

Processing cropped hand regions instead of full frame.

Running segmentation every 2–3 frames and interpolating.

Deliverables

Updated index.html and main.js with:

Model loading and initialization.

Live segmentation pipeline.

Nail mask rendering logic.

Model file(s) hosted on a CDN or local /models folder.

Clear documentation for:

How to update the model in the future.

Performance tuning tips.

Add multi-hand segmentation (up to 2 hands).

Export before/after screenshots of nail try-on.

Support custom nail art textures, not just flat colors