Building AR Try-On with MediaPipe and Three.js

The brief for EyeBrowse was simple: let customers try on glasses virtually, like Warby Parker. The reality was anything but simple. Browser-based AR means no native APIs, no ARKit, no depth sensor — just a webcam feed and whatever you can compute in JavaScript at 30fps.

Here is how the system was built.

The 3-Layer Pipeline

The virtual try-on system is a pipeline with three distinct layers, each solving a different problem:

1. Face Mesh Extraction — MediaPipe/TensorFlow FaceMesh detects 468 3D landmarks per frame
2. Head Pose Computation — Landmark geometry is converted into yaw, pitch, and roll angles
3. Frame Overlay — Either a 2D image cross-fade or a full 3D GLB model is positioned on the face

Each layer runs independently, which means the rendering strategy (2D vs 3D) can swap without touching the face detection code. That separation turned out to be critical when the 3D path showed stability issues on certain mobile GPUs.

Layer 1: Face Mesh

TensorFlow's FaceMesh model returns 468 landmarks with x, y, and z coordinates normalized to the video dimensions. The key landmarks for glasses placement are:

• Iris centers (indices 468-472, 473-477) — used for pupillary distance measurement
• Nose bridge (index 6) — the anchor point for frame positioning
• Ear attachment points (indices 234, 454) — temple arm alignment
• Forehead and chin groups — head pose vectors

const keypoints = face.keypoints;
const leftIris = keypoints.find(k => k.name === 'leftIris')
  || keypoints[LEFT_IRIS_CENTER];
const rightIris = keypoints[RIGHT_IRIS_CENTER];

One thing that catches teams off guard: the detector initialization can take up to 15 seconds on cold start, especially on mobile. A 2-second watchdog timer detects stalls and retries initialization. Without it, roughly 8% of mobile sessions would hang on a loading spinner forever.

Layer 2: Head Pose

This is where it gets interesting. Flat landmark coordinates have to be converted into 3D rotation angles so the glasses track the user's head movement naturally.

The approach uses anatomical landmark groups to construct two 3D vectors — an ear-to-ear "right" vector and a forehead-to-chin "up" vector — then computes their cross product to get the forward-facing direction:

const rightFace = avg3([454, 356, 389]);
const leftFace = avg3([234, 162, 127]);
const foreheadGroup = avg3([10, 109, 338]);
const chinGroup = avg3([152, 148, 377]);
 
// forward = cross(right, up)
const yaw = Math.atan2(-fx, fz) * (180 / Math.PI);
const pitch = -Math.asin(clamp(fy)) * (180 / Math.PI);

In practice, the full 3D computation produced angles that were too aggressive — glasses would appear upside-down at extreme angles. The shipped version uses a 2D heuristic for yaw ((noseTipX - faceCenterX) * 90) that feels more natural, even though it is technically less accurate. Sometimes the perceptually correct answer beats the mathematically correct one.

Layer 3: The Rendering Split

2D Path: Directional Image Cross-Fade

The 2D path composites pre-processed transparent WebP images of frames onto the video feed. The key innovation is directional blending — as the user turns their head, the system smoothly cross-fades between front, left-angle, and right-angle images of the same frame:

function sideBlendFactor(yaw: number): number {
  const absYaw = Math.abs(yaw);
  if (absYaw < 12) return 0;       // pure front
  if (absYaw > 30) return 1;       // full side
  return easeInOut((absYaw - 12) / 18); // smooth transition
}

Temple arms fade based on head angle — when the user turns right, the left temple arm gradually disappears since it would be occluded by the face. This sells the illusion more than any other single detail.

3D Path: GLB Models with React Three Fiber

For frames with 3D models (GLB format), the system renders them in an orthographic Three.js scene overlaid on the video feed. The environment lighting uses a two-step approach: an immediate RoomEnvironment for instant visibility, then an async HDR environment map swap for studio-quality reflections once loaded.

The trickiest part was the temple fade shader — a custom shader uniform that applies smoothstep falloff to temple arms based on head rotation, preventing the "X-ray vision" effect where the user's head appears to be visible through the temple arm.

The PD Measurement Problem

Pupillary distance (PD) is critical for fitting — wrong PD means the lenses do not align with the pupil centers. Traditional measurement requires a ruler or optometrist visit.

The system uses an iris-diameter calibration approach. The average human iris is 11.7mm (±0.5mm), and MediaPipe gives 5 landmark points around each iris. By computing the pixel diameter of the iris and comparing it to the known physical size, the pipeline derives a px-to-mm ratio and calculates the distance between pupil centers.

The system captures 9 frames at 120ms intervals, takes the median measurement to reject outliers, and applies lens distortion correction. Below 0.7 confidence, the result blends with the population average (63mm) rather than showing a potentially wrong number:

if (confidence >= 0.7) return measuredPd;
if (confidence <= 0.35) return 63; // population average
const blend = (confidence - 0.35) / 0.35;
return 63 + (measuredPd - 63) * blend;

Memory Management for Kiosks

The kiosk deployment runs for hours at a time, and customers try on dozens of frames per session. Without careful memory management, the browser would crash within 30 minutes.

An LRU cache capped at 60 processed frame images keeps memory bounded. Each entry is a blob URL created from canvas operations (background removal, compositing). When the cache is full, the oldest entry is evicted and its blob URL revoked — without that, orphaned blob URLs would leak megabytes per frame.

Lessons That Survive

What would change in a v2: invest earlier in the 3D path. The 2D cross-fade approach works well for front-facing views but breaks down at extreme angles. With the 3D path stable, every frame renders correctly at every angle — no need for multiple pre-processed images per frame.

The other open lane is WebGPU for the face mesh inference. The current TensorFlow.js pipeline is CPU-bound on older devices, and WebGPU's compute shaders could potentially double the frame rate on supported browsers.

The AR try-on is live at eyebrowse.ai — try it with your webcam.