Rep Sensor
Computer vision rep counting that runs entirely in your browser
33 body landmarks
30fps inference
Zero server uploads
Squat, push-up, curl + more
01The Problem
Camera-based fitness tracking tools usually require cloud processing — your video gets uploaded, analyzed server-side, and results come back. That creates privacy concerns, latency, and ongoing costs. I wanted to run the full ML inference pipeline locally in the browser.
02The Approach
Used Google's MediaPipe Pose Landmarker (WASM) to detect 33 body landmarks in real-time from the device camera. Rep counting is done by tracking joint angles through their range of motion — a squat is counted when the hip landmark crosses the knee landmark and returns. All processing runs on-device.
03Architecture Decisions
MediaPipe WASM for in-browser ML
MediaPipe's Pose Landmarker runs as a WASM module in the browser. It detects 33 full-body landmarks at 30fps on a modern mobile device. No GPU server required — the ML model weights are cached locally after first load.
Angle-based rep detection
Each exercise defines a set of tracked joints and angle thresholds. Squat: track hip-knee-ankle angle. Curl: track shoulder-elbow-wrist angle. A state machine transitions between 'up' and 'down' phases based on angle crossing the threshold.
Exercise-specific calibration
Different body proportions affect landmark positions. A calibration step captures the user's starting position and normalizes landmark coordinates relative to their anatomy — making the angle thresholds work across different body types.
Integration with beef workout app
Rep counts from the sensor can be sent directly to the beef workout logging flow. Completing a set on-camera syncs the rep count, saving manual entry.
04Key Insight
MediaPipe landmark detection is impressive but noisy. Raw landmark positions jitter frame-to-frame even when the person is stationary. Smoothing landmark positions with a rolling average (5-frame window) before computing angles dramatically reduces false rep counts — the signal quality improvement is non-obvious until you actually build the pipeline.
05Why It Matters
Demonstrates browser-side ML integration — a real emerging skill. The privacy-first design (zero server uploads) is more important than it sounds: people are uncomfortable with their gym camera footage going to a cloud server. Running fully local removes that friction.