Threshold turns a webcam into a volumetric instrument. Point it at yourself and an AI depth model, gesture tracking, and a generative audio engine take over — your face and hands become a live field of ASCII glyphs, dithered blocks, or particles, extruded by depth and colored by mood, with a soundtrack that reacts to how much you move.

It's built entirely client-side: React Three Fiber and Three.js for rendering, Transformers.js for depth estimation, MediaPipe for hand tracking, Tone.js for audio, Zustand for state, Leva for the control panel. No backend, no accounts, nothing leaves your browser.

The premise was simple to state and hard to build: what if a webcam filter wasn't flat? Most "AI camera effects" apply a 2D mask over your face. I wanted the camera feed to become actual topography — a grid of shapes physically pushed forward and back by an AI's read of depth, in real time, at 60fps.

The aesthetic brief I gave myself was "high-fidelity terminal" — somewhere between MSCHF's absurdist repurposing, early Flash-era web experiments, Yugo Nakamura's interaction design, and Ryoji Ikeda's audiovisual minimalism. Scanlines, chromatic aberration, volumetric bloom, a monospace hardware feel. The Leva control panel needed to look like a lab workstation, not a debug menu.

One rule I set early and kept: no static pre-processing. Whatever showed up on screen had to be computed live, every frame, from whatever the camera saw that instant.

Full AI depth estimation was too slow for a live mirror. I started with Transformers.js's depth-anything-small-hf running per-frame, and it couldn't hold 60fps. I replaced the primary interactive loop with real-time pixel-to-luminance sampling instead — a pragmatic trade that kept the feel of depth without the AI model in the hot path. True per-pixel depth inference still exists in the codebase, but it's not what drives the live experience.

Two coloring systems disagreeing looks like a bug — because it is one. Mood color (chromeColor, an emissive tint) and mode color (getGradientColor, a diffuse gradient) were built as separate systems and would occasionally contradict each other, reading as random flicker. The fix wasn't a patch — it was making mood the single source of truth for hue when mood mode is on.

A missing feature can look identical to a rendering bug. The ASCII mode had a full glyph atlas texture wired up, but no per-cell UV-offset logic — so every cell silently rendered the same glyph. It looked like a broken shader. It was an incomplete one. Fixed with an InstancedBufferAttribute and a shader patch via onBeforeCompile.

Audio reliability bugs are usually race conditions in disguise. Three separate places called Promise.race([Tone.start(), timeout(500)]) without ever checking whether the audio context had actually resumed. Under load, all three could "succeed" against the timeout while the context stayed suspended — silent audio, no error. The fix was a single shared ensureAudioContext() gate that awaits the real context state.

Perceptual and linear scales don't mix. Default volume was 33dB quieter than intended because per-layer gain reductions compounded on a linear dB curve. Swapping in a perceptual volumeToDb mapping fixed it in one pass.

Webcam → signal. The camera feed feeds two paths: brightness sampling for the live depth-proxy, and MediaPipe hand tracking (throttled to every third frame to manage GPU heat) for gesture control.

Signal → state. Everything lands in a single Zustand store — render mode, theme, mood, gesture state, session energy.

State → render. An R3F InstancedMesh scene reads the store every frame, coloring each cell via the shared gradient system in theme.ts. A fixed post-processing chain runs on top: Bloom → HueSaturation → ChromaticAberration → Glitch → Scanline → Noise → Vignette.

State → audio. A parallel Tone.js engine builds textures (glitch, bloom, bass) and a Markov-chain melody generator, driven by the same motion-energy signal that drives the visuals.

Two systems run orthogonally on top of all this: mood (manual — five palettes, each with its own scale, tempo, waveform, and texture) and session arc (automatic — an energy meter from 0–100 that charges from movement and decays over time, moving you through calm → building → climax phases with hysteresis so it doesn't flicker between states).

This needs a webcam and a browser gesture (a click) before audio will play — standard autoplay policy, not a bug. It leans on WebGL and on-device inference, so it wants a reasonably modern machine; on weaker hardware, expect fan noise before you expect frame drops.

It's an experiment, not a product. Some of the internal naming has aged in ways I've left alone on purpose — the "sound texture" options (glitch/bloom/bass) happen to share names with unrelated post-processing effects, a coincidence from building fast rather than a designed pun. Nothing here is trying to be finished software; it's trying to be an honest record of a specific idea, worked all the way through.