Eye2S

Brief

A real-time augmented-reality perception system running on a MacBook's own camera. Where a detector normally has to be retrained to know a new object, Eye2S learns one from six frames held up to the lens: GrabCut isolates the object from the fingers holding it, and DINOv2 embeddings recognise it afterwards at instance level — not "a pen" but that particular pen. Hands are read as gestures rather than used to drive a cursor, and behaviour is attached to an object by dropping a Python file into a folder, hot-loaded without a restart.

What he built

• 01Few-shot object learning: six frames, GrabCut segmentation to cut the object out of the hand holding it, a named identity added at runtime
• 02Instance-level recognition on DINOv2 embeddings (fp16 on Apple MPS), with YOLOE visual-prompt detection so learned objects are found hands-free
• 03MediaPipe hand tracking and learned gestures — recorded over four takes, matched by a representation invariant to speed, position and distance
• 04Hot-reloading plugin system: drop a .py file in and it loads live; a faulty plugin is quarantined so the HUD never goes down with it
• 05Code-free automation rules — on sight of an object, run a shell command, AppleScript, Shortcut, app or webhook
• 06Identity arbitration so one physical object stays one detection: nested boxes merged, ties resolved by score, an unresolved label marked and excluded from automation
• 07Threaded render loop with Kalman-extrapolated boxes to keep the overlay smooth while inference runs behind it

Stack

• Python
• PyTorch
• DINOv2
• YOLOE
• MediaPipe
• OpenCV
• Apple MPS