Researchers Demonstrate Body Tracking From Cameras On VR Controllers

Carnegie Mellon researchers built a body tracking system using cameras on VR controllers.

Consumer VR systems today only track the position of your head and hands. No system on the market comes with tracking for your torso or limbs, but some PC VR enthusiasts attach HTC’s Vive Trackers to their hip and feet, a use case supported by software like VRChat and LIV. Since Vive Trackers cost $139 each, if you don’t already have SteamVR Base Stations the total extra cost exceeds $700. There are unofficial body tracking solutions using old hardware with community made drivers, such as for Microsoft’s Kinect, but this only works from a front-facing angle and the tracking quality isn’t great.

Extra cameras on headsets pointing down at your body could theoretically track limbs, but the view to your legs would often be occluded by your torso or arms, especially as headsets get thinner. Companies like Meta could alternatively release wireless external tracking stations with a camera, battery and mobile chipset running computer vision algorithms, but developers couldn’t rely on users buying and setting up such a costly accessory.

ControllerPose is a prototype solution using two fisheye cameras attached to each VR controller, pointed back at you. The software stitches together the view of the two cameras to achieve an even wider field of view, to see as much of your body as possible. Their pose estimates are accurate to within 8.59 cm, the researchers report.

To process the cameras the researchers tried two approaches: streaming to a nearby PC, and processing onboard with a Raspberry Pi Compute Module 4 and Google Coral. Using a PC (or the standalone headset) would be cheaper, but processing onboard would result in lower latency.

In September last year screenshots emerged appearing to show the upcoming Project Cambria’s controllers with three onboard cameras for self contained inside-out tracking. These cameras don’t appear to be pointed toward you, but a leaked tutorial video did appear to show body tracking. It’s possible that either Cambria or a future Meta headset could perform body tracking using a technique similar to ControllerPose.

The downside of ControllerPose is, according to the researchers, 32% of the time in “typical VR experiences” the controllers don’t have a good view of the body – they’re either too close or off to the side. Future systems could perhaps integrate cameras on the controllers and cameras on the headset, but it seems that only an external solution could achieve fully reliable body tracking. Still, could such a solution work in enough scenarios to be useful to most people?