How can XR devices understand hand movements with a single camera? Find out in the EgoForce Conference paper!
Researchers from the German Research Center for Artificial Intelligence (DFKI), Rhineland-Palatinate Technical University of Kaiserslautern-Landau (RPTU), and the Max Planck Institute for Informatics (MPII) presented their latest research at SIGGRAPH 2026 on 21 July 2026 in Los Angeles, USA.
[READ THE PAPER HERE]
The paper "EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera" introduces a framework for reconstructing the absolute 3D pose and shape of the hands using only a single head-mounted RGB camera.
Advancing egocentric hand tracking
Accurate hand tracking is essential for natural interaction in extended reality. EgoForce addresses this challenge by incorporating forearm information to improve depth estimation and hand pose reconstruction while supporting fisheye, perspective, and wide field-of-view camera models within a single architecture. The framework combines a unified hand-arm architecture (HALO), a differentiable forearm representation, and a ray-space solver for accurate camera-space 3D hand pose estimation across different optical systems.
Results and relevance for IRIS-XR
The framework was evaluated on several egocentric benchmarks, achieving state-of-the-art performance. On the HOT3D dataset, EgoForce reduced camera-space joint error by up to 28% compared with previous methods while demonstrating robust tracking during hand-object interactions and severe occlusions. The system also runs interactively on smart glasses using a single RGB camera.
The research reflects several technological directions that are central to IRIS-XR, including AI-based perception, lightweight sensing, and natural interaction for the next generation of XR devices.
Acknowledgments
This work was partially funded by the Horizon Europe programme under the projects dAIEDGE, Grant Agreement No. 101120726, and IRIS-XR, Grant Agreement No. 101298672. The authors thank the anonymous reviewers for their valuable feedback.
Join our journey!
Subscribe to our newsletter and follow us on LinkedIn, BlueSky, YouTube and X to keep up to date!
