Human Data-Driven Calibration of Hand Exoskeletons Using Redundant Sensors for Improved Teleoperation

Haiyun Zhang1, Stefano Dalla Gasperina1, Saad N. Yousaf1, Toshimitsu Tsuboi2, Tetsuya Narita2, Ashish D. Deshpande1,3

1Walker Department of Mechanical Engineering, The University of Texas at Austin, USA
2Sony Group Corporation, Tokyo, Japan
3Meta Reality Labs Research, Redmond, WA, USA
Project funded by Sony Group Corporation
arXiv

Abstract

Hand exoskeletons are critical tools for dexterous teleoperation and immersive manipulation interfaces, but achieving accurate hand tracking remains a challenge due to user-specific anatomical variability and donning inconsistencies. These issues lead to kinematic misalignments that degrade tracking performance and limit applicability in precision tasks. This paper presents a subject-specific calibration framework for exoskeleton-based hand tracking that leverages redundant joint sensing and a residual-weighted optimization strategy to estimate virtual link parameters. Implemented on the Maestro exoskeleton, our method improves joint angle and fingertip position estimation across users with diverse hand geometries. We introduce a data-driven approach to empirically tune cost function weights using motion capture ground truth, enabling more accurate and consistent calibration across participants. Quantitative results from seven subjects show substantial reductions in joint and fingertip tracking errors compared to uncalibrated and evenly weighted models. Qualitative visualizations using a Unity-based virtual hand further confirm improvements in motion fidelity. The proposed framework generalizes across exoskeleton designs with closed-loop kinematics and minimal sensing, and lays the foundation for high-fidelity teleoperation and learning-from-demonstration applications.

Motivation

This study presents a human-data-driven calibration framework for enhancing hand tracking accuracy in exoskeleton systems. The approach comprises the following key components: (0) a two-phase calibration protocol in which subjects first fully extend their fingers, followed by flexion at the metacarpophalangeal (MCP) joints while maintaining interphalangeal (IP/PIP) joints in an extended posture; (1) optimization of the virtual linkage between exoskeleton joint centers and corresponding anatomical finger joints; (2) evaluation of root mean square errors (RMSEs) between estimated and measured joint angles for both redundant exoskeleton joints and anatomically configured human joints; and (3) construction of a weighted cost function through a systematic weight distribution search based on data collected from seven human participants.
Calibration Diagram

Real-Time Teleoperation in a Unity-Based Simulation Environment

The teleoperation is implemented by mapping the hand tracking data to a virtual humanoid hand within a Unity-based simulation environment. In the visualization, the red hand represents motion captured prior to calibration, while the green hand reflects motion following the application of the calibration procedure.

Real-Time Visualization of the Kinematic Calibration Procedure

The calibration algorithm is executed online prior to the hand tracking experiments. The three accompanying video segments illustrate the two-phase calibration procedure and demonstrate how the resulting parameters are integrated into the real-time kinematic models.

Thumb Calibration Process

Index Calibration Process

Middle Calibration Process

Comparative Visualization of Pre- and Post-Calibration Kinematics

The kinematic models before and after calibration exhibit clear visual differences. The first segment of the video toggles between the pre-calibrated and post-calibrated models across various hand postures to highlight alignment discrepancies. The subsequent segments present continuous hand motion sequences, captured separately under the pre- and post-calibration conditions, to demonstrate improvements in motion fidelity.

BibTeX

@article{zhang2025human,
  title={Human-Exoskeleton Kinematic Calibration to Improve Hand Tracking for Dexterous Teleoperation},
  author={Zhang, Haiyun and Gasperina, Stefano Dalla and Yousaf, Saad N and Tsuboi, Toshimitsu and Narita, Tetsuya and Deshpande, Ashish D},
  journal={arXiv preprint arXiv:2507.23592},
  year={2025}
}