Leveraging Depth Cameras and Wearable Pressure Sensors for Full-body Kinematics and Dynamics Capture

We present a new method for full-body motion capture that uses input data captured by three depth cameras and a pair of pressure- sensing shoes. Our system is appealing because it is low-cost, non-intrusive and fully automatic, and can accurately reconstruct both full-body kinematics and dynamics data. We first introduce a novel tracking process that automatically reconstructs 3D skeletal poses using input data captured by three Kinect cameras and wear- able pressure sensors. We formulate the problem in an optimiza- tion framework and incrementally update 3D skeletal poses with observed depth data and pressure data via iterative linear solvers. The system is highly accurate because we integrate depth data from multiple depth cameras, foot pressure data, detailed full-body ge- ometry, and environmental contact constraints into a unified frame- work. In addition, we develop an efficient physics-based motion reconstruction algorithm for solving internal joint torques and con- tact forces in the quadratic programming framework. During re- construction, we leverage Newtonian physics, friction cone con- straints, contact pressure information, and 3D kinematic poses ob- tained from the kinematic tracking process to reconstruct full-body dynamics data. We demonstrate the power of our approach by cap- turing a wide range of human movements and achieve state-of-the- art accuracy in our comparison against alternative systems.