CHORE: Contact, Human and Object REconstruction from a single RGB image

Xianghui Xie1, Bharat Lal Bhatnagar1,2, Gerard Pons-Moll1,2
1Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
2 University of Tübingen, Germany
ECCV 2022, Tel Aviv

We present CHORE, an approach to jointly reconstruct 3D human, object and their contacts from single RGB image. Our method is able to accurately reason the interactions and recover the spatial arrangement as well as fine-grained contacts between the human and the object.

Abstract

While most works in computer vision and learning have focused on perceiving 3D humans from single images in isolation, in this work we focus on capturing 3D humans interacting with objects. The problem is extremely challenging due to heavy occlusions between human and object, diverse interaction types and depth ambiguity. In this paper, we introduce CHORE, a novel method that learns to jointly reconstruct human and object from a single image. CHORE takes inspiration from recent advances in implicit surface learning and classical model-based fitting. We compute a neural reconstruction of human and object represented implicitly with two unsigned distance fields, and additionally predict a correspondence field to a parametric body as well as an object pose field. This allows us to robustly fit a parametric body model and a 3D object template, while reasoning about interactions. Furthermore, prior pixel-aligned implicit learning methods use synthetic data and make assumptions that are not met in real data. We propose a simple yet effective depth-aware scaling that allows more efficient shape learning on real data. Our experiments show that our joint reconstruction learned with the proposed strategy significantly outperforms the SOTA.

Key idea

Our key idea is to jointly reason human and object at the same time and learn strong spatial arrangement priors from data.

Long narrated video

More Results on video from phone camera

(Per frame estimation, no tracking)

Citation

@inproceedings{xie2022chore,
    title = {CHORE: Contact, Human and Object REconstruction from a single RGB image},
    author = {Xie, Xianghui and Bhatnagar, Bharat Lal and Pons-Moll, Gerard},
    booktitle = {European Conference on Computer Vision ({ECCV})},
    month = {October},
    organization = {{Springer}},
    year = {2022},
}