Do we live in a simulation? Perhaps we should consider the possibility. Replicating real-world observations into a digital twin offers numerous potential benefits. For instance, in autonomous navigation, one could recreate safety-critical scenarios to test an agent's behavior more efficiently and without risking human lives. True-to-life simulations can enable counterfactual analysis, aiding in the interpretability of AI decision-making. Furthermore, they allow us to relive captured experiences for immersive entertainment.
In my research, I develop tools that enable these capabilities through the reconstruction of scene properties, such as geometry and color, and through perception methods like 3D scene segmentation and 3D object detection. This also includes the controllable generation and manipulation of content. To ensure scalability, my focus is particularly on representations that respect the symmetries in the data, such as rotation equivariance, and that can leverage large datasets at scale by minimizing the need for supervision.
Among these topics, in this talk, I will specifically highlight several of my recent papers. These include studies on 3D object detection from single images [1], neural fields for dynamic outdoor scene reconstruction [2], and piecewise equivariant representations [3].
[1] 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features. Chenfeng Xu, Huan Ling, Sanja Fidler, Or Litany. CVPR 2024
[2] Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering, Ido Sobol, Chenfeng Xu, Or Litany. NeurIPS 2024
[3] EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision. Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, Yue Wang. ICLR 2024
Bio: Or Litany is a Senior Research Scientist at the NVIDIA Toronto AI research team, and an Assistant Professor at the Technion where he leads the LIT-Lab specializing in 3D computer vision and generative AI. He is an Azrieli Faculty Fellow and a Taub Fellow. Previously, he conducted postdoctoral research at Stanford University under the guidance of Prof. Leonidas Guibas, and at Meta AI Research (FAIR), where he was hosted by Prof. Jitendra Malik.
