Bringing Computer Vision to Hidden Objects

A Tutorial on Millimeter-Wave Imaging and Reconstruction of Occluded Scenes

Introduction

Millimeter-wave (mmWave) signals, such as those used in 5G, 6G, and next-generation WiFi, are a unique modality that can travel through many everyday occlusions (e.g. cardboard, fabric, fog, etc), allowing them to sense objects or scenes that are hidden from view. This unique capability has sparked recent interest in the computer vision community and beyond for using these signals to enable novel perception tasks with applications spanning autonomous driving, robotics, shipping and logistics, and more. The goal of this tutorial is to introduce audience members to this modality, and equip them with the knowledge needed to begin research in this area. We will cover both fundamental millimeter-wave imaging concepts, as well as recent, state-of-the-art methods. We will discuss different applications of millimeter-wave sensing, including a deep-dive into two areas: through-occlusion 3D object reconstruction, and all-weather scene understanding. We will additionally cover existing datasets, benchmarks, and tools so that audience members new to this area can begin research in this field. This tutorial is designed to be accessible for an audience with no prior millimeter-wave experience. Topics to be covered include:

  • How millimeter wave signals differ from visible light
  • How millimeter wave signals differ from other through-occlusion modalities (e.g., X-Ray, Ultrasound, etc)
  • Various applications of mmWave sensing, including how they have been used in CV community
  • Classical methods for using mmWave signals to produce a 2D or 3D image
  • Limitations of classical mmWave imaging
  • State-of-the-art methods for using mmWave signals to perform surface normal estimation for 3D object reconstruction
  • State-of-the-art methods for using mmWave signals for complete scene reconstruction, segmentation, and object detection
  • How researchers can get started in this area, including existing datasets, benchmarks, and tools

See the following videos for a preview on what mmWave signals are capable of!

Timeline

Time Details
8:00 - 8:15 AM Introduction
8:15 - 9:15 AM Basics of Millimeter-Wave Signals How They Differ From Visible Light and What They Can Do We will start by covering various applications of mmWave sensing, fundamental properties of mmWave signals, classical imaging approaches and limitations of these approaches.
9:15 - 10:00 AM Advanced Through-Occlusion Object Reconstruction Using Surface Normal Estimates for Through-Occlusion 3D Object Reconstruction and Completion Next, we will discuss a recent state-of-the-art mmWave reconstruction method, including how it opens many exciting directions for future work.
10:00 - 10:15 AM Coffee Break
10:15 - 11:00 AM Enabling Visual-Quality Scene Understanding in Poor Visibility We will discuss how mmWave signals can be used to reconstruct lidar-like scenes in poor visibility, and how this can be used to enable safer autonomous vehicles.
11:00 - 11:45 AM How to Begin Millimeter-Wave Research A Tour of Existing Datasets, Tools, and Open Problems We will conclue by discussing how researchers can start research with mmWave signals, discussing existing datasets, benchmarks, and tools they can use to get started. This will include a hands-on demonstration for setting up and using this tool. We will also discuss several different open problems in the field that researchers can begin .

Speakers & Organizers

Mingmin Zhao

Assistant Professor

University of Pennsylvania

Laura Dodds

PhD Student

Massachusetts Institute of Technology

Hailan Shanbhag

PhD Student

École Polytechnique Fédérale de Lausanne