Overview

Single-photon cameras are becoming increasingly popular in time-of-flight 3D imaging because they can time-tag individual photons with extreme resolution. However, their performance is susceptible to hardware limitations, such as system bandwidth, maximum laser power, sensor data rates, and in-sensor memory and compute resources. Compressive histograms were recently introduced as a solution to the challenge of data rates through an online in-sensor compression of photon timestamp data. Although compressive histograms work within limited in-sensor memory and computational resources, they underperform when subjected to real-world illumination hardware constraints. To address this, we present a constrained optimization approach for designing practical coding functions for compressive single-photon 3D imaging. We jointly optimize an illumination and coding matrix (i.e., the coding functions) that adheres to hardware constraints. We show through extensive simulations that our coding functions consistently outperform traditional coding designs under both bandwidth and peak power constraints. This advantage is particularly pronounced in systems constrained by peak power. Finally, we show that our approach adapts to arbitrary parameterized impulse responses by evaluating it on a real-world system with a non-ideal impulse response function.

Optimized Coding Functions

We propose a constrained optimization to find an illumination function and a coding matrix for compressive single-photon 3D cameras. We solve the constrained optimization problem via gradient descent by constructing a neural network. We optimize under the following hardware illumination constraints: peak power, bandwidth and impulse response function irregularities. Under finite bandwidth and infinite peak power, the optimal illumination converges to a pulsed illumination. Under conditions of finite bandwidth and peak power, we observe that the optimal illumination learns to redistribute its energy. Instead, it converges to a symmetric waveform with multiple peaks.

Optimized coding functions under finite bandwidth and infinite peak power.

Optimized coding functions under finite bandwidth and infinite peak power.

Is the Gaussian Pulse Optimal for Peak Power-limited Single-Photon 3D Cameras?

One way to constrain a Gaussian pulse in peak power–limited systems is to clip its amplitude at the peak power limit. While this enforces the constraint, it also reduces the total transmitted energy. Alternatively, the pulse can be widened to preserve total energy, but this comes at the cost of lower temporal resolution. We show that both clipped and widened Gaussian implementations are sub-optimal, even when evaluated on the non-compressed, full-resolution histogram (FRH). In contrast, our proposed hardware-aware coding functions remain robust under low peak-power conditions and achieve higher or comparable performance, despite providing over 170× compression relative to the full-resolution histograms.

Depth map and error plots for peak power-limited systems.

Adapts to Arbitrary Parameterized IRF

We show that our coding functions can adapt to arbitrarily parameterized impulse response functions, including those affected by irregularities. To test this, we evaluated our coding functions on real-world hardware using a scanning-based single-photon 3D camera. Our results demonstrate that the proposed codes outperform traditional compressive histograms.

3D reconstruction visualization using real-world data.

Citation

Acknowledgements

The website template was borrowed from Michaël Gharbi.