Spectral Reconstruction with Uncertainty Quantification via
Differentiable Rendering and Null-Space Sampling
Yale University
ACM SIGGRAPH Asia 2025
Summary
Abstract
Spectral information plays a crucial role in many domains, including remote sensing, cultural heritage analysis, food inspection, and material appearance modeling. Spectral measurements, such as hyperspectral imaging, provide a powerful means of acquiring this information but often require expensive equipment and time-consuming capture procedures.
We propose a new method for recovering spectral information from multispectral images using differentiable rendering, which naturally incorporates 3D geometry and light transport. However, the inverse problem is ill-posed: conventional pipelines produce a single spectrum that may differ significantly from the ground truth. To address this ambiguity, we introduce a spectral upsampling framework based on null-space sampling, which generates multiple candidate spectra consistent with the input multi-band image. This enables uncertainty quantification across wavelengths and informs the design of additional measurements to improve reconstruction. We also demonstrate how to incorporate interreflections into the algorithm to enhance reconstruction accuracy.
We validate our method on synthetic scenes using real-world spectral data and RGB renderings, and demonstrate its effectiveness in physical experiments. Our approach not only avoids the cost and complexity of hyperspectral imaging, but also significantly accelerates the reconstruction process compared to brute-force methods that treat each wavelength independently. Moreover, it supports spectral material authoring by generating diverse, physically plausible spectra from a single RGB input, enabling greater flexibility and artistic control in spectral rendering.
We propose a new method for recovering spectral information from multispectral images using differentiable rendering, which naturally incorporates 3D geometry and light transport. However, the inverse problem is ill-posed: conventional pipelines produce a single spectrum that may differ significantly from the ground truth. To address this ambiguity, we introduce a spectral upsampling framework based on null-space sampling, which generates multiple candidate spectra consistent with the input multi-band image. This enables uncertainty quantification across wavelengths and informs the design of additional measurements to improve reconstruction. We also demonstrate how to incorporate interreflections into the algorithm to enhance reconstruction accuracy.
We validate our method on synthetic scenes using real-world spectral data and RGB renderings, and demonstrate its effectiveness in physical experiments. Our approach not only avoids the cost and complexity of hyperspectral imaging, but also significantly accelerates the reconstruction process compared to brute-force methods that treat each wavelength independently. Moreover, it supports spectral material authoring by generating diverse, physically plausible spectra from a single RGB input, enabling greater flexibility and artistic control in spectral rendering.
Acknowledgement
We would like to thank Nadia Zikiou for helping with measuring the ground truth spectra for the physical experiment. This work was supported by the National Science Foundation under grant 2303328.