Wang, TY;
Ritschel, T;
Mitra, NJ;
(2018)
Joint Material and Illumination Estimation from Photo Sets in the Wild.
In:
2018 International Conference on 3D Vision (3DV).
(pp. pp. 22-31).
IEEE
Preview |
Text
1710.08313v2.pdf - Accepted Version Download (9MB) | Preview |
Abstract
Faithful manipulation of shape, material, and illumination in 2D Internet images would greatly benefit from a reliable factorization of appearance into material (i.e. diffuse and specular) and illumination (i.e. environment maps). On the one hand, current methods that produce very high fidelity results, typically require controlled settings, expensive devices, or significant manual effort. To the other hand, methods that are automatic and work on 'in the wild' Internet images, often extract only low-frequency lighting or diffuse materials. In this work, we propose to make use of a set of photographs in order to jointly estimate the non-diffuse materials and sharp lighting in an uncontrolled setting. Our key observation is that seeing multiple instances of the same material under different illumination (i.e. environment), and different materials under the same illumination provide valuable constraints that can be exploited to yield a high-quality solution (i.e. specular materials and environment illumination) for all the observed materials and environments. Similar constraints also arise when observing multiple materials in a single environment, or a single material across multiple environments. Technically, we enable this by a novel scalable formulation using parametric mixture models that allows for simultaneous estimation of all materials and illumination directly from a set of (uncontrolled) Internet images. The core of this approach is an optimization procedure that uses two neural networks that are trained on synthetic images to predict good gradients in parametric space given observation of reflected light. We evaluate our method on a range of synthetic and real examples to generate high-quality estimates, qualitatively compare our results against state-of-the-art alternatives via a user study, and demonstrate photo-consistent image manipulation that is otherwise very challenging to achieve.
| Type: | Proceedings paper |
|---|---|
| Title: | Joint Material and Illumination Estimation from Photo Sets in the Wild |
| Event: | 6th International Conference on 3D Vision (3DV), 5-8 September 2018, Verona, Italy |
| Location: | Verona, ITALY |
| Dates: | 05 September 2018 - 08 September 2018 |
| ISBN-13: | 978-1-5386-8425-2 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1109/3DV.2018.00014 |
| Publisher version: | https://doi.org/10.1109/3DV.2018.00014 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Lighting , Geometry , Rendering (computer graphics) , Estimation , Shape , Internet , Optimization |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10066113 |
Archive Staff Only
 |
View Item |
