In this video, we present a novel point-based, differentiable neural rendering pipeline.
From a set of input views and an initial 3D reconstruction, our system is able to synthesize
novel views and refine the scene's parameters.
Our pipeline consists of three major stages.
Rasterization, neural rendering, and tone mapping.
The rasterizer projects the point cloud to a given viewpoint and blends the assigned
neural descriptors into a multiscale output image.
Points that face away from the camera or fail a fuzzy depth test are discarded.
A deep neural network converts the multiscale input to a single HDR image.
The main tasks of the network are hole filling and shading.
The intermediate HDR image is passed to a differentiable tone mapper, which generates
the final LDR output.
During training, the physical parameters of the tone mapper are estimated.
For example, the camera response function and the per image exposure value.
In this experiment, we fit our pipeline to a data set of 688 images and 73 million points.
After a few iterations, we allow our system to also optimize the camera pose of each frame.
This further improves the sharpness of the rendering because the frames are now pixel
perfectly aligned.
Note here that our system is the first inverse rendering approach that can directly process
images of fisheye cameras.
We therefore can eliminate a potential lossy undistortion operation.
Here you can see a virtual flight over the playground scene from the tanks and temples
dataset.
This was generated by first fitting our pipeline to the dataset consisting of around 300 images
and 8 million points.
After that, we can synthesize novel views from arbitrary camera locations.
The same technique has been used to generate this video of a tank.
Here you can see the neural rendering of a rebuilt Roman vessel.
As you can see in the bottom right, the input images were captured with different exposure
settings.
Our system is still able to generate consistent novel views due to a physically based tone
mapper.
This also allows us to change the learned parameters of the tone mapper at inference
time.
Here you can see how we change the exposure time at a fixed camera location.
Thanks for watching.