CSE5519 Advances in Computer Vision (Topic C: 2022: Neural Rendering)

Use function to approximate the scene.

Block-NeRF: Scalable Large Scene Neural View Synthesis

Extend NeRF with appearance embeddings and learned pose refinement.

Mask out moving objects.

Do individual NeRFs at each intersection of streets.

Novelty in Block-NeRF

Block size and placement

place one Block-NeRF at each intersection of streets. Covering the intersection itself and 75% of the way until it converges to next intersection.

Appearance embeddings

Label each image with lighting conditions and weather conditions to improve the generalization of the model.

Learned pose refinement and exposure input

Add learnable weights to refine the input pose of the camera and also the exposure of the camera to counter the noise and different camera settings for training data.

Moving object masking

Mask out moving objects which violates the assumption of static scene.

Visibility predection

Additional MLP used to approximate the visibility of a sampled point.

Block-NeRF compositing

Use inverse distance weight for rendering from multiple NeRFs.

Appearance matching

Additional training procedure that freeze model except the target and let the target NeRF train to fit the points on the intersection of target model and other freezed models.

Tip

This paper shows a new way to scale up the NeRF model by using multiple NeRFs to approximate the scene and effectively produce high-quality consistent view synthesis. The appearance embedding shows the role of additional information to improve the generalization of the model and I’m impressed by the rendering result over different lighting conditions and weather conditions.

However, as far as I know, we do have a lots of models that use unsupervised methods to produce those extra information. I wonder how the training could integrate with ViT or other pre-trained unsupervised models predicting the camera parameters and lighting conditions so that we can train the model over more diverse data instead of the fully labeled data over Waymo vehicles.