CSE5519 Advances in Computer Vision (Topic C: 2023: Neural Rendering)
NoPe-NeRF: Optimising neural radiance field with no pose prior
Incorporating undistorted monocular depth priors.
THese priors are generated by correcting scale and shift parameters during training, with which we are then able to constrain the relative posed between consecutive frames.
Novelty in Methods
By noticing that the test views are sampled from video sequences, are close to the training views. We fit a bezier curve from the estimated training poses and sample interpolated posed for each method to render novel view videos.
Novelty in Implementations
- Replacing ReLU activation function with Softplus
- Sample 128 points along each ray uniformly with noise.
Loss function construction
- Distortion parameters impact on pose accuracy
- Inter-frame consistency loss impacts pose accuracy
- NeRF Losses impact on pose accuracy
This paper presents a new method integrating the monocular depth estimation with Neural Rendering. Note that the model is trained on video sequences with inter-frame losses. The author even fits a Bezier curve from the estimated training poses and samples interpolated poses for novel view videos. Is this unfair for other models that don’t assume the trajectory of predicted views is continuous?
How does the model accurately perform for arbitrary poses instead of selecting from poses in video sequences?