This repository contains an expansion build over the CVPR 2022 paper: Dense Depth Priors for Neural Radiance Fields from Sparse Input Views.
Original paper resources: Arxiv | Video | Project Page
Our expansion resources: Presentation | Report
conda create —name nerfacc python=3.10
conda activate nerfacc
conda install nvidia/label/cuda-11.8.0::cuda nvidia/label/cuda-11.8.0::cuda-toolkit ninja -y
pip3 install -r requirements.txt
pip3 install nerfacc -f https://nerfacc-bucket.s3.us-west-2.amazonaws.com/whl/torch-2.0.0_cu118.html
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
# This may be useful in case of problems with gcc
# conda install -c conda-forge gcc gxx gcc_linux-64"<11.0.0" libgcc"<11.0.0" gxx_linux-64 -y
# This might be useful if some cuda files can not be found e.g. cudart
# ln -s ~/anaconda3/envs/<py310>/lib ~/anaconda3/envs/<py310>/lib64
Default cuda version can be changed accordingly, but requires modification in the requirements.txt file + probably change of pytorch version.
Download a scene from the ScanNet++ dataset, download the desired scene and undistort the images.
Run the SuperPoint keypoint detector and the SuperGlue feature matching. Then run COLMAP bundle adjustment step on all RGB images of ScanNet++.
For this, the RGB files need to be isolated from the other scene data, f.ex. create a temporary directory tmp
and copy each <scene>/color/<rgb_filename>
to tmp/<scene>/color/<rgb_filename>
.
Then run:
colmap feature_extractor --database_path scannet_sift_database.db --image_path tmp
When working with different relative paths or filenames, the database reading in scannet_dataset.py
needs to be adapted accordingly.
Run the notebook depth_alignment\generate_config.ipynb
to generate the config.json
file needed to relate COLMAP scale to metric scale.
Run the Google Colab notebook depth_estimation\estimate_depth_marigold.ipynb
to obtain monocular affine-invariant depth predictions for all the images in the scene.
You can skip the scene preparation and directly download the ScanNet++ scene. To prepare a scene and render sparse depth maps from COLMAP sparse reconstructions, run:
cd preprocessing
mkdir build
cd build
cmake ..
make -j
./extract_scannet_scene <path to scene> <path to ScanNet>
The scene directory must contain the following:
train.csv
: list of training views from the ScanNet scenetest.csv
: list of test views from the ScanNet sceneconfig.json
: parameters for the scene:name
: name of the scenemax_depth
: maximal depth value in the scene, larger values are invalidateddist2m
: scaling factor that scales the sparse reconstruction to metersrgb_only
: write RGB only, f.ex. to get input for COLMAP
colmap
: directory containing 2 sparse reconstruction:sparse
: reconstruction run on train and test images together to determine the camera posessparse_train
, reconstruction run on train images alone to determine the sparse depth maps.
Please check the provided scene as an example.
To obtain metric depth prior run the notebook depth_alignment\align_depth_map_MG_scannetpp.ipynb
to transform the affine-invariant depth prior to metric scale.
#for original paper
python3 run_nerf.py train --scene_id <scene, e.g. scene0710_00> --data_dir <directory containing the scenes> --depth_prior_network_path <path to depth prior checkpoint> --ckpt_dir <path to write checkpoints>
# for nerfacc
python3 run_nerfacc.py train \
--scene_id <scene> --data_dir <directory containing the scenes> \
--depth_completion <resnet|marigold> --depth_prior_network_path <if resnet> \
--ckpt_dir <path to write checkpoints> --max_num_rays 1024 --occ_resolution 128 \
--render_step_size 1e-3 --occ_num_levels 6 --model_type ngp --netwidth 128 \
--log2_hashmap_size 22 --N_training_steps 15000 --lrate 1e-2 --ngp_max_resolution 524288 \
--i_print 1000 --i_img 3000 # frequence of printing and evaluation
Checkpoints are written into a subdirectory of the provided checkpoint directory. The subdirectory is named by the training start time in the format jjjjmmdd_hhmmss
, which also serves as experiment name in the following.
We thank Dense Depth Priors NeRF which we use as a baseline, additionally we thank nerf-pytorch and CSPN, from which the original repository borrows code.