train

Deep Polarization Reconstruction with PDAVIS Events

INI

UZH / ETH Zurich

Local Experimental Environment

• Ubuntu 20.04.3 LTS
• NVIDIA GeForce RTX 3080 (10G Memory)
• NVIDIA-SMI = 470.94
• cuda = 11.3
• Python = 3.8.10
• torch-1.11.0+cu113 torchaudio-0.11.0+cu113 torchvision-0.12.0+cu113
• cv2 = 4.5.5
• numpy = 1.22.2
• pandas = 1.4.1
• h5py = 3.6.0

3090/V100 Server Environment Configuration

• Python 3.8.13
• PyTorch 1.12.0a0+8a1a93a
• sudo docker pull nvcr.io/nvidia/pytorch:22.05-py3
• pip install -r requirements.txt
• apt-get update
• apt-get install libgl1

Downloads

• E2PD
  • [ Onedrive ] or [ Baidu Disk, fetch code: e2pd ]
• Models
  • FireNet [ Onedrive ] or [ Baidu Disk, fetch code: ckpt ]
  • E2P [ Onedrive ] or [ Baidu Disk, fetch code: ckpt ]
• Results
  • Synthetic [ Onedrive ] or [ Baidu Disk, fetch code: outs ]
  • Real [ Onedrive ] or [ Baidu Disk, fetch code: outr ]

One Command to Play Back One HDF5 file

• python infer.py real-28.h5

xxx.h5 is the test file in infer folder where the corresponding outputs will be saved.

Using E2PD

Unzip E2PD.zip to ./data/ folder

• txt file location

  • ./data/E2PD/train.txt
  • ./data/E2PD/test.txt

• hdf5 file location

  • ./data/E2PD/synthetic/xxxxxx_iad.h5
  • ./data/E2PD/real/real-xx.h5

Recording Data using PDAVIS

• Using PDAVIS with jAER to record aedat2 files and save them at ./data/new/real/
• The recording should end with the extention .aedat

• Modify my_extract_aps.py to point root variable to your same data folder and jaer variable to your installed jaer folder. Then extract PDAVIS APS frames using jaer script dvs-slice-avi-writer.sh
  • python my_extract_aps.py
• manually extract PDAVIS events from jaer using EventFilter DavisTextOutputWriter; see jaer user guide.
  • Set the options "rewindBeforeRecording", "dvsEvents", and "closeOnRewind". Leave "useSignedPolarity" and "timestampLast" unselected.
  • Click the "StartRecordingAndSaveAs" button and choose output file with -events.txt extension; you can select the ./data/new/real/...timecode.txt file and delete the timecode part.
    • (** Because of bug in the jAER DavisTextOutputWriter before commit 8689ac38de9b795a01ab940b6c1595240e129528 on 21 Apr 2023, it is essential to open the output xxx-events.txt and delete the first n rows as they are duplicated. n will be large number of perhaps 10k lines! **. If you don't do this you will get bogus timestamps that mess up the synchronization between events and frames and the training will have meaningless GT target.) If you are running jAER from compiled fresh pull after the commit above, you should not encounter this bug.
    • (** To find how many rows are duplicated, we can check the timestamp of the first frame in xxx-timecode.txt and then use this timestamp to find where the timestamp discontinuity appears in xxx-events.txt **)
    • 
• Modify my_video2frame.py as above to point to correct data folder, and then convert avi to frames with
  • python my_video2frame.py
• Use my_merge_h5.py to merge events, frame, frame_idx, frame_ts, intensity, aolp, dolp into one hdf5 file for the use of DNN with the below command. (It will open a directory chooser for you to select the folder where you are working.)
  • python my_merge_h5.py
• create train/test list txt with my_create_real_list.py
  • python my_create_real_list.py

Now you should have the new list real.txt of training files to split over the existing lists train.txt and test.txt

Training input hdf5 files

The HDF5 (.h5) files should have the following contents

Train

• sh my_train.sh

Steps are illustrated below.

1. The main config is e2p.json
2. It points to train.txt which lists the training input files.
3. These files must be at the locations listed in train.txt

Notes on training

Most recent model checkpoints are all from the new architecture i.e. the angular outputs are 2-dim sin and cos to generate smooth aolp predictions that cross the 2pi cut smoothly.

The most important things about the configuration are probably the loss.
For DoLP it is mse loss, for aolp it is masked aolp mse loss. Masked means only counting loss on gt_dolp > 0.35. We found dolp is more important than aolp in training. so you should set a larger weight for the dolp loss. The model picks up a better dolp prediction and then the aolp prediction is meaningful and smooth.

Using tensorboard to visualize training

• Last two rows are predictions and ground truth.
• Left is intensity, middle is aolp and right is dolp,
• For dolp yellow means high degree. black/dark red is low degree.
• For aolp, 0 means -pi/2, 1 means pi/2, 0 angle is horizontally polarized light.

Test

• test e2p model
  • python my_test.py
• test firenet model
  • python my_test_firenet.py
• test e2vid model
  • python my_test_e2vid.py
• align e2p and firenet for visual comparison
  • python align3_images.py
• split the visual comparison results
  • python split_visual3.py
• output location
  • ./output_synthetic/e2p
  • ./output_real/e2p

Evaluation

• obtain the ground truth from h5 file
  • python utils/h5gt2iad.py
• evaluation
  • python eval.py
• output location
  • ./txt_eval/e2p.txt

Dataset Visualization

Process a list of files set at start of h5gt2iad.py in line: test_list_txt = 'train/data/movingcam/test_real2.txt'

• python utils/h5gt2iad.py h5gt2iad.py

Turn these into movies with visualize_pvideo.py. Edit the line near start to point to the correct folder:

• python visualize_pvideo.py