Depth Map Error

[jeffreyianwilson]

I am getting failure at the depth map reconstruction phase with both Meshroom 2018.1 and 2019.1 There must be a driver issue (430.64 now) because both worked a few months ago, or a Windows 10 issue perhaps? I have a GTX 2070 which has been installed since then as well but I dont think it is the GPU as I have had the same 1080 Ti at work and it is failing now as well (I am using the Nvidia DCC driver there)

[fabiencastan]

Could you put the log of the DepthMap node with the 2019.1 release to see what graphic card is detected?

[jeffreyianwilson]

It crashed at 9 of 49 chunks.

Program called with the following parameters:

• downscale = 2
• exportIntermediateResults = 0
• imagesFolder = "E:/Work/Projects/03 - Scan Processing/MeshroomCache/PrepareDenseScene/ab69f750f9de54f594ac84428ebf97eb65510f50"
• input = "E:/Work/Projects/03 - Scan Processing/MeshroomCache/StructureFromMotion/0845480a917741095d3a9fe3989e1c0b3d148547/sfm.abc"
• maxViewAngle = 70
• minViewAngle = 2
• nbGPUs = 0
• output = "E:/Work/Projects/03 - Scan Processing/MeshroomCache/DepthMap/56bc0cfa9b13d3346bb1a8ffffcd469b77592e11"
• rangeSize = 3
• rangeStart = 27
• refineGammaC = 15.5
• refineGammaP = 8
• refineMaxTCams = 6
• refineNDepthsToRefine = 31
• refineNSamplesHalf = 150
• refineNiters = 100
• refineSigma = 15
• refineUseTcOrRcPixSize = 0
• refineWSH = 3
• sgmGammaC = 5.5
• sgmGammaP = 8
• sgmMaxTCams = 10
• sgmWSH = 4
• verboseLevel = "info"

[09:02:53.902217][info] CUDA-Enabled GPU.
Device information:
- id: 0
- name: GeForce RTX 2070
- compute capability: 7.5
- total device memory: 8192 MB
- device memory available: 6716 MB
- per-block shared memory: 49152
- warp size: 32
- max threads per block: 1024
- max threads per SM(X): 1024
- max block sizes: {1024,1024,64}
- max grid sizes: {2147483647,65535,65535}
- max 2D array texture: {131072,65536}
- max 3D array texture: {16384,16384,16384}
- max 2D linear texture: {131072,65000,2097120}
- max 2D layered texture: {32768,32768,2048}
- number of SM(x)s: 36
- registers per SM(x): 65536
- registers per block: 65536
- concurrent kernels: yes
- mapping host memory: yes
- unified addressing: yes
- texture alignment: 512 byte
- pitch alignment: 32 byte

[09:02:54.436046][info] Supported CUDA-Enabled GPU detected.
[09:02:54.710960][info] Found 2 image dimension(s):
[09:02:54.710960][info] - [6000x4000]
[09:02:54.710960][info] - [7360x4912]
[09:02:55.296772][info] Overall maximum dimension: [7360x4912]
[09:02:55.296772][info] Create depth maps.
number of CUDA devices: 1
0: GeForce RTX 2070
[09:02:55.296772][info] # GPU devices: 1, # CPU threads: 12
[09:02:55.296772][info] Plane sweeping parameters:
- scale: 2
- step: 6
[09:02:55.296772][info] PlaneSweepingCuda:
- _nImgsInGPUAtTime: 2
- scales: 2
- subPixel: Yes
- varianceWSH: 4

[jeffreyianwilson]

What is a configuration you would recommend which you know works at this time? Is this a Meshroom issue or an AliceVision issue?

[fabiencastan]

The problem is on the AliceVision side, but that's not a problem to have the issue here.
I'm working on a GTX 1080. We have not yet tested on RTX graphic cards, but I don't see any reason that can explain your issue.
Could you test it with a simple dataset?
https://github.com/alicevision/dataset_monstree/tree/master/mini6

[jeffreyianwilson]

The problem is on the AliceVision side, but that's not a problem to have the issue here.
I'm working on a GTX 1080. We have not yet tested on RTX graphic cards, but I don't see any reason that can explain your issue.
Could you test it with a simple dataset?
https://github.com/alicevision/dataset_monstree/tree/master/mini6

Running it now..

[jeffreyianwilson]

I am installing Ubuntu today, I will test Meshroom in it as well.

[jeffreyianwilson]

The problem is on the AliceVision side, but that's not a problem to have the issue here.
I'm working on a GTX 1080. We have not yet tested on RTX graphic cards, but I don't see any reason that can explain your issue.
Could you test it with a simple dataset?
https://github.com/alicevision/dataset_monstree/tree/master/mini6

It successfully processed all of the depth maps without crashing.

[fabiencastan]

Could you go back to your dataset and try to change the downscale to 4?

[jeffreyianwilson]

Could you go back to your dataset and try to change the downscale to 4?

Processing now, so far so good.

[jeffreyianwilson]

It successfully processed downscaled to 4, here is the log for the last chunk

Program called with the following parameters:

• downscale = 4
• exportIntermediateResults = 0
• imagesFolder = "E:/Work/Projects/03 - Scan Processing/MeshroomCache/PrepareDenseScene/ab69f750f9de54f594ac84428ebf97eb65510f50"
• input = "E:/Work/Projects/03 - Scan Processing/MeshroomCache/StructureFromMotion/0845480a917741095d3a9fe3989e1c0b3d148547/sfm.abc"
• maxViewAngle = 70
• minViewAngle = 2
• nbGPUs = 0
• output = "E:/Work/Projects/03 - Scan Processing/MeshroomCache/DepthMap/7421beddb2c144dd3a29c094ba75cf7306ab741e"
• rangeSize = 3
• rangeStart = 147
• refineGammaC = 15.5
• refineGammaP = 8
• refineMaxTCams = 6
• refineNDepthsToRefine = 31
• refineNSamplesHalf = 150
• refineNiters = 100
• refineSigma = 15
• refineUseTcOrRcPixSize = 0
• refineWSH = 3
• sgmGammaC = 5.5
• sgmGammaP = 8
• sgmMaxTCams = 10
• sgmWSH = 4
• verboseLevel = "info"

[11:53:32.841298][info] CUDA-Enabled GPU.
Device information:
- id: 0
- name: GeForce RTX 2070
- compute capability: 7.5
- total device memory: 8192 MB
- device memory available: 6716 MB
- per-block shared memory: 49152
- warp size: 32
- max threads per block: 1024
- max threads per SM(X): 1024
- max block sizes: {1024,1024,64}
- max grid sizes: {2147483647,65535,65535}
- max 2D array texture: {131072,65536}
- max 3D array texture: {16384,16384,16384}
- max 2D linear texture: {131072,65000,2097120}
- max 2D layered texture: {32768,32768,2048}
- number of SM(x)s: 36
- registers per SM(x): 65536
- registers per block: 65536
- concurrent kernels: yes
- mapping host memory: yes
- unified addressing: yes
- texture alignment: 512 byte
- pitch alignment: 32 byte

[11:53:33.287156][info] Supported CUDA-Enabled GPU detected.
[11:53:33.603055][info] Found 2 image dimension(s):
[11:53:33.603055][info] - [6000x4000]
[11:53:33.603055][info] - [7360x4912]
[11:53:34.148879][info] Overall maximum dimension: [7360x4912]
[11:53:34.148879][info] Create depth maps.
number of CUDA devices: 1
0: GeForce RTX 2070
[11:53:34.148879][info] # GPU devices: 1, # CPU threads: 12
[11:53:34.148879][info] Plane sweeping parameters:
- scale: 2
- step: 6
[11:53:34.148879][info] PlaneSweepingCuda:
- _nImgsInGPUAtTime: 2
- scales: 2
- subPixel: Yes
- varianceWSH: 4
CUDA device no 0 for 0
Device 0 memory - used: 1475.025024, free: 6716.975098, total: 8192.000000
Device 0 memory - used: 1827.275024, free: 6364.725098, total: 8192.000000
[11:54:02.049294][info] Estimate depth map, view id: 2120689537
Device 0 memory - used: 1882.400024, free: 6309.600098, total: 8192.000000
total size of volume map in GPU memory: 55.076660
nDepths 1359, nDepthsToSearch 1359
ps_computeSimilarityVolume elapsed time: 22.000000 ms
Device 0 memory - used: 1856.650024, free: 6335.350098, total: 8192.000000
total size of volume map in GPU memory: 29.341797
nDepths 724, nDepthsToSearch 724
ps_computeSimilarityVolume elapsed time: 8.000000 ms
Device 0 memory - used: 1856.650024, free: 6335.350098, total: 8192.000000
total size of volume map in GPU memory: 29.341797
nDepths 724, nDepthsToSearch 724
ps_computeSimilarityVolume elapsed time: 8.000000 ms
Device 0 memory - used: 1863.400024, free: 6328.600098, total: 8192.000000
total size of volume map in GPU memory: 36.069336
nDepths 890, nDepthsToSearch 890
ps_computeSimilarityVolume elapsed time: 10.000000 ms
Device 0 memory - used: 1852.775024, free: 6339.225098, total: 8192.000000
total size of volume map in GPU memory: 25.491699
nDepths 629, nDepthsToSearch 629
ps_computeSimilarityVolume elapsed time: 7.000000 ms
Device 0 memory - used: 1850.275024, free: 6341.725098, total: 8192.000000
total size of volume map in GPU memory: 22.979004
nDepths 567, nDepthsToSearch 567
ps_computeSimilarityVolume elapsed time: 10.000000 ms
Device 0 memory - used: 1869.150024, free: 6322.850098, total: 8192.000000
total size of volume map in GPU memory: 41.864746
nDepths 1033, nDepthsToSearch 1033
ps_computeSimilarityVolume elapsed time: 10.000000 ms
Device 0 memory - used: 1856.650024, free: 6335.350098, total: 8192.000000
total size of volume map in GPU memory: 29.341797
nDepths 724, nDepthsToSearch 724
ps_computeSimilarityVolume elapsed time: 9.000000 ms
Device 0 memory - used: 1856.650024, free: 6335.350098, total: 8192.000000
total size of volume map in GPU memory: 29.341797
nDepths 724, nDepthsToSearch 724
ps_computeSimilarityVolume elapsed time: 9.000000 ms
Device 0 memory - used: 1850.337524, free: 6341.662598, total: 8192.000000
total size of volume map in GPU memory: 23.019531
nDepths 568, nDepthsToSearch 568
ps_computeSimilarityVolume elapsed time: 7.000000 ms
ps_SGMoptimizeSimVolume
ps_updateAggrVolume
ps_aggregatePathVolume
ps_aggregatePathVolume done
ps_updateAggrVolume done
ps_updateAggrVolume
ps_aggregatePathVolume
ps_aggregatePathVolume done
ps_updateAggrVolume done
ps_updateAggrVolume
ps_aggregatePathVolume
ps_aggregatePathVolume done
ps_updateAggrVolume done
ps_updateAggrVolume
ps_aggregatePathVolume
ps_aggregatePathVolume done
ps_updateAggrVolume done
SGM volume gpu elapsed time: 135.000000 ms
ps_SGMoptimizeSimVolume done
[11:54:07.008702][info] Refine depth map, view id: 2120689537
gpu elapsed time: 78.000000 ms
gpu elapsed time: 93.000000 ms
gpu elapsed time: 98.000000 ms
gpu elapsed time: 88.000000 ms
gpu elapsed time: 43.000000 ms
gpu elapsed time: 81.000000 ms
gpu elapsed time: 71.000000 ms
gpu elapsed time: 13.000000 ms
gpu elapsed time: 70.000000 ms
gpu elapsed time: 71.000000 ms
gpu elapsed time: 68.000000 ms
gpu elapsed time: 15.000000 ms
gpu elapsed time: 66.000000 ms
gpu elapsed time: 102.000000 ms
gpu elapsed time: 104.000000 ms
gpu elapsed time: 74.000000 ms
gpu elapsed time: 105.000000 ms
gpu elapsed time: 104.000000 ms
gpu elapsed time: 102.000000 ms
gpu elapsed time: 59.000000 ms
gpu elapsed time: 102.000000 ms
gpu elapsed time: 102.000000 ms
gpu elapsed time: 102.000000 ms
gpu elapsed time: 73.000000 ms
gpu elapsed time: 69.000000 ms
gpu elapsed time: 72.000000 ms
gpu elapsed time: 70.000000 ms
gpu elapsed time: 66.000000 ms
rc: 147, gpu elapsed time: 28.000000 ms
rc: 147, gpu elapsed time: 28.000000 ms
rc: 147, gpu elapsed time: 29.000000 ms
rc: 147, gpu elapsed time: 28.000000 ms
[11:54:34.043861][info] Estimate depth map, view id: 2139453226
Device 0 memory - used: 1877.587524, free: 6314.412598, total: 8192.000000
total size of volume map in GPU memory: 50.253906
nDepths 1240, nDepthsToSearch 1240
ps_computeSimilarityVolume elapsed time: 12.000000 ms
Device 0 memory - used: 1877.587524, free: 6314.412598, total: 8192.000000
total size of volume map in GPU memory: 50.253906
nDepths 1240, nDepthsToSearch 1240
ps_computeSimilarityVolume elapsed time: 13.000000 ms
Device 0 memory - used: 1874.337524, free: 6317.662598, total: 8192.000000
total size of volume map in GPU memory: 47.052246
nDepths 1161, nDepthsToSearch 1161
ps_computeSimilarityVolume elapsed time: 13.000000 ms
Device 0 memory - used: 1877.587524, free: 6314.412598, total: 8192.000000
total size of volume map in GPU memory: 50.253906
nDepths 1240, nDepthsToSearch 1240
ps_computeSimilarityVolume elapsed time: 14.000000 ms
Device 0 memory - used: 1849.337524, free: 6342.662598, total: 8192.000000
total size of volume map in GPU memory: 22.006348
nDepths 543, nDepthsToSearch 543
ps_computeSimilarityVolume elapsed time: 8.000000 ms
Device 0 memory - used: 1848.525024, free: 6343.475098, total: 8192.000000
total size of volume map in GPU memory: 21.236328
nDepths 524, nDepthsToSearch 524
ps_computeSimilarityVolume elapsed time: 7.000000 ms
Device 0 memory - used: 1849.337524, free: 6342.662598, total: 8192.000000
total size of volume map in GPU memory: 22.006348
nDepths 543, nDepthsToSearch 543
ps_computeSimilarityVolume elapsed time: 8.000000 ms
Device 0 memory - used: 1859.150024, free: 6332.850098, total: 8192.000000
total size of volume map in GPU memory: 31.854492
nDepths 786, nDepthsToSearch 786
ps_computeSimilarityVolume elapsed time: 9.000000 ms
Device 0 memory - used: 1849.337524, free: 6342.662598, total: 8192.000000
total size of volume map in GPU memory: 22.006348
nDepths 543, nDepthsToSearch 543
ps_computeSimilarityVolume elapsed time: 7.000000 ms
Device 0 memory - used: 1857.212524, free: 6334.787598, total: 8192.000000
total size of volume map in GPU memory: 29.909180
nDepths 738, nDepthsToSearch 738
ps_computeSimilarityVolume elapsed time: 8.000000 ms
ps_SGMoptimizeSimVolume
ps_updateAggrVolume
ps_aggregatePathVolume
ps_aggregatePathVolume done
ps_updateAggrVolume done
ps_updateAggrVolume
ps_aggregatePathVolume
ps_aggregatePathVolume done
ps_updateAggrVolume done
ps_updateAggrVolume
ps_aggregatePathVolume
ps_aggregatePathVolume done
ps_updateAggrVolume done
ps_updateAggrVolume
ps_aggregatePathVolume
ps_aggregatePathVolume done
ps_updateAggrVolume done
SGM volume gpu elapsed time: 131.000000 ms
ps_SGMoptimizeSimVolume done
[11:54:38.679263][info] Refine depth map, view id: 2139453226
gpu elapsed time: 99.000000 ms
gpu elapsed time: 98.000000 ms
gpu elapsed time: 97.000000 ms
gpu elapsed time: 96.000000 ms
gpu elapsed time: 107.000000 ms
gpu elapsed time: 110.000000 ms
gpu elapsed time: 104.000000 ms
gpu elapsed time: 81.000000 ms
gpu elapsed time: 11.000000 ms
gpu elapsed time: 59.000000 ms
gpu elapsed time: 83.000000 ms
gpu elapsed time: 85.000000 ms
gpu elapsed time: 17.000000 ms
gpu elapsed time: 77.000000 ms
gpu elapsed time: 97.000000 ms
gpu elapsed time: 85.000000 ms
gpu elapsed time: 113.000000 ms
gpu elapsed time: 110.000000 ms
gpu elapsed time: 110.000000 ms
gpu elapsed time: 107.000000 ms
gpu elapsed time: 24.000000 ms
gpu elapsed time: 96.000000 ms
gpu elapsed time: 121.000000 ms
gpu elapsed time: 112.000000 ms
gpu elapsed time: 73.000000 ms
gpu elapsed time: 78.000000 ms
gpu elapsed time: 75.000000 ms
gpu elapsed time: 69.000000 ms
rc: 148, gpu elapsed time: 38.000000 ms
rc: 148, gpu elapsed time: 30.000000 ms
rc: 148, gpu elapsed time: 28.000000 ms
rc: 148, gpu elapsed time: 35.000000 ms
[11:54:45.184754][info] Task done in (s): 72.345000

[skinkie]

Might be related to #409 ?

[capitalPSA]

+1
I'm having a similar error; this could be related to the GPU since I'm using an RTX 2070 with CUDA 10.0 (and CUDA 10.1).
The sample datasets referenced above are failing on my end with a runtime error.

@fabiencastan is there a timeline for testing RTX graphic cards?

[simogasp]

The problem could be fixed by alicevision/AliceVision#671 which has been merged into the current develop of AliceVision. You can try to get the latest AliceVision develop branch as well as the latest develop of meshroom and see if the problem is fixed. There are no binaries yet available, they will be very soon with the 2019.2 release which is on the way in the next few days.

[skinkie]

@simogasp I have an open ticket at alicevision/AliceVision#643 and the build of @natowi on Windows doesn't have working Cuda for us. So I would suggest to give full build instructions so a 2019.3 could be better evaluated.