fix pools termination in BOP dataset generation

[matteomastrogiuseppe]

See issue (#1084).

Pools are not explicitely closed, and in my case it leads to a progressive filling up of RAM memory.
Manually calling the terminator shouldn't change anything, while at the same time avoiding potential issues.

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Matteo Mastrogiuseppe.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email email@example.com
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cornerfarmer]

BlenderProcHelper result for run 574:

Done with: [amass_human_poses_python 49s, auto_shading_python 9s, basic_python 13s, blenderkit_python 18s, bop_object_on_surface_sampling_python 108s, bop_object_physics_positioning_python 148s, bop_object_pose_sampling_python 75s, bop_scene_replication_python 143s, camera_depth_of_field_python 21s, camera_object_pose_python 23s, camera_random_trajectories_python 49s, camera_sampling_python 19s, coco_annotations_python 13s, diffuse_color_image_python 18s, dust_python 29s, entity_displacement_modifier_python 17s, entity_manipulation_python 10s, front_3d_object_sampling_python 85s, front_3d_python 100s, front_3d_with_improved_mat_python 107s, git_log 1s, haven_python 25s, ikea_python 19s, kinect_azure_noise_python 14s, lens_distortion_python 15s, lens_distortion_python 12s, light_sampling_python 12s, material_manipulation_python 10s, material_randomizer_python 13s, matterport3d_python 69s, motion_blur_rolling_shutter_python 28s, motion_blur_rolling_shutter_python 20s, multi_render_python 175s, nocs_python 18s, object_pose_sampling_python 10s, on_surface_object_sampling_python 10s, optical_flow_python 26s, physics_convex_decomposition_python 144s, physics_positioning_python 23s, pix3d_python 17s, point_clouds_python 26s, random_backgrounds_python 17s, random_room_constructor_python 83s, replica_python 342s, scenenet_python 52s, scenenet_with_cctextures_python 65s, semantic_segmentation_python 12s, shapenet_python 19s, shapenet_with_scenenet_python 51s, shapenet_with_suncg_python 67s, spotlight_python 20s, stereo_matching_python 43s, stereo_matching_with_projector_python 139s, suncg_basic_python 33s, suncg_with_cam_sampling_python 53s, suncg_with_improved_mat_python 65s, suncg_with_object_replacer_python 56s, tests_python 57s, urdf_loading_and_manipulation_python 46s]
Done with 59 of 59 tests.
There were errors: ❌
Unit tests output:

${INTERNAL_NODES_DISC_SPACE}/Dominik_Winkelbauer/blender/blender-3.5.1-linux-x64/3.5/python/lib/python3.10/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
${INTERNAL_NODES_DISC_SPACE}/Dominik_Winkelbauer/blender/blender-3.5.1-linux-x64/3.5/python/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
${INTERNAL_NODES_DISC_SPACE}/Dominik_Winkelbauer/blender/blender-3.5.1-linux-x64/3.5/python/lib/python3.10/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
${INTERNAL_NODES_DISC_SPACE}/Dominik_Winkelbauer/blender/blender-3.5.1-linux-x64/3.5/python/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
................
----------------------------------------------------------------------
Ran 16 tests in 52.169s

OK

urdf_loading_and_manipulation (main_python)

The output of urdf_loading_and_manipulation:main_python:000000_scene_gt_info_json has changed:

Expand

+ root['0'][2]['bbox_visib'][3]: 23
- root['0'][2]['bbox_visib'][3]: 25
+ root['4'][6]['bbox_visib'][3]: 17
- root['4'][6]['bbox_visib'][3]: 16
+ root['9'][2]['bbox_visib'][3]: 13
- root['9'][2]['bbox_visib'][3]: 14

stereo_matching (main_python)

The output of stereo_matching:main_python:disparity has changed: Diff: 0.0252, Pixel diff: 0.0033

Compare image	Newly generated image

The output of stereo_matching:main_python:disparity has changed: Diff: 0.0162, Pixel diff: 0.0046

Compare image	Newly generated image

Pylint output

Expand

Messages

Module blenderproc.python.camera.CameraProjection (blenderproc/python/camera/CameraProjection.py)

Line	Col.	Type	Symbol	ID	Obj	Message
4	0	warning	unused-import	W0611		Unused create_primitive imported from blenderproc.python.types.MeshObjectUtility
10	0	convention	ungrouped-imports	C0412		Imports from package blenderproc are not grouped
29	31	convention	trailing-whitespace	C0303		Trailing whitespace
35	0	convention	trailing-whitespace	C0303		Trailing whitespace
41	0	convention	line-too-long	C0301		Line too long (147/120)
85	8	refactor	no-else-return	R1705	depth_at_points_via_raytracing	Unnecessary "else" after "return", remove the "else" and de-indent the code inside it
89	0	convention	trailing-whitespace	C0303		Trailing whitespace
90	0	convention	line-too-long	C0301		Line too long (134/120)
141	0	convention	trailing-whitespace	C0303		Trailing whitespace
159	7	convention	trailing-whitespace	C0303		Trailing whitespace
161	33	convention	trailing-whitespace	C0303		Trailing whitespace
165	0	convention	line-too-long	C0301		Line too long (132/120)
167	0	convention	trailing-newlines	C0305		Trailing newlines

Module blenderproc.python.loader.CCMaterialLoader (blenderproc/python/loader/CCMaterialLoader.py)

Line	Col.	Type	Symbol	ID	Obj	Message
73	0	convention	line-too-long	C0301		Line too long (152/120)

Module blenderproc.python.postprocessing.PostProcessingUtility (blenderproc/python/postprocessing/PostProcessingUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
15	0	convention	line-too-long	C0301		Line too long (137/120)

Module blenderproc.python.types.EntityUtility (blenderproc/python/types/EntityUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
168	11	convention	unidiomatic-typecheck	C0123	Entity.duplicate	Use isinstance() rather than type() for a typecheck.

Module blenderproc.python.types.LightUtility (blenderproc/python/types/LightUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
125	12	refactor	consider-using-augmented-assign	R6104	Light.setup_as_projector	Use '/=' to do an augmented assign directly
216	0	convention	trailing-whitespace	C0303		Trailing whitespace

Module blenderproc.python.types.MeshObjectUtility (blenderproc/python/types/MeshObjectUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
556	0	convention	line-too-long	C0301		Line too long (130/120)
567	7	convention	trailing-whitespace	C0303		Trailing whitespace
575	8	warning	unused-variable	W0612	create_from_point_cloud	Unused variable 'p'
603	0	convention	trailing-whitespace	C0303		Trailing whitespace
678	28	convention	trailing-whitespace	C0303		Trailing whitespace

Module blenderproc.python.utility.InstallUtility (blenderproc/python/utility/InstallUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
129	0	convention	trailing-whitespace	C0303		Trailing whitespace

Module blenderproc.python.writer.BopWriterUtility (blenderproc/python/writer/BopWriterUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
9	0	convention	wrong-import-order	C0411		standard import "from typing import List, Optional, Dict, Tuple" should be placed before "import trimesh"
10	0	convention	wrong-import-order	C0411		standard import "import warnings" should be placed before "import trimesh"
11	0	convention	wrong-import-order	C0411		standard import "import datetime" should be placed before "import trimesh"
18	0	convention	wrong-import-order	C0411		standard import "import sys" should be placed before "import trimesh"
62	0	convention	line-too-long	C0301		Line too long (131/120)
79	8	warning	redefined-outer-name	W0621	write_bop	Redefining name 'dataset_objects' from outer scope (line 536)
172	44	warning	protected-access	W0212	write_bop	Access to a protected member _pyrender_init of a client class
172	15	refactor	consider-using-with	R1732	write_bop	Consider using 'with' for resource-allocating operations
176	0	convention	trailing-whitespace	C0303		Trailing whitespace
182	0	convention	trailing-whitespace	C0303		Trailing whitespace
305	21	warning	redefined-outer-name	W0621	_BopWriterUtility.get_frame_gt	Redefining name 'dataset_objects' from outer scope (line 536)
392	38	warning	redefined-outer-name	W0621	_BopWriterUtility.write_frames	Redefining name 'dataset_objects' from outer scope (line 536)
518	0	convention	trailing-whitespace	C0303		Trailing whitespace
530	78	convention	trailing-whitespace	C0303		Trailing whitespace
534	8	warning	global-variable-undefined	W0601	_BopWriterUtility._pyrender_init	Global variable 'renderer' undefined at the module level
534	8	warning	global-variable-undefined	W0601	_BopWriterUtility._pyrender_init	Global variable 'renderer_large' undefined at the module level
534	8	warning	global-variable-undefined	W0601	_BopWriterUtility._pyrender_init	Global variable 'dataset_objects' undefined at the module level
552	0	convention	line-too-long	C0301		Line too long (173/120)
567	78	convention	trailing-whitespace	C0303		Trailing whitespace
573	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_masks_iteration	Using global for 'renderer' but no assignment is done
573	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_masks_iteration	Using global for 'dataset_objects' but no assignment is done
580	0	convention	trailing-whitespace	C0303		Trailing whitespace
670	0	convention	line-too-long	C0301		Line too long (160/120)
671	0	convention	trailing-whitespace	C0303		Trailing whitespace
672	0	convention	trailing-whitespace	C0303		Trailing whitespace
675	0	convention	line-too-long	C0301		Line too long (196/120)
688	11	convention	trailing-whitespace	C0303		Trailing whitespace
690	78	convention	trailing-whitespace	C0303		Trailing whitespace
696	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_info_iteration	Using global for 'renderer_large' but no assignment is done
696	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_info_iteration	Using global for 'dataset_objects' but no assignment is done
696	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_info_iteration	Using global for 'renderer' but no assignment is done
700	11	warning	protected-access	W0212	_BopWriterUtility._calc_gt_info_iteration	Access to a protected member _renderer of a client class
701	12	warning	protected-access	W0212	_BopWriterUtility._calc_gt_info_iteration	Access to a protected member _renderer of a client class
702	12	warning	protected-access	W0212	_BopWriterUtility._calc_gt_info_iteration	Access to a protected member _renderer of a client class
709	0	convention	trailing-whitespace	C0303		Trailing whitespace
837	0	convention	line-too-long	C0301		Line too long (202/120)
838	0	convention	trailing-whitespace	C0303		Trailing whitespace
846	44	warning	redefined-outer-name	W0621	_BopWriterUtility.calc_gt_coco	Redefining name 'dataset_objects' from outer scope (line 536)

Module blenderproc.scripts.quickstart (blenderproc/scripts/quickstart.py)

Line	Col.	Type	Symbol	ID	Obj	Message
1	0	convention	missing-module-docstring	C0114		Missing module docstring
11	-1	warning	pointless-string-statement	W0105		String statement has no effect
13	0	convention	wrong-import-position	C0413		Import "import numpy as np" should be placed at the top of the module

Metrics

Count per types

Name	Count
convention	43
refactor	3
warning	19

Count per messages

Name	Count
trailing-whitespace	23
line-too-long	11
trailing-newlines	1
no-else-return	1
ungrouped-imports	1
unused-import	1
consider-using-augmented-assign	1
unused-variable	1
unidiomatic-typecheck	1
redefined-outer-name	4
protected-access	4
consider-using-with	1
global-variable-undefined	3
global-variable-not-assigned	5
wrong-import-order	4
missing-module-docstring	1
pointless-string-statement	1
wrong-import-position	1

Count per modules

Name	Count
blenderproc.python.camera.CameraProjection	13
blenderproc.python.loader.CCMaterialLoader	1
blenderproc.python.postprocessing.PostProcessingUtility	1
blenderproc.python.types.LightUtility	2
blenderproc.python.types.MeshObjectUtility	5
blenderproc.python.types.EntityUtility	1
blenderproc.python.utility.InstallUtility	1
blenderproc.python.writer.BopWriterUtility	38
blenderproc.scripts.quickstart	3

Count per path

Name	Count
blenderproc/python/camera/CameraProjection.py	13
blenderproc/python/loader/CCMaterialLoader.py	1
blenderproc/python/postprocessing/PostProcessingUtility.py	1
blenderproc/python/types/LightUtility.py	2
blenderproc/python/types/MeshObjectUtility.py	5
blenderproc/python/types/EntityUtility.py	1
blenderproc/python/utility/InstallUtility.py	1
blenderproc/python/writer/BopWriterUtility.py	38
blenderproc/scripts/quickstart.py	3

***The tests are done.*** Check the log:

Last update: 04/08/2024, 21:44:13

[cornerfarmer]

Thanks a lot for the bug report and PR! Is it possible for you to sign the CLA? This is necessary, so I can merge this.

[cla-bot]

For contributing to BlenderProc you need to sign our Contributor License Agreement. As an individual please sign CLA_individuals.pdf, as a company please sign CLA_entities.pdf and send it to blenderproc@dlr.de

[matteomastrogiuseppe]

Hey @cornerfarmer, which kind of CLA should I sign? For individuals or for companies? My main email is associated with the Italian Institute of Technology, where I'm working rn. Many thanks!

[cornerfarmer]

Hey @matteomastrogiuseppe, I think it makes more sense to use the one for individuals, you can also use your work email/address there. Thanks a lot!

[cornerfarmer]

Thank you again for the PR!

[cornerfarmer]

Apparently github got confused in the merging process. So this PR is actually merged not closed, the changes are in the main branch.