DAOS-16355 client: pydaos.torch module (#15475) #15536
Conversation
|
Ticket title is 'pydaos.torch modules' |
mjmac
changed the title
|
Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15536/1/execution/node/379/log |
|
Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15536/1/execution/node/387/log |
|
Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15536/1/execution/node/385/log |
|
Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15536/1/execution/node/295/log |
|
Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15536/1/execution/node/273/log |
|
Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15536/1/execution/node/317/log |
This patch marks all pool and container handles as if they were created with g2l in the child processes after fork. It prevents misinteractions if one of the child processes closes the handle. The marking is done by iterating through all the pool and container handles which was not supported by the hhash code. This patch also: - adds support for fork to pydaos. - introduces daos_reinit() to be called after fork. - fixes IL to set the atfork callback when no extra eq are used. - remove support for creating an event queue for each pydaos put/get operation. This makes the global event queue the only option. This should probably be moved to a per-thread eq in the future. Signed-off-by: Johann Lombardi <johann.lombardi@gmail.com>
Disable call to pthread_atfork and daos_reinit() in pydaos until DAOS-16637 is understood. Signed-off-by: Johann Lombardi <johann.lombardi@gmail.com>
mjmac
force-pushed
the
mjmac/DAOS-16355-google-2.6
branch
from
117c94f to
e50b3c3
Compare
Modifications to utils/node_local_test.py to better control which tests are run and which aren't. * Added ability to specify multiple --test arguments * Added ability to exclude specific tests via --exclude-test arg * Added ability to "parameterize" test names to specify particular variants * Added ability to exclude previous unnamed tests POSIX tests were run with two variants: cached and uncached. Test names can now optionally include "_caching_on" and "caching_off" suffices to control which version to run or not run. Names without suffices are automatically expanded. Note that previously some tests used the the "_with_caching" suffix, so this will change some reported test names. Use of the needs_dfuse_with_opt and needs_dfuse decorators made interposing on test lists a bit difficult; needs_dfuse_with_opt now keeps track of all tests it and needs_dfuse interpose on and which (caching) variants each test uses; at runtime, a list of test exclusions is checked to see which and how many variants actually need to be run. Manually tested with a few variations of command line invocations. Signed-off-by: Nicholas Murphy <ncmurphy@google.com>
Introducing pydaos.torch module that allows use DAOS POSIX containers as a datasource for pytorch framework in form of pydaos.torch.Dataset and pydaos.torch.IterableDataset classes. Signed-off-by: Denis Barakhtanov <dbarahtanov@enakta.com>
mjmac
force-pushed
the
mjmac/DAOS-16355-google-2.6
branch
from
e50b3c3 to
178c4b9
Compare
|
@jolivier23: This needs to be force landed due to the NLT on ubuntu issue. |
Introducing pydaos.torch module that allows use DAOS POSIX containers
as a datasource for pytorch framework in form of pydaos.torch.Dataset and
pydaos.torch.IterableDataset classes.
Signed-off-by: Denis Barakhtanov dbarahtanov@enakta.com