DAOS-7203 control: Add histogram support to Prometheus exporter
GitHub Actions / Functional on EL 8.8 Test Results (old)
failed
Feb 21, 2024 in 0s
1 errors, 4 skipped, 130 pass in 1h 34m 45s
135 tests 130 ✅ 1h 34m 45s ⏱️
41 suites 4 💤
41 files 0 ❌ 1 🔥
Results for commit 427bb06.
Annotations
Check failure on line 0 in FTEST_control.TestWithTelemetryBasic
github-actions / Functional on EL 8.8 Test Results (old)
1-./control/dmg_telemetry_basic.py:TestWithTelemetryBasic.test_telemetry_list;run-container-hosts-pool-server_config-engines-0-storage-0-test-timeouts-dadb (FTEST_control.TestWithTelemetryBasic) with error
Functional on EL 8.8/control/dmg_telemetry_basic.py/results.xml [took 1m 33s]
Raw output
Test interrupted by SIGTERM Runner error occurred: Timeout reached Original status: ERROR {'name': '1-./control/dmg_telemetry_basic.py:TestWithTelemetryBasic.test_telemetry_list;run-container-hosts-pool-server_config-engines-0-storage-0-test-timeouts-dadb', 'logdir': '/var/tmp/ftest/avocado/job-results/job-2024-02-21T22.00-952da71/test-results/1-._control_dmg_telemetry_basic.py_TestWithTelemetryBasic.test_telemetry_list_run-container-hosts-pool-server_config-engines-0-storage-0-test-timeouts-dadb', 'logfile': '/var/tmp/ftest/avocado/job-results/job-2024-02-21T22.00-952da71/test-results/1-._control_dmg_telemetry_basic.py_TestWithTelemetryBasic.test_telemetry_list_run-container-hosts-pool-server_config-engines-0-storage-0-test-timeouts-dadb/debug.log', 'status': 'ERROR', 'running': False, 'paused': False, 'time_start': 1708552800.8852448, 'time_elapsed': 93.31788730621338, 'time_end': 1708552894.2031322, 'fail_reason': 'Test interrupted by SIGTERM', 'fail_class': 'RuntimeError', 'traceback': 'Traceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/avocado/core/test.py", line 856, in _run_avocado\n raise test_exception\n File "/usr/lib/python3.6/site-packages/avocado/core/test.py", line 762, in _run_avocado\n testMethod()\n File "/usr/lib/daos/TESTING/ftest/control/dmg_telemetry_basic.py", line 84, in test_telemetry_list\n self.verify_telemetry_list()\n File "/usr/lib/daos/TESTING/ftest/util/telemetry_test_base.py", line 82, in verify_telemetry_list\n self.fail("\\n".join(errors))\n File "/usr/lib/daos/TESTING/ftest/util/apricot/apricot/test.py", line 1374, in fail\n self.dump_engines_stacks("Test has failed")\n File "/usr/lib/daos/TESTING/ftest/util/apricot/apricot/test.py", line 1363, in dump_engines_stacks\n dump_engines_stacks(self.hostlist_servers)\n File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 827, in dump_engines_stacks\n result = pcmd(hosts, "; ".join(commands), verbose, timeout, None)\n File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 594, in pcmd\n results = run_pcmd(hosts, command, verbose, timeout, expect_rc)\n File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 449, in run_pcmd\n task = run_task(hosts, command, timeout)\n File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 312, in run_task\n task.run(**kwargs)\n File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 877, in run\n self.resume(timeout)\n File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 831, in resume\n self._resume()\n File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 794, in _resume\n self._run(self.timeout)\n File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 404, in _run\n self._engine.run(timeout)\n File "/usr/lib/python3.6/site-packages/ClusterShell/Engine/Engine.py", line 723, in run\n self.runloop(timeout)\n File "/usr/lib/python3.6/site-packages/ClusterShell/Engine/EPoll.py", line 122, in runloop\n evlist = self.epolling.poll(poll_timeo)\n File "/usr/lib/python3.6/site-packages/avocado/plugins/runner.py", line 77, in sigterm_handler\n raise RuntimeError("Test interrupted by SIGTERM")\nRuntimeError: Test interrupted by SIGTERM\n', 'timeout': 90, 'whiteboard': '', 'phase': 'FINISHED', 'class_name': 'TestWithTelemetryBasic', 'job_logdir': '/var/tmp/ftest/avocado/job-results/job-2024-02-21T22.00-952da71', 'job_unique_id': '952da71e67f9eeb30bc1bd2e53a5eab6e24dd89b', 'params': [('/run/hosts', 'test_servers', 'wolf-101vm[2-3]'), ('/run/hosts', 'test_clients', 'wolf-101vm4'), ('/run/timeouts', 'test_telemetry_list', 60), ('/run/timeouts', 'test_container_telemetry', 230), ('/run/server_config', 'name', 'daos_server'), ('/run/server_config', 'engines_per_host', 1), ('/run/server_config', 'system_ram_reserved', 1), ('/run/server_config/engines/0', 'targets', 4), ('/run/server_config/engines/0', 'nr_xs_helpers', 0), ('/run/server_config/engines/0/storage/0', 'class', 'ram'), ('/run/server_config/engines/0/storage/0', 'scm_mount', '/mnt/daos'), ('/run/pool', 'scm_size', '2G'), ('/run/pool', 'control_method', 'dmg'), ('/run/container', 'control_method', 'daos'), ('/run/test', 'container_qty', 5), ('/run/test', 'open_close_qty', 3)]}
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/avocado/core/test.py", line 856, in _run_avocado
raise test_exception
File "/usr/lib/python3.6/site-packages/avocado/core/test.py", line 762, in _run_avocado
testMethod()
File "/usr/lib/daos/TESTING/ftest/control/dmg_telemetry_basic.py", line 84, in test_telemetry_list
self.verify_telemetry_list()
File "/usr/lib/daos/TESTING/ftest/util/telemetry_test_base.py", line 82, in verify_telemetry_list
self.fail("\n".join(errors))
File "/usr/lib/daos/TESTING/ftest/util/apricot/apricot/test.py", line 1374, in fail
self.dump_engines_stacks("Test has failed")
File "/usr/lib/daos/TESTING/ftest/util/apricot/apricot/test.py", line 1363, in dump_engines_stacks
dump_engines_stacks(self.hostlist_servers)
File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 827, in dump_engines_stacks
result = pcmd(hosts, "; ".join(commands), verbose, timeout, None)
File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 594, in pcmd
results = run_pcmd(hosts, command, verbose, timeout, expect_rc)
File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 449, in run_pcmd
task = run_task(hosts, command, timeout)
File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 312, in run_task
task.run(**kwargs)
File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 877, in run
self.resume(timeout)
File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 831, in resume
self._resume()
File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 794, in _resume
self._run(self.timeout)
File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 404, in _run
self._engine.run(timeout)
File "/usr/lib/python3.6/site-packages/ClusterShell/Engine/Engine.py", line 723, in run
self.runloop(timeout)
File "/usr/lib/python3.6/site-packages/ClusterShell/Engine/EPoll.py", line 122, in runloop
evlist = self.epolling.poll(poll_timeo)
File "/usr/lib/python3.6/site-packages/avocado/plugins/runner.py", line 77, in sigterm_handler
raise RuntimeError("Test interrupted by SIGTERM")
RuntimeError: Test interrupted by SIGTERM
22:00:00 INFO | *** SETUP running on <LinuxDistro: name=rocky, version=8, release=8, arch=x86_64> ***
22:00:00 INFO | Test.random seed = 1818195128947727738
22:00:00 DEBUG| Shared test directory: /mnt/share/tmp.n2PlrdWW35
22:00:00 DEBUG| Common test directory: /var/tmp/daos_testing/test_telemetry_list
22:00:00 DEBUG| PARAMS (key=fault_list, path=/run/faults/*, default=None) => None
22:00:00 DEBUG| PARAMS (key=start_agents_once, path=/run/setup/*, default=True) => True
22:00:00 DEBUG| PARAMS (key=start_servers_once, path=/run/setup/*, default=True) => True
22:00:00 DEBUG| PARAMS (key=server_manager_class, path=/run/setup/*, default=Systemctl) => 'Systemctl'
22:00:00 DEBUG| PARAMS (key=agent_manager_class, path=/run/setup/*, default=Systemctl) => 'Systemctl'
22:00:00 DEBUG| PARAMS (key=start_servers, path=/run/setup/*, default=True) => True
22:00:00 DEBUG| PARAMS (key=start_agents, path=/run/setup/*, default=True) => True
22:00:00 DEBUG| PARAMS (key=slurm_exclude_servers, path=/run/setup/*, default=False) => False
22:00:00 DEBUG| PARAMS (key=name, path=/server_config/, default=daos_server) => 'daos_server'
22:00:00 DEBUG| PARAMS (key=server_config_namespace, path=/run/setup/*, default=None) => None
22:00:00 DEBUG| PARAMS (key=manager_class, path=/, default=Orterun) => 'Orterun'
22:00:00 DEBUG| PARAMS (key=test_servers, path=/run/hosts/*, default=None) => 'wolf-101vm[2-3]'
22:00:00 DEBUG| PARAMS (key=server_partition, path=/run/hosts/*, default=None) => None
22:00:00 DEBUG| PARAMS (key=server_reservation, path=/run/hosts/*, default=None) => None
22:00:00 DEBUG| PARAMS (key=test_clients, path=/run/hosts/*, default=None) => 'wolf-101vm4'
22:00:00 DEBUG| PARAMS (key=client_partition, path=/run/hosts/*, default=None) => None
22:00:00 DEBUG| PARAMS (key=client_reservation, path=/run/hosts/*, default=None) => None
22:00:00 DEBUG| PARAMS (key=access_points_qty, path=/run/setup/*, default=1) => 1
22:00:00 DEBUG| PARAMS (key=access_points, path=/run/setup/*, default=wolf-101vm2) => <ClusterShell.NodeSet.NodeSet object at 0x7f96b07d4b70>
22:00:00 DEBUG| PARAMS (key=access_points_suffix, path=/run/setup/*, default=None) => None
22:00:00 DEBUG| Writing hostfile: /var/tmp/avocado_h2o1e_rp/avocado_job_ae8idk9u/1-._control_dmg_telemetry_basic.py_TestWithTelemetryBasic.test_telemetry_list_run-container-hosts-pool-server_config-engines-0-storage-0-test-timeouts-dadb/hostfile_a1gqwkkn (hosts=wolf-101vm4, slots=1)
22:00:00 INFO | ----------------------------------------------------------------------------------------------------
22:00:00 INFO | --- HOST INFORMATION ---
22:00:00 INFO | servers: wolf-101vm[2-3]
22:00:00 INFO | clients: wolf-101vm4
22:00:00 INFO | server_partition: None
22:00:00 INFO | client_partition: None
22:00:00 INFO | server_reservation: None
22:00:00 INFO | client_reservation: None
22:00:00 INFO | access_points: wolf-101vm2
22:00:00 INFO | ----------------------------------------------------------------------------------------------------
22:00:00 DEBUG| Common test directory (/var/tmp/daos_testing/test_telemetry_list) contents:
22:00:02 INFO | ----------------------------------------------------------------------------------------------------
22:00:02 INFO | Stopping any of the following commands left running on wolf-101vm[2-4]: orterun,mpirun
22:00:02 DEBUG| Searching for any processes on wolf-101vm[2-4] that match '(orterun|mpirun)'
22:00:02 DEBUG| Running on wolf-101vm[2-4] with a 60 second timeout: /usr/bin/pgrep --list-full '(orterun|mpirun)'
22:00:02 DEBUG| wolf-101vm[2-4] (rc=1): <no output>
22:00:02 DEBUG| No processes found on wolf-101vm[2-4] that match '(orterun|mpirun)'
22:00:02 INFO | ----------------------------------------------------------------------------------------------------
22:00:02 INFO | Updating file permissions for /var/tmp/daos_testing for use with systemctl
22:00:02 INFO | Command: chmod a+rw /var/tmp/daos_testing
Results:
wolf-101vm[2-4]: exit_status=0, interrupted=False:
22:00:02 INFO | ----------------------------------------------------------------------------------------------------
22:00:02 DEBUG| --- SETTING UP SERVER GROUPS: {'daos_server': {'hosts': <ClusterShell.NodeSet.NodeSet object at 0x7f96a42391d0>, 'access_points': <ClusterShell.NodeSet.NodeSet object at 0x7f96b07d4b38>, 'svr_config_file': None, 'dmg_config_file': None, 'svr_config_temp': None, 'dmg_config_temp': None}} ---
22:00:02 INFO | ----------------------------------------------------------------------------------------------------
22:00:02 INFO | --- CONFIGURING SERVER MANAGER ---
22:00:02 DEBUG| PARAMS (key=collect_log_timeout, path=/run/server_manager/*, default=120) => 120
22:00:02 DEBUG| PARAMS (key=storage_format_timeout, path=/run/server_manager/*, default=40) => 40
22:00:02 DEBUG| PARAMS (key=storage_prepare_timeout, path=/run/server_manager/*, default=40) => 40
22:00:02 DEBUG| PARAMS (key=storage_reset_timeout, path=/run/server_manager/*, default=120) => 120
22:00:02 DEBUG| PARAMS (key=env_vars, path=/run/client/*, default=[]) => []
22:00:02 DEBUG| PARAMS (key=env_vars, path=/run/systemctl/*, default=[]) => []
22:00:02 DEBUG| PARAMS (key=allow_proxy, path=/run/daos_server/*, default=False) => False
22:00:02 DEBUG| PARAMS (key=config, path=/run/daos_server/*, default=/etc/daos/daos_server.yml) => '/etc/daos/daos_server.yml'
22:00:02 DEBUG| PARAMS (key=debug, path=/run/daos_server/*, default=True) => True
22:00:02 DEBUG| PARAMS (key=json, path=/run/daos_server/*, default=False) => False
22:00:02 DEBUG| PARAMS (key=json_logs, path=/run/daos_server/*, default=False) => False
22:00:02 DEBUG| PARAMS (key=pattern_timeout, path=/run/daos_server/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=daos_server_sub_command, path=/run/daos_server/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=env_vars, path=/run/client/*, default=[]) => []
22:00:02 DEBUG| PARAMS (key=env_vars, path=/run/daos_server/*, default=[]) => []
22:00:02 DEBUG| PARAMS (key=firstcore, path=/run/daos_server/start/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=group, path=/run/daos_server/start/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=insecure, path=/run/daos_server/start/*, default=False) => False
22:00:02 DEBUG| PARAMS (key=modules, path=/run/daos_server/start/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=port, path=/run/daos_server/start/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=recreate, path=/run/daos_server/start/*, default=False) => False
22:00:02 DEBUG| PARAMS (key=sock_dir, path=/run/daos_server/start/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=storage, path=/run/daos_server/start/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=targets, path=/run/daos_server/start/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=xshelpernr, path=/run/daos_server/start/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=bdev_exclude, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=client_env_vars, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=control_log_file, path=/run/server_config/*, default=daos_control.log) => 'daos_control.log'
22:00:02 DEBUG| Added the directory: control_log_file => /var/tmp/daos_testing/daos_control.log
22:00:02 DEBUG| PARAMS (key=control_log_mask, path=/run/server_config/*, default=DEBUG) => 'DEBUG'
22:00:02 DEBUG| PARAMS (key=core_dump_filter, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=crt_ctx_share_addr, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=crt_timeout, path=/run/server_config/*, default=10) => 10
22:00:02 DEBUG| PARAMS (key=disable_hugepages, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=disable_srx, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=disable_vfio, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=disable_vmd, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=enable_hotplug, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=engines_per_host, path=/run/server_config/*, default=0) => 1
22:00:02 DEBUG| PARAMS (key=fabric_auth_key, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=fault_cb, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=fault_path, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=group_name, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=helper_log_file, path=/run/server_config/*, default=daos_server_helper.log) => 'daos_server_helper.log'
22:00:02 DEBUG| Added the directory: helper_log_file => /var/tmp/daos_testing/daos_server_helper.log
22:00:02 DEBUG| PARAMS (key=hyperthreads, path=/run/server_config/*, default=False) => False
22:00:02 DEBUG| PARAMS (key=nr_hugepages, path=/run/server_config/*, default=0) => 0
22:00:02 DEBUG| PARAMS (key=provider, path=/run/server_config/*, default=ofi+tcp;ofi_rxm) => 'ofi+tcp;ofi_rxm'
22:00:02 DEBUG| PARAMS (key=socket_dir, path=/run/server_config/*, default=/var/run/daos_server) => '/var/run/daos_server'
22:00:02 DEBUG| PARAMS (key=system_ram_reserved, path=/run/server_config/*, default=None) => 1
22:00:02 DEBUG| PARAMS (key=telemetry_port, path=/run/server_config/*, default=9191) => 9191
22:00:02 DEBUG| PARAMS (key=user_name, path=/run/server_config/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=access_points, path=/run/common_config/*, default=['localhost']) => ['localhost']
22:00:02 DEBUG| PARAMS (key=name, path=/run/common_config/*, default=daos_server) => 'daos_server'
22:00:02 DEBUG| PARAMS (key=port, path=/run/common_config/*, default=10001) => 10001
22:00:02 DEBUG| PARAMS (key=allow_insecure, path=/run/server_config/transport_config/*, default=False) => False
22:00:02 DEBUG| PARAMS (key=ca_cert, path=/run/server_config/transport_config/*, default=daosCA.crt) => 'daosCA.crt'
22:00:02 DEBUG| Added the directory: ca_cert => /etc/daos/certs/daosCA.crt
22:00:02 DEBUG| PARAMS (key=cert, path=/run/server_config/transport_config/*, default=server.crt) => 'server.crt'
22:00:02 DEBUG| Added the directory: cert => /etc/daos/certs/server.crt
22:00:02 DEBUG| PARAMS (key=client_cert_dir, path=/run/server_config/transport_config/*, default=clients) => 'clients'
22:00:02 DEBUG| Added the directory: client_cert_dir => /etc/daos/certs/clients
22:00:02 DEBUG| PARAMS (key=key, path=/run/server_config/transport_config/*, default=server.key) => 'server.key'
22:00:02 DEBUG| Added the directory: key => /etc/daos/certs/server.key
22:00:02 DEBUG| PARAMS (key=device, path=/run/server_config/control_metadata/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=path, path=/run/server_config/control_metadata/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=crt_ctx_share_addr, path=/run/server_config/engines/0/*, default=0) => 0
22:00:02 DEBUG| PARAMS (key=env_vars, path=/run/server_config/engines/0/*, default=['ABT_ENV_MAX_NUM_XSTREAMS=100', 'ABT_MAX_NUM_XSTREAMS=100', 'DAOS_MD_CAP=1024', 'DAOS_SCHED_WATCHDOG_ALL=1', 'DD_MASK=mgmt,io,md,epc,rebuild', 'D_LOG_FILE_APPEND_PID=1', 'COVFILE=/tmp/test.cov']) => ['ABT_ENV_MAX_NUM_XSTREAMS=100', 'ABT_MAX_NUM_XSTREAMS=100', 'DAOS_MD_CAP=1024', 'DAOS_SCHED_WATCHDOG_ALL=1', 'DD_MASK=mgmt,io,md,epc,rebuild', 'D_LOG_FILE_APPEND_PID=1', 'COVFILE=/tmp/test.cov']
22:00:02 DEBUG| PARAMS (key=fabric_iface, path=/run/server_config/engines/0/*, default=eth0) => 'eth0'
22:00:02 DEBUG| PARAMS (key=fabric_iface_port, path=/run/server_config/engines/0/*, default=31416) => 31416
22:00:02 DEBUG| PARAMS (key=first_core, path=/run/server_config/engines/0/*, default=0) => 0
22:00:02 DEBUG| PARAMS (key=log_file, path=/run/server_config/engines/0/*, default=daos_server.log) => 'daos_server.log'
22:00:02 DEBUG| Added the directory: log_file => /var/tmp/daos_testing/daos_server.log
22:00:02 DEBUG| PARAMS (key=log_mask, path=/run/server_config/engines/0/*, default=INFO) => 'INFO'
22:00:02 DEBUG| PARAMS (key=nr_xs_helpers, path=/run/server_config/engines/0/*, default=4) => 0
22:00:02 DEBUG| PARAMS (key=pinned_numa_node, path=/run/server_config/engines/0/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=targets, path=/run/server_config/engines/0/*, default=8) => 4
22:00:02 DEBUG| Checking for storage tier in /run/server_config/engines/0/storage/0/*
22:00:02 DEBUG| PARAMS (key=class, path=/run/server_config/engines/0/storage/0/*, default=None) => 'ram'
22:00:02 DEBUG| PARAMS (key=scm_list, path=/run/server_config/engines/0/storage/0/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=scm_mount, path=/run/server_config/engines/0/storage/0/*, default=None) => '/mnt/daos'
22:00:02 DEBUG| PARAMS (key=scm_size, path=/run/server_config/engines/0/storage/0/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=scm_hugepages_disabled, path=/run/server_config/engines/0/storage/0/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=bdev_list, path=/run/server_config/engines/0/storage/0/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=bdev_number, path=/run/server_config/engines/0/storage/0/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=bdev_size, path=/run/server_config/engines/0/storage/0/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=bdev_roles, path=/run/server_config/engines/0/storage/0/*, default=None) => None
22:00:02 DEBUG| Checking for storage tier in /run/server_config/engines/0/storage/1/*
22:00:02 DEBUG| PARAMS (key=class, path=/run/server_config/engines/0/storage/1/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=_hostlist, path=/run/dmg/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=configpath, path=/run/dmg/*, default=/etc/daos/daos_control.yml) => '/etc/daos/daos_control.yml'
22:00:02 DEBUG| PARAMS (key=debug, path=/run/dmg/*, default=True) => True
22:00:02 DEBUG| PARAMS (key=hostfile, path=/run/dmg/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=insecure, path=/run/dmg/*, default=False) => False
22:00:02 DEBUG| PARAMS (key=json, path=/run/dmg/*, default=False) => False
22:00:02 DEBUG| PARAMS (key=pattern_timeout, path=/run/dmg/*, default=10) => 10
22:00:02 DEBUG| PARAMS (key=dmg_sub_command, path=/run/dmg/*, default=None) => None
22:00:02 DEBUG| PARAMS (key=env_vars, path=/run/client/*, default=[]) => []
22:00:02 DEBUG| PARAMS (key=env_vars, path=/run/dmg/*, default=[]) => []
22:00:02 DEBUG| PARAMS (key=hostlist, path=/run/dmg/*, default=localhost) => 'localhost'
22:00:02 DEBUG| PARAMS (key=name, path=/run/dmg/*, default=daos_server) => 'daos_server'
22:00:02 DEBUG| PARAMS (key=port, path=/run/dmg/*, default=10001) => 10001
22:00:02 DEBUG| PARAMS (key=allow_insecure, path=/run/dmg/transport_config/*, default=False) => False
22:00:02 DEBUG| PARAMS (key=ca_cert, path=/run/dmg/transport_config/*, default=daosCA.crt) => 'daosCA.crt'
22:00:02 DEBUG| Added the directory: ca_cert => /etc/daos/certs/daosCA.crt
22:00:02 DEBUG| PARAMS (key=cert, path=/run/dmg/transport_config/*, default=admin.crt) => 'admin.crt'
22:00:02 DEBUG| Added the directory: cert => /etc/daos/certs/admin.crt
22:00:02 DEBUG| PARAMS (key=key, path=/run/dmg/transport_config/*, default=admin.key) => 'admin.key'
22:00:02 DEBUG| Added the directory: key => /etc/daos/certs/admin.key
22:00:02 DEBUG| Updated param access_points => ['wolf-101vm2']
22:00:02 INFO | ----------------------------------------------------------------------------------------------------
22:00:02 INFO | --- VERIFYING STATES OF 1 SERVERS GROUP ---
22:00:02 DEBUG| Copying certificates for dmg:
22:00:03 DEBUG| /var/tmp/daos_testing/daosCA/certs/daosCA.crt -> /etc/daos/certs/daosCA.crt
22:00:04 DEBUG| /var/tmp/daos_testing/daosCA/certs/admin.crt -> /etc/daos/certs/admin.crt
22:00:05 DEBUG| /var/tmp/daos_testing/daosCA/certs/admin.key -> /etc/daos/certs/admin.key
22:00:06 DEBUG| Copied certificates for dmg (in /etc/daos/certs):
22:00:06 DEBUG| wolf-101vm1: total 40
22:00:06 DEBUG| wolf-101vm1: drwxr-xr-x 3 root root 4096 Feb 21 21:46 .
22:00:06 DEBUG| wolf-101vm1: drwxr-xr-x 3 root root 4096 Feb 21 21:55 ..
22:00:06 DEBUG| wolf-101vm1: -rw-r--r-- 1 jenkins jenkins 5238 Feb 21 22:00 admin.crt
22:00:06 DEBUG| wolf-101vm1: -r-------- 1 jenkins jenkins 2459 Feb 21 22:00 admin.key
22:00:06 DEBUG| wolf-101vm1: -rw-r--r-- 1 daos_agent daos_agent 5238 Feb 21 21:54 agent.crt
22:00:06 DEBUG| wolf-101vm1: -r-------- 1 daos_agent daos_agent 2459 Feb 21 21:54 agent.key
22:00:06 DEBUG| wolf-101vm1: drwx------ 2 daos_server daos_server 4096 Feb 21 21:29 clients
22:00:06 DEBUG| wolf-101vm1: -rw-r--r-- 1 jenkins jenkins 1436 Feb 21 22:00 daosCA.crt
22:00:06 DEBUG| Updated param dmg.yaml.hostlist => ['wolf-101vm2', 'wolf-101vm3']
22:00:06 DEBUG| Updated param ranks => None
22:00:06 DEBUG| Updated param verbose => True
22:00:06 INFO | Writing yaml configuration file /var/tmp/daos_testing/test_telemetry_list/test_daos_server_dmg.yaml
22:00:06 INFO | Copying /var/tmp/daos_testing/test_telemetry_list/test_daos_server_dmg.yaml yaml configuration file to /etc/daos/daos_control.yml on wolf-101vm1
22:00:07 INFO | Command environment vars:
{}
22:00:07 INFO | Running '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j system query --verbose'
22:00:07 DEBUG| [stderr] DEBUG 22:00:07.501032 main.go:209: debug output enabled
22:00:07 DEBUG| [stderr] DEBUG 22:00:07.501756 main.go:241: control config loaded from /etc/daos/daos_control.yml
22:00:07 DEBUG| [stderr] DEBUG 22:00:07.523286 system.go:279: DAOS system query request: *mgmt.SystemQueryReq (sys:"daos_server-2.5.100" state_mask:65535)
22:00:07 DEBUG| [stderr] DEBUG 22:00:07.524019 rpc.go:278: request hosts: [wolf-101vm2:10001 wolf-101vm3:10001]
22:00:07 DEBUG| [stderr] DEBUG 22:00:07.530232 response.go:168: wolf-101vm3:10001: err: client: code = 506 description = "the server at wolf-101vm3:10001 refused the connection"
22:00:07 DEBUG| [stderr] DEBUG 22:00:07.530347 response.go:168: wolf-101vm3:10001: err: client: code = 506 description = "the server at wolf-101vm3:10001 refused the connection"
22:00:07 DEBUG| [stdout] {
22:00:07 DEBUG| [stderr] ERROR: dmg: system query failed: client: code = 506 description = "the server at wolf-101vm3:10001 refused the connection": unable to contact the DAOS Management Service
22:00:07 DEBUG| [stdout] "response": null,
22:00:07 DEBUG| [stdout] "error": "system query failed: client: code = 506 description = \"the server at wolf-101vm3:10001 refused the connection\": unable to contact the DAOS Management Service",
22:00:07 DEBUG| [stdout] "status": -1025
22:00:07 DEBUG| [stdout] }
22:00:07 INFO | Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j system query --verbose' finished with 1 after 0.06561040878295898s
22:00:07 INFO | Error occurred running '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j system query --verbose': Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j system query --verbose' failed.
stdout: b'{\n "response": null,\n "error": "system query failed: client: code = 506 description = \\"the server at wolf-101vm3:10001 refused the connection\\": unable to contact the DAOS Management Service",\n "status": -1025\n}\n'
stderr: b'DEBUG 22:00:07.501032 main.go:209: debug output enabled\nDEBUG 22:00:07.501756 main.go:241: control config loaded from /etc/daos/daos_control.yml\nDEBUG 22:00:07.523286 system.go:279: DAOS system query request: *mgmt.SystemQueryReq (sys:"daos_server-2.5.100" state_mask:65535)\nDEBUG 22:00:07.524019 rpc.go:278: request hosts: [wolf-101vm2:10001 wolf-101vm3:10001]\nDEBUG 22:00:07.530232 response.go:168: wolf-101vm3:10001: err: client: code = 506 description = "the server at wolf-101vm3:10001 refused the connection"\nDEBUG 22:00:07.530347 response.go:168: wolf-101vm3:10001: err: client: code = 506 description = "the server at wolf-101vm3:10001 refused the connection"\nERROR: dmg: system query failed: client: code = 506 description = "the server at wolf-101vm3:10001 refused the connection": unable to contact the DAOS Management Service\n'
additional_info: None
22:00:07 INFO | <SERVER> Assigning expected server states: {}
22:00:07 INFO | <SERVER> Verifying server states: group=daos_server, hosts=wolf-101vm[2-3]
22:00:07 INFO | Unable to obtain current server state. Undefined expected server states due to a failure starting the server.
22:00:07 INFO | ----------------------------------------------------------------------------------------------------
22:00:07 INFO | --- STOPPING SERVERS ---
22:00:07 INFO | Stopping 1 group(s) of servers
22:00:07 INFO | <SERVER> Stopping server systemd command
22:00:07 INFO | Command: sudo -n systemctl stop daos_server.service
Results:
wolf-101vm[2-3]: exit_status=0, interrupted=False:
22:00:08 INFO | Command: sudo -n systemctl disable daos_server.service
Results:
wolf-101vm[2-3]: exit_status=0, interrupted=False:
22:00:08 INFO | Command: sudo -n systemctl stop daos_server.service
Results:
wolf-101vm[2-3]: exit_status=0, interrupted=False:
22:00:08 INFO | Command: sudo -n systemctl disable daos_server.service
Results:
wolf-101vm[2-3]: exit_status=0, interrupted=False:
22:00:08 DEBUG| Searching for any processes on wolf-101vm[2-3] that match '(daos_server|daos_engine)'
22:00:08 DEBUG| Running on wolf-101vm[2-3] with a 60 second timeout: /usr/bin/pgrep --list-full '(daos_server|daos_engine)'
22:00:08 DEBUG| wolf-101vm[2-3] (rc=1): <no output>
22:00:08 DEBUG| No processes found on wolf-101vm[2-3] that match '(daos_server|daos_engine)'
22:00:08 INFO | No remote '(daos_server|daos_engine)' processes killed on wolf-101vm[2-3] (none found), done.
22:00:08 INFO | ----------------------------------------------------------------------------------------------------
22:00:08 INFO | --- STARTING SERVERS ---
22:00:08 INFO | Starting server: group=daos_server, hosts=wolf-101vm[2-3], config=/etc/daos/daos_server.yml
22:00:08 INFO | <SERVER> Preparing to start daos_server on wolf-101vm[2-3] with systemd
22:00:08 INFO | Writing yaml configuration file /var/tmp/daos_testing/test_telemetry_list/test_daos_server_server.yaml
22:00:08 INFO | Copying /var/tmp/daos_testing/test_telemetry_list/test_daos_server_server.yaml yaml configuration file to /etc/daos/daos_server.yml on wolf-101vm[2-3]
22:00:11 DEBUG| Updating pattern timeout based upon server config
22:00:11 DEBUG| Detected bdev_list entries: []
22:00:11 DEBUG| Updated param DaosServerCommand.pattern_timeout => 40
22:00:11 DEBUG| Copying certificates for daos_server:
22:00:11 DEBUG| /var/tmp/daos_testing/daosCA/certs/daosCA.crt -> /etc/daos/certs/daosCA.crt
22:00:14 DEBUG| /var/tmp/daos_testing/daosCA/certs/server.crt -> /etc/daos/certs/server.crt
22:00:16 DEBUG| /var/tmp/daos_testing/daosCA/certs/server.key -> /etc/daos/certs/server.key
22:00:19 DEBUG| /var/tmp/daos_testing/daosCA/certs/agent.crt -> /etc/daos/certs/clients/agent.crt
22:00:21 DEBUG| Copied certificates for daos_server (in /etc/daos/certs/clients, /etc/daos/certs):
22:00:22 DEBUG| wolf-101vm3: /etc/daos/certs:
22:00:22 DEBUG| wolf-101vm3: total 40
22:00:22 DEBUG| wolf-101vm3: drwxr-xr-x 3 root root 4096 Feb 21 21:47 .
22:00:22 DEBUG| wolf-101vm3: drwxr-xr-x 3 root root 4096 Feb 21 22:00 ..
22:00:22 DEBUG| wolf-101vm3: -rw-r--r-- 1 daos_agent daos_agent 5238 Feb 21 21:54 agent.crt
22:00:22 DEBUG| wolf-101vm3: -r-------- 1 daos_agent daos_agent 2459 Feb 21 21:54 agent.key
22:00:22 DEBUG| wolf-101vm3: drwx------ 2 daos_server daos_server 4096 Feb 21 21:47 clients
22:00:22 DEBUG| wolf-101vm3: -rw-r--r-- 1 daos_server daos_server 1436 Feb 21 22:00 daosCA.crt
22:00:22 DEBUG| wolf-101vm3: -rw-r--r-- 1 daos_server daos_server 5287 Feb 21 22:00 server.crt
22:00:22 DEBUG| wolf-101vm3: -r-------- 1 daos_server daos_server 2459 Feb 21 22:00 server.key
22:00:22 DEBUG| wolf-101vm3:
22:00:22 DEBUG| wolf-101vm3: /etc/daos/certs/clients:
22:00:22 DEBUG| wolf-101vm3: total 16
22:00:22 DEBUG| wolf-101vm3: drwx------ 2 daos_server daos_server 4096 Feb 21 21:47 .
22:00:22 DEBUG| wolf-101vm3: drwxr-xr-x 3 root root 4096 Feb 21 21:47 ..
22:00:22 DEBUG| wolf-101vm3: -rw-r--r-- 1 daos_server daos_server 5238 Feb 21 22:00 agent.crt
22:00:22 DEBUG| wolf-101vm2: /etc/daos/certs:
22:00:22 DEBUG| wolf-101vm2: total 28
22:00:22 DEBUG| wolf-101vm2: drwxr-xr-x 3 root root 4096 Feb 21 21:45 .
22:00:22 DEBUG| wolf-101vm2: drwxr-xr-x 3 root root 4096 Feb 21 22:00 ..
22:00:22 DEBUG| wolf-101vm2: drwx------ 2 daos_server daos_server 4096 Feb 21 21:45 clients
22:00:22 DEBUG| wolf-101vm2: -rw-r--r-- 1 daos_server daos_server 1436 Feb 21 22:00 daosCA.crt
22:00:22 DEBUG| wolf-101vm2: -rw-r--r-- 1 daos_server daos_server 5287 Feb 21 22:00 server.crt
22:00:22 DEBUG| wolf-101vm2: -r-------- 1 daos_server daos_server 2459 Feb 21 22:00 server.key
22:00:22 DEBUG| wolf-101vm2:
22:00:22 DEBUG| wolf-101vm2: /etc/daos/certs/clients:
22:00:22 DEBUG| wolf-101vm2: total 16
22:00:22 DEBUG| wolf-101vm2: drwx------ 2 daos_server daos_server 4096 Feb 21 21:45 .
22:00:22 DEBUG| wolf-101vm2: drwxr-xr-x 3 root root 4096 Feb 21 21:45 ..
22:00:22 DEBUG| wolf-101vm2: -rw-r--r-- 1 daos_server daos_server 5238 Feb 21 22:00 agent.crt
22:00:22 DEBUG| Copying certificates for dmg:
22:00:22 DEBUG| /var/tmp/daos_testing/daosCA/certs/daosCA.crt -> /etc/…"metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_gc_sys_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of bytes used for garbage collection system metadata.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_heap_alloc_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of heap bytes allocated and still in use.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_heap_idle_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of heap bytes waiting to be used.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_heap_inuse_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of heap bytes that are in use.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_heap_objects",
22:01:07 DEBUG| [stdout] "description": "Number of allocated objects.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_heap_released_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of heap bytes released to OS.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_heap_sys_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of heap bytes obtained from system.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_last_gc_time_seconds",
22:01:07 DEBUG| [stdout] "description": "Number of seconds since 1970 of last garbage collection.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "counter",
22:01:07 DEBUG| [stdout] "name": "go_memstats_lookups_total",
22:01:07 DEBUG| [stdout] "description": "Total number of pointer lookups.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "counter",
22:01:07 DEBUG| [stdout] "name": "go_memstats_mallocs_total",
22:01:07 DEBUG| [stdout] "description": "Total number of mallocs.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_mcache_inuse_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of bytes in use by mcache structures.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_mcache_sys_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of bytes used for mcache structures obtained from system.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_mspan_inuse_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of bytes in use by mspan structures.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_mspan_sys_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of bytes used for mspan structures obtained from system.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_next_gc_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of heap bytes when next garbage collection will take place.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_other_sys_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of bytes used for other system allocations.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_stack_inuse_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of bytes in use by the stack allocator.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_
22:01:07 DEBUG| [stdout] stack_sys_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of bytes obtained from system for stack allocator.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_memstats_sys_bytes",
22:01:07 DEBUG| [stdout] "description": "Number of bytes obtained from system.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "go_threads",
22:01:07 DEBUG| [stdout] "description": "Number of OS threads created.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "counter",
22:01:07 DEBUG| [stdout] "name": "process_cpu_seconds_total",
22:01:07 DEBUG| [stdout] "description": "Total user and system CPU time spent in seconds.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "process_max_fds",
22:01:07 DEBUG| [stdout] "description": "Maximum number of open file descriptors.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "process_open_fds",
22:01:07 DEBUG| [stdout] "description": "Number of open file descriptors.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "process_resident_memory_bytes",
22:01:07 DEBUG| [stdout] "description": "Resident memory size in bytes.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "process_start_time_seconds",
22:01:07 DEBUG| [stdout] "description": "Start time of the process since unix epoch in seconds.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "process_virtual_memory_bytes",
22:01:07 DEBUG| [stdout] "description": "Virtual memory size in bytes.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] {
22:01:07 DEBUG| [stdout] "type": "gauge",
22:01:07 DEBUG| [stdout] "name": "process_virtual_memory_max_bytes",
22:01:07 DEBUG| [stdout] "description": "Maximum amount of virtual memory available in bytes.",
22:01:07 DEBUG| [stdout] "metrics": null
22:01:07 DEBUG| [stdout] }
22:01:07 DEBUG| [stdout] ]
22:01:07 DEBUG| [stdout] },
22:01:07 DEBUG| [stdout] "error": null,
22:01:07 DEBUG| [stdout] "status": 0
22:01:07 DEBUG| [stdout] }
22:01:07 INFO | Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j telemetry metrics list --host-list=wolf-101vm3' finished with 0 after 1.5714735984802246s
22:01:07 INFO | Detected 2/2 telemetry metrics list hosts
22:01:07 INFO | Difference between expected and actual: set()
22:01:07 INFO | Symmetric difference between expected and actual: set()
22:01:07 INFO | wolf-101vm2: detected 383/324 telemetry metric names
22:01:07 INFO | wolf-101vm2: difference between expected and actual: set()
22:01:07 INFO | wolf-101vm2: symmetric difference between expected and actual: {'engine_io_ops_dkey_enum_active_samples', 'engine_io_ops_obj_coll_punch_active_samples', 'engine_io_latency_bulk_update_samples', 'engine_io_ops_compound_latency_samples', 'engine_io_ops_tgt_punch_active_samples', 'engine_io_ops_ec_rep_active_samples', 'engine_io_latency_bulk_fetch_samples', 'engine_dmabuff_grab_retries_samples', 'engine_io_latency_bio_update_samples', 'engine_io_ops_tgt_dkey_punch_active_samples', 'engine_io_ops_key2anchor_active_samples', 'engine_io_ops_dkey_enum_latency_samples', 'engine_io_ops_compound_active_samples', 'engine_io_ops_key_query_latency_samples', 'engine_net_swim_delay_samples', 'engine_io_ops_key_query_active_samples', 'engine_io_latency_bio_fetch_samples', 'engine_io_ops_akey_punch_latency_samples', 'engine_io_ops_obj_coll_punch_latency_samples', 'engine_io_latency_fetch_samples', 'engine_dmabuff_wal_qd_samples', 'engine_io_latency_update_samples', 'engine_io_ops_obj_sync_active_samples', 'engine_sched_cycle_duration_samples', 'engine_io_ops_tgt_dkey_punch_latency_samples', 'engine_io_ops_obj_sync_latency_samples', 'engine_io_ops_tgt_akey_punch_latency_samples', 'engine_dmabuff_wal_waiters_samples', 'engine_io_ops_akey_enum_latency_samples', 'engine_io_ops_recx_enum_latency_samples', 'engine_io_ops_akey_enum_active_samples', 'engine_dmabuff_wal_sz_samples', 'engine_io_ops_obj_punch_active_samples', 'engine_io_ops_key2anchor_latency_samples', 'engine_io_latency_vos_fetch_samples', 'engine_io_ops_ec_agg_latency_samples', 'engine_io_ops_migrate_active_samples', 'engine_io_ops_fetch_active_samples', 'engine_io_ops_migrate_latency_samples', 'engine_io_dtx_committed_samples', 'engine_io_ops_ec_rep_latency_samples', 'engine_io_latency_tgt_update_samples', 'engine_io_ops_akey_punch_active_samples', 'engine_io_ops_dkey_punch_latency_samples', 'engine_io_ops_obj_punch_latency_samples', 'engine_io_ops_tgt_update_active_samples', 'engine_io_ops_obj_coll_query_latency_samples', 'engine_io_ops_obj_enum_active_samples', 'engine_io_ops_ec_agg_active_samples', 'engine_sched_cycle_size_samples', 'engine_io_ops_dkey_punch_active_samples', 'engine_io_latency_vos_update_samples', 'engine_io_ops_obj_coll_query_active_samples', 'engine_io_ops_recx_enum_active_samples', 'engine_io_ops_update_active_samples', 'engine_io_ops_obj_enum_latency_samples', 'engine_io_ops_tgt_punch_latency_samples', 'engine_io_ops_tgt_akey_punch_active_samples', 'engine_io_dtx_committable_samples'}
22:01:07 INFO | wolf-101vm3: detected 383/324 telemetry metric names
22:01:07 INFO | wolf-101vm3: difference between expected and actual: set()
22:01:07 INFO | wolf-101vm3: symmetric difference between expected and actual: {'engine_io_ops_dkey_enum_active_samples', 'engine_io_ops_obj_coll_punch_active_samples', 'engine_io_latency_bulk_update_samples', 'engine_io_ops_compound_latency_samples', 'engine_io_ops_tgt_punch_active_samples', 'engine_io_ops_ec_rep_active_samples', 'engine_io_latency_bulk_fetch_samples', 'engine_dmabuff_grab_retries_samples', 'engine_io_latency_bio_update_samples', 'engine_io_ops_tgt_dkey_punch_active_samples', 'engine_io_ops_key2anchor_active_samples', 'engine_io_ops_dkey_enum_latency_samples', 'engine_io_ops_compound_active_samples', 'engine_io_ops_key_query_latency_samples', 'engine_net_swim_delay_samples', 'engine_io_ops_key_query_active_samples', 'engine_io_latency_bio_fetch_samples', 'engine_io_ops_akey_punch_latency_samples', 'engine_io_ops_obj_coll_punch_latency_samples', 'engine_io_latency_fetch_samples', 'engine_dmabuff_wal_qd_samples', 'engine_io_latency_update_samples', 'engine_io_ops_obj_sync_active_samples', 'engine_sched_cycle_duration_samples', 'engine_io_ops_tgt_dkey_punch_latency_samples', 'engine_io_ops_obj_sync_latency_samples', 'engine_io_ops_tgt_akey_punch_latency_samples', 'engine_dmabuff_wal_waiters_samples', 'engine_io_ops_akey_enum_latency_samples', 'engine_io_ops_recx_enum_latency_samples', 'engine_io_ops_akey_enum_active_samples', 'engine_dmabuff_wal_sz_samples', 'engine_io_ops_obj_punch_active_samples', 'engine_io_ops_key2anchor_latency_samples', 'engine_io_latency_vos_fetch_samples', 'engine_io_ops_ec_agg_latency_samples', 'engine_io_ops_migrate_active_samples', 'engine_io_ops_fetch_active_samples', 'engine_io_ops_migrate_latency_samples', 'engine_io_dtx_committed_samples', 'engine_io_ops_ec_rep_latency_samples', 'engine_io_latency_tgt_update_samples', 'engine_io_ops_akey_punch_active_samples', 'engine_io_ops_dkey_punch_latency_samples', 'engine_io_ops_obj_punch_latency_samples', 'engine_io_ops_tgt_update_active_samples', 'engine_io_ops_obj_coll_query_latency_samples', 'engine_io_ops_obj_enum_active_samples', 'engine_io_ops_ec_agg_active_samples', 'engine_sched_cycle_size_samples', 'engine_io_ops_dkey_punch_active_samples', 'engine_io_latency_vos_update_samples', 'engine_io_ops_obj_coll_query_active_samples', 'engine_io_ops_recx_enum_active_samples', 'engine_io_ops_update_active_samples', 'engine_io_ops_obj_enum_latency_samples', 'engine_io_ops_tgt_punch_latency_samples', 'engine_io_ops_tgt_akey_punch_active_samples', 'engine_io_dtx_committable_samples'}
22:01:07 INFO | Test has failed, dumping ULT stacks
22:01:07 INFO | Dumping ULT stacks of engines on wolf-101vm[2-3]
22:01:31 ERROR|
22:01:31 ERROR| Reproduced traceback from: /usr/lib/python3.6/site-packages/avocado/core/test.py:767
22:01:31 ERROR| Traceback (most recent call last):
22:01:31 ERROR| File "/usr/lib/daos/TESTING/ftest/control/dmg_telemetry_basic.py", line 84, in test_telemetry_list
22:01:31 ERROR| self.verify_telemetry_list()
22:01:31 ERROR| File "/usr/lib/daos/TESTING/ftest/util/telemetry_test_base.py", line 82, in verify_telemetry_list
22:01:31 ERROR| self.fail("\n".join(errors))
22:01:31 ERROR| File "/usr/lib/daos/TESTING/ftest/util/apricot/apricot/test.py", line 1374, in fail
22:01:31 ERROR| self.dump_engines_stacks("Test has failed")
22:01:31 ERROR| File "/usr/lib/daos/TESTING/ftest/util/apricot/apricot/test.py", line 1363, in dump_engines_stacks
22:01:31 ERROR| dump_engines_stacks(self.hostlist_servers)
22:01:31 ERROR| File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 827, in dump_engines_stacks
22:01:31 ERROR| result = pcmd(hosts, "; ".join(commands), verbose, timeout, None)
22:01:31 ERROR| File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 594, in pcmd
22:01:31 ERROR| results = run_pcmd(hosts, command, verbose, timeout, expect_rc)
22:01:31 ERROR| File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 449, in run_pcmd
22:01:31 ERROR| task = run_task(hosts, command, timeout)
22:01:31 ERROR| File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 312, in run_task
22:01:31 ERROR| task.run(**kwargs)
22:01:31 ERROR| File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 877, in run
22:01:31 ERROR| self.resume(timeout)
22:01:31 ERROR| File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 831, in resume
22:01:31 ERROR| self._resume()
22:01:31 ERROR| File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 794, in _resume
22:01:31 ERROR| self._run(self.timeout)
22:01:31 ERROR| File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 404, in _run
22:01:31 ERROR| self._engine.run(timeout)
22:01:31 ERROR| File "/usr/lib/python3.6/site-packages/ClusterShell/Engine/Engine.py", line 723, in run
22:01:31 ERROR| self.runloop(timeout)
22:01:31 ERROR| File "/usr/lib/python3.6/site-packages/ClusterShell/Engine/EPoll.py", line 122, in runloop
22:01:31 ERROR| evlist = self.epolling.poll(poll_timeo)
22:01:31 ERROR| File "/usr/lib/python3.6/site-packages/avocado/plugins/runner.py", line 77, in sigterm_handler
22:01:31 ERROR| raise RuntimeError("Test interrupted by SIGTERM")
22:01:31 ERROR| RuntimeError: Test interrupted by SIGTERM
22:01:31 ERROR|
22:01:31 DEBUG| Local variables:
22:01:31 DEBUG| -> self <class 'dmg_telemetry_basic.TestWithTelemetryBasic'>: 1-./control/dmg_telemetry_basic.py:TestWithTelemetryBasic.test_telemetry_list;run-container-hosts-pool-server_config-engines-0-storage-0-test-timeouts-dadb
22:01:31 INFO | ====================================================================================================
22:01:31 INFO | *** TEARDOWN called due to TIMEOUT: 90 second timeout exceeded ***
22:01:31 INFO | test execution has been terminated by avocado
22:01:31 DEBUG| no pre-teardown steps defined
22:01:31 INFO | ----------------------------------------------------------------------------------------------------
22:01:31 INFO | --- STOPPING AGENTS ---
22:01:31 INFO | ----------------------------------------------------------------------------------------------------
22:01:31 INFO | --- VERIFYING STATES OF 1 AGENTS GROUP ---
22:01:31 INFO | Command: systemctl is-active daos_agent.service
Results:
wolf-101vm[1,4]: exit_status=0, interrupted=False: active
22:01:31 INFO | <AGENT> Verifying agent states: group=daos_server, hosts=wolf-101vm[1,4]
22:01:31 INFO | Rank Host UUID Expected State Current State Result
22:01:31 INFO | ---- --------------- ------------------------------------ ---------------------- -------------- ------
22:01:31 INFO | 0 wolf-101vm1 - active active PASS
22:01:31 INFO | 1 wolf-101vm4 - active active PASS
22:01:31 INFO | Agents are configured to run across multiple test variants, not stopping
22:01:31 INFO | ----------------------------------------------------------------------------------------------------
22:01:31 INFO | --- STOPPING SERVERS ---
22:01:31 INFO | ----------------------------------------------------------------------------------------------------
22:01:31 INFO | --- VERIFYING STATES OF 1 SERVERS GROUP ---
22:01:31 DEBUG| Updated param ranks => None
22:01:31 DEBUG| Updated param verbose => True
22:01:31 INFO | Command environment vars:
{}
22:01:31 INFO | Running '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j system query --verbose'
22:01:31 DEBUG| [stderr] DEBUG 22:01:31.613811 main.go:209: debug output enabled
22:01:31 DEBUG| [stderr] DEBUG 22:01:31.614391 main.go:241: control config loaded from /etc/daos/daos_control.yml
22:01:31 DEBUG| [stderr] DEBUG 22:01:31.635874 system.go:279: DAOS system query request: *mgmt.SystemQueryReq (sys:"daos_server-2.5.100" state_mask:65535)
22:01:31 DEBUG| [stderr] DEBUG 22:01:31.636568 rpc.go:278: request hosts: [wolf-101vm2:10001 wolf-101vm3:10001]
22:01:31 DEBUG| [stderr] DEBUG 22:01:31.758936 response.go:168: wolf-101vm2:10001: *mgmt.SystemQueryResp@6 joined:0-1
22:01:31 DEBUG| [stdout] {
22:01:31 DEBUG| [stdout] "response": {
22:01:31 DEBUG| [stdout] "members": [
22:01:31 DEBUG| [stdout] {
22:01:31 DEBUG| [stdout] "addr": "10.8.2.149:10001",
22:01:31 DEBUG| [stdout] "state": "joined",
22:01:31 DEBUG| [stdout] "fault_domain": "/wolf-101vm2.wolf.hpdd.intel.com",
22:01:31 DEBUG| [stdout] "rank": 0,
22:01:31 DEBUG| [stdout] "incarnation": 1585498280318205952,
22:01:31 DEBUG| [stdout] "uuid": "5482ff94-a36c-4106-b992-ba83bdcd39a1",
22:01:31 DEBUG| [stdout] "fabric_uri": "ofi+tcp;ofi_rxm://10.8.2.149:31416",
22:01:31 DEBUG| [stdout] "fabric_contexts": 6,
22:01:31 DEBUG| [stdout] "info": "",
22:01:31 DEBUG| [stdout] "last_update": "2024-02-21T22:00:44.528024236Z"
22:01:31 DEBUG| [stdout] },
22:01:31 DEBUG| [stdout] {
22:01:31 DEBUG| [stdout] "addr": "10.8.2.150:10001",
22:01:31 DEBUG| [stdout] "state": "joined",
22:01:31 DEBUG| [stdout] "fault_domain": "/wolf-101vm3.wolf.hpdd.intel.com",
22:01:31 DEBUG| [stdout] "rank": 1,
22:01:31 DEBUG| [stdout] "incarnation": 1585498280083062784,
22:01:31 DEBUG| [stdout] "uuid": "4449282a-4787-4a34-beab-d596eddbe371",
22:01:31 DEBUG| [stdout] "fabric_uri": "ofi+tcp;ofi_rxm://10.8.2.150:31416",
22:01:31 DEBUG| [stdout] "fabric_contexts": 6,
22:01:31 DEBUG| [stdout] "info": "",
22:01:31 DEBUG| [stdout] "last_update": "2024-02-21T22:00:43.529361272Z"
22:01:31 DEBUG| [stdout] }
22:01:31 DEBUG| [stdout] ]
22:01:31 DEBUG| [stdout] },
22:01:31 DEBUG| [stdout] "error": null,
22:01:31 DEBUG| [stdout] "status": 0
22:01:31 DEBUG| [stdout] }
22:01:32 INFO | Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j system query --verbose' finished with 0 after 1.1766183376312256s
22:01:32 INFO | <SERVER> Verifying server states: group=daos_server, hosts=wolf-101vm[2-3]
22:01:32 INFO | Rank Host UUID Expected State Current State Result
22:01:32 INFO | ---- --------------- ------------------------------------ ---------------------- -------------- ------
22:01:32 INFO | 0 wolf-101vm2 5482ff94-a36c-4106-b992-ba83bdcd39a1 joined joined PASS
22:01:32 INFO | 1 wolf-101vm3 4449282a-4787-4a34-beab-d596eddbe371 joined joined PASS
22:01:32 INFO | Searching for any existing pools
22:01:32 DEBUG| Updated param no_query => False
22:01:32 DEBUG| Updated param verbose => False
22:01:32 INFO | Command environment vars:
{}
22:01:32 INFO | Running '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j pool list'
22:01:32 DEBUG| [stderr] DEBUG 22:01:32.802634 main.go:209: debug output enabled
22:01:32 DEBUG| [stderr] DEBUG 22:01:32.803197 main.go:241: control config loaded from /etc/daos/daos_control.yml
22:01:32 DEBUG| [stderr] DEBUG 22:01:32.824699 pool.go:1251: DAOS system list-pools request: *mgmt.ListPoolsReq (sys:"daos_server-2.5.100")
22:01:32 DEBUG| [stderr] DEBUG 22:01:32.825712 rpc.go:278: request hosts: [wolf-101vm2:10001 wolf-101vm3:10001]
22:01:32 DEBUG| [stderr] DEBUG 22:01:32.942515 response.go:168: wolf-101vm2:10001: *mgmt.ListPoolsResp6 0 pools:
22:01:32 DEBUG| [stdout] {
22:01:32 DEBUG| [stdout] "response": {
22:01:32 DEBUG| [stdout] "status": 0,
22:01:32 DEBUG| [stdout] "pools": null
22:01:32 DEBUG| [stdout] },
22:01:32 DEBUG| [stdout] "error": null,
22:01:32 DEBUG| [stdout] "status": 0
22:01:32 DEBUG| [stdout] }
22:01:33 INFO | Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j pool list' finished with 0 after 1.1709704399108887s
22:01:33 INFO | Servers are configured to run across multiple test variants, not stopping
22:01:33 INFO | Removing temporary test files in /var/tmp/daos_testing/test_telemetry_list from wolf-101vm[1-4]
22:01:33 DEBUG| Running on wolf-101vm[1-4] with a 120 second timeout: sudo -n rm -fr /var/tmp/daos_testing/test_telemetry_list
22:01:34 DEBUG| wolf-101vm[1-4] (rc=0): <no output>
22:01:34 DEBUG| DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
22:01:34 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data sources: variant, test, file)
22:01:34 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data sources: variant, test, file)
22:01:34 ERROR| Traceback (most recent call last):
22:01:34 ERROR| File "/usr/lib/python3.6/site-packages/avocado/core/test.py", line 856, in _run_avocado
raise test_exception
22:01:34 ERROR| File "/usr/lib/python3.6/site-packages/avocado/core/test.py", line 762, in _run_avocado
testMethod()
22:01:34 ERROR| File "/usr/lib/daos/TESTING/ftest/control/dmg_telemetry_basic.py", line 84, in test_telemetry_list
self.verify_telemetry_list()
22:01:34 ERROR| File "/usr/lib/daos/TESTING/ftest/util/telemetry_test_base.py", line 82, in verify_telemetry_list
self.fail("\n".join(errors))
22:01:34 ERROR| File "/usr/lib/daos/TESTING/ftest/util/apricot/apricot/test.py", line 1374, in fail
self.dump_engines_stacks("Test has failed")
22:01:34 ERROR| File "/usr/lib/daos/TESTING/ftest/util/apricot/apricot/test.py", line 1363, in dump_engines_stacks
dump_engines_stacks(self.hostlist_servers)
22:01:34 ERROR| File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 827, in dump_engines_stacks
result = pcmd(hosts, "; ".join(commands), verbose, timeout, None)
22:01:34 ERROR| File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 594, in pcmd
results = run_pcmd(hosts, command, verbose, timeout, expect_rc)
22:01:34 ERROR| File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 449, in run_pcmd
task = run_task(hosts, command, timeout)
22:01:34 ERROR| File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 312, in run_task
task.run(**kwargs)
22:01:34 ERROR| File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 877, in run
self.resume(timeout)
22:01:34 ERROR| File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 831, in resume
self._resume()
22:01:34 ERROR| File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 794, in _resume
self._run(self.timeout)
22:01:34 ERROR| File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 404, in _run
self._engine.run(timeout)
22:01:34 ERROR| File "/usr/lib/python3.6/site-packages/ClusterShell/Engine/Engine.py", line 723, in run
self.runloop(timeout)
22:01:34 ERROR| File "/usr/lib/python3.6/site-packages/ClusterShell/Engine/EPoll.py", line 122, in runloop
evlist = self.epolling.poll(poll_timeo)
22:01:34 ERROR| File "/usr/lib/python3.6/site-packages/avocado/plugins/runner.py", line 77, in sigterm_handler
raise RuntimeError("Test interrupted by SIGTERM")
22:01:34 ERROR| RuntimeError: Test interrupted by SIGTERM
22:01:34 ERROR| ERROR 1-./control/dmg_telemetry_basic.py:TestWithTelemetryBasic.test_telemetry_list;run-container-hosts-pool-server_config-engines-0-storage-0-test-timeouts-dadb -> RuntimeError: Test interrupted by SIGTERM
22:01:34 INFO |
Runner error occurred: Timeout reached
Original status: ERROR
{'name': '1-./control/dmg_telemetry_basic.py:TestWithTelemetryBasic.test_telemetry_list;run-container-hosts-pool-server_config-engines-0-storage-0-test-timeouts-dadb', 'logdir': '/var/tmp/ftest/avocado/job-results/job-2024-02-21T22.00-952da71/test-results/1-._control_dmg_telemetry_basic.py_TestWithTelemetryBasic.test_telemetry_list_run-container-hosts-pool-server_config-engines-0-storage-0-test-timeouts-dadb', 'logfile': '/var/tmp/ftest/avocado/job-results/job-2024-02-21T22.00-952da71/test-results/1-._control_dmg_telemetry_basic.py_TestWithTelemetryBasic.test_telemetry_list_run-container-hosts-pool-server_config-engines-0-storage-0-test-timeouts-dadb/debug.log', 'status': 'ERROR', 'running': False, 'paused': False, 'time_start': 1708552800.8852448, 'time_elapsed': 93.31788730621338, 'time_end': 1708552894.2031322, 'fail_reason': 'Test interrupted by SIGTERM', 'fail_class': 'RuntimeError', 'traceback': 'Traceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/avocado/core/test.py", line 856, in _run_avocado\n raise test_exception\n File "/usr/lib/python3.6/site-packages/avocado/core/test.py", line 762, in _run_avocado\n testMethod()\n File "/usr/lib/daos/TESTING/ftest/control/dmg_telemetry_basic.py", line 84, in test_telemetry_list\n self.verify_telemetry_list()\n File "/usr/lib/daos/TESTING/ftest/util/telemetry_test_base.py", line 82, in verify_telemetry_list\n self.fail("\\n".join(errors))\n File "/usr/lib/daos/TESTING/ftest/util/apricot/apricot/test.py", line 1374, in fail\n self.dump_engines_stacks("Test has failed")\n File "/usr/lib/daos/TESTING/ftest/util/apricot/apricot/test.py", line 1363, in dump_engines_stacks\n dump_engines_stacks(self.hostlist_servers)\n File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 827, in dump_engines_stacks\n result = pcmd(hosts, "; ".join(commands), verbose, timeout, None)\n File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 594, in pcmd\n results = run_pcmd(hosts, command, verbose, timeout, expect_rc)\n File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 449, in run_pcmd\n task = run_task(hosts, command, timeout)\n File "/usr/lib/daos/TESTING/ftest/util/general_utils.py", line 312, in run_task\n task.run(**kwargs)\n File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 877, in run\n self.resume(timeout)\n File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 831, in resume\n self._resume()\n File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 794, in _resume\n self._run(self.timeout)\n File "/usr/lib/python3.6/site-packages/ClusterShell/Task.py", line 404, in _run\n self._engine.run(timeout)\n File "/usr/lib/python3.6/site-packages/ClusterShell/Engine/Engine.py", line 723, in run\n self.runloop(timeout)\n File "/usr/lib/python3.6/site-packages/ClusterShell/Engine/EPoll.py", line 122, in runloop\n evlist = self.epolling.poll(poll_timeo)\n File "/usr/lib/python3.6/site-packages/avocado/plugins/runner.py", line 77, in sigterm_handler\n raise RuntimeError("Test interrupted by SIGTERM")\nRuntimeError: Test interrupted by SIGTERM\n', 'timeout': 90, 'whiteboard': '', 'phase': 'FINISHED', 'class_name': 'TestWithTelemetryBasic', 'job_logdir': '/var/tmp/ftest/avocado/job-results/job-2024-02-21T22.00-952da71', 'job_unique_id': '952da71e67f9eeb30bc1bd2e53a5eab6e24dd89b', 'params': [('/run/hosts', 'test_servers', 'wolf-101vm[2-3]'), ('/run/hosts', 'test_clients', 'wolf-101vm4'), ('/run/timeouts', 'test_telemetry_list', 60), ('/run/timeouts', 'test_container_telemetry', 230), ('/run/server_config', 'name', 'daos_server'), ('/run/server_config', 'engines_per_host', 1), ('/run/server_config', 'system_ram_reserved', 1), ('/run/server_config/engines/0', 'targets', 4), ('/run/server_config/engines/0', 'nr_xs_helpers', 0), ('/run/server_config/engines/0/storage/0', 'class', 'ram'), ('/run/server_config/engines/0/storage/0', 'scm_mount', '/mnt/daos'), ('/run/pool', 'scm_size', '2G'), ('/run/pool', 'control_method', 'dmg'), ('/run/container', 'control_method', 'daos'), ('/run/test', 'container_qty', 5), ('/run/test', 'open_close_qty', 3)]}
Loading