DAOS-7203 control: Add histogram support to Prometheus exporter
GitHub Actions / Functional Hardware Medium Test Results (old)
failed
Feb 22, 2024 in 0s
1 fail, 26 skipped, 103 pass in 2h 12m 10s
130 tests 103 ✅ 2h 12m 10s ⏱️
34 suites 26 💤
34 files 1 ❌
Results for commit 427bb06.
Annotations
Check warning on line 0 in FTEST_control.TestWithTelemetryNvme
github-actions / Functional Hardware Medium Test Results (old)
2-./control/dmg_telemetry_nvme.py:TestWithTelemetryNvme.test_telemetry_list_nvme;run-hosts-server_config-engines-0-storage-0-1-telemetry_metrics-bdb4 (FTEST_control.TestWithTelemetryNvme) failed
Functional Hardware Medium/control/dmg_telemetry_nvme.py/results.xml [took 44s]
Raw output
Symmetric difference found in telemetry metric names on wolf-138 Symmetric difference found in telemetry metric names on wolf-139
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/avocado/core/test.py", line 856, in _run_avocado
raise test_exception
File "/usr/lib/python3.6/site-packages/avocado/core/test.py", line 762, in _run_avocado
testMethod()
File "/usr/lib/daos/TESTING/ftest/control/dmg_telemetry_nvme.py", line 75, in test_telemetry_list_nvme
self.verify_telemetry_list()
File "/usr/lib/daos/TESTING/ftest/util/telemetry_test_base.py", line 82, in verify_telemetry_list
self.fail("\n".join(errors))
File "/usr/lib/daos/TESTING/ftest/util/apricot/apricot/test.py", line 1375, in fail
super().fail(message)
File "/usr/lib/python3.6/site-packages/avocado/core/test.py", line 953, in fail
raise exceptions.TestFail(message)
avocado.core.exceptions.TestFail: Symmetric difference found in telemetry metric names on wolf-138
Symmetric difference found in telemetry metric names on wolf-139
08:29:33 INFO | *** SETUP running on <LinuxDistro: name=rocky, version=8, release=8, arch=x86_64> ***
08:29:33 INFO | Test.random seed = 8813242062464701771
08:29:33 DEBUG| Shared test directory: /mnt/share/tmp.EaqGisIQgx
08:29:33 DEBUG| Common test directory: /var/tmp/daos_testing/test_telemetry_list_nvme
08:29:33 DEBUG| PARAMS (key=fault_list, path=/run/faults/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=start_agents_once, path=/run/setup/*, default=True) => True
08:29:33 DEBUG| PARAMS (key=start_servers_once, path=/run/setup/*, default=True) => True
08:29:33 DEBUG| PARAMS (key=server_manager_class, path=/run/setup/*, default=Systemctl) => 'Systemctl'
08:29:33 DEBUG| PARAMS (key=agent_manager_class, path=/run/setup/*, default=Systemctl) => 'Systemctl'
08:29:33 DEBUG| PARAMS (key=start_servers, path=/run/setup/*, default=True) => True
08:29:33 DEBUG| PARAMS (key=start_agents, path=/run/setup/*, default=True) => True
08:29:33 DEBUG| PARAMS (key=slurm_exclude_servers, path=/run/setup/*, default=False) => False
08:29:33 DEBUG| PARAMS (key=name, path=/server_config/, default=daos_server) => 'daos_server'
08:29:33 DEBUG| PARAMS (key=server_config_namespace, path=/run/setup/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=manager_class, path=/, default=Orterun) => 'Orterun'
08:29:33 DEBUG| PARAMS (key=test_servers, path=/run/hosts/*, default=None) => 'wolf-[138-139]'
08:29:33 DEBUG| PARAMS (key=server_partition, path=/run/hosts/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=server_reservation, path=/run/hosts/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=test_clients, path=/run/hosts/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=client_partition, path=/run/hosts/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=client_reservation, path=/run/hosts/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=access_points_qty, path=/run/setup/*, default=1) => 1
08:29:33 DEBUG| PARAMS (key=access_points, path=/run/setup/*, default=wolf-138) => <ClusterShell.NodeSet.NodeSet object at 0x7f10ecf60128>
08:29:33 DEBUG| PARAMS (key=access_points_suffix, path=/run/setup/*, default=None) => None
08:29:33 INFO | ----------------------------------------------------------------------------------------------------
08:29:33 INFO | --- HOST INFORMATION ---
08:29:33 INFO | servers: wolf-[138-139]
08:29:33 INFO | clients:
08:29:33 INFO | server_partition: None
08:29:33 INFO | client_partition: None
08:29:33 INFO | server_reservation: None
08:29:33 INFO | client_reservation: None
08:29:33 INFO | access_points: wolf-138
08:29:33 INFO | ----------------------------------------------------------------------------------------------------
08:29:33 DEBUG| Common test directory (/var/tmp/daos_testing/test_telemetry_list_nvme) contents:
08:29:33 INFO | ----------------------------------------------------------------------------------------------------
08:29:33 DEBUG| --- SETTING UP SERVER GROUPS: {'daos_server': {'hosts': <ClusterShell.NodeSet.NodeSet object at 0x7f10ecf606d8>, 'access_points': <ClusterShell.NodeSet.NodeSet object at 0x7f10ecf600b8>, 'svr_config_file': None, 'dmg_config_file': None, 'svr_config_temp': None, 'dmg_config_temp': None}} ---
08:29:33 INFO | ----------------------------------------------------------------------------------------------------
08:29:33 INFO | --- CONFIGURING SERVER MANAGER ---
08:29:33 DEBUG| PARAMS (key=collect_log_timeout, path=/run/server_manager/*, default=120) => 120
08:29:33 DEBUG| PARAMS (key=storage_format_timeout, path=/run/server_manager/*, default=40) => 40
08:29:33 DEBUG| PARAMS (key=storage_prepare_timeout, path=/run/server_manager/*, default=40) => 40
08:29:33 DEBUG| PARAMS (key=storage_reset_timeout, path=/run/server_manager/*, default=120) => 120
08:29:33 DEBUG| PARAMS (key=env_vars, path=/run/client/*, default=[]) => []
08:29:33 DEBUG| PARAMS (key=env_vars, path=/run/systemctl/*, default=[]) => []
08:29:33 DEBUG| PARAMS (key=allow_proxy, path=/run/daos_server/*, default=False) => False
08:29:33 DEBUG| PARAMS (key=config, path=/run/daos_server/*, default=/etc/daos/daos_server.yml) => '/etc/daos/daos_server.yml'
08:29:33 DEBUG| PARAMS (key=debug, path=/run/daos_server/*, default=True) => True
08:29:33 DEBUG| PARAMS (key=json, path=/run/daos_server/*, default=False) => False
08:29:33 DEBUG| PARAMS (key=json_logs, path=/run/daos_server/*, default=False) => False
08:29:33 DEBUG| PARAMS (key=pattern_timeout, path=/run/daos_server/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=daos_server_sub_command, path=/run/daos_server/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=env_vars, path=/run/client/*, default=[]) => []
08:29:33 DEBUG| PARAMS (key=env_vars, path=/run/daos_server/*, default=[]) => []
08:29:33 DEBUG| PARAMS (key=firstcore, path=/run/daos_server/start/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=group, path=/run/daos_server/start/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=insecure, path=/run/daos_server/start/*, default=False) => False
08:29:33 DEBUG| PARAMS (key=modules, path=/run/daos_server/start/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=port, path=/run/daos_server/start/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=recreate, path=/run/daos_server/start/*, default=False) => False
08:29:33 DEBUG| PARAMS (key=sock_dir, path=/run/daos_server/start/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=storage, path=/run/daos_server/start/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=targets, path=/run/daos_server/start/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=xshelpernr, path=/run/daos_server/start/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=bdev_exclude, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=client_env_vars, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=control_log_file, path=/run/server_config/*, default=daos_control.log) => 'daos_control.log'
08:29:33 DEBUG| Added the directory: control_log_file => /var/tmp/daos_testing/daos_control.log
08:29:33 DEBUG| PARAMS (key=control_log_mask, path=/run/server_config/*, default=DEBUG) => 'DEBUG'
08:29:33 DEBUG| PARAMS (key=core_dump_filter, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=crt_ctx_share_addr, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=crt_timeout, path=/run/server_config/*, default=10) => 10
08:29:33 DEBUG| PARAMS (key=disable_hugepages, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=disable_srx, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=disable_vfio, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=disable_vmd, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=enable_hotplug, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=engines_per_host, path=/run/server_config/*, default=0) => 1
08:29:33 DEBUG| PARAMS (key=fabric_auth_key, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=fault_cb, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=fault_path, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=group_name, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=helper_log_file, path=/run/server_config/*, default=daos_server_helper.log) => 'daos_server_helper.log'
08:29:33 DEBUG| Added the directory: helper_log_file => /var/tmp/daos_testing/daos_server_helper.log
08:29:33 DEBUG| PARAMS (key=hyperthreads, path=/run/server_config/*, default=False) => False
08:29:33 DEBUG| PARAMS (key=nr_hugepages, path=/run/server_config/*, default=0) => 0
08:29:33 DEBUG| PARAMS (key=provider, path=/run/server_config/*, default=ofi+verbs;ofi_rxm) => 'ofi+verbs;ofi_rxm'
08:29:33 DEBUG| PARAMS (key=socket_dir, path=/run/server_config/*, default=/var/run/daos_server) => '/var/run/daos_server'
08:29:33 DEBUG| PARAMS (key=system_ram_reserved, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=telemetry_port, path=/run/server_config/*, default=9191) => 9191
08:29:33 DEBUG| PARAMS (key=user_name, path=/run/server_config/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=access_points, path=/run/common_config/*, default=['localhost']) => ['localhost']
08:29:33 DEBUG| PARAMS (key=name, path=/run/common_config/*, default=daos_server) => 'daos_server'
08:29:33 DEBUG| PARAMS (key=port, path=/run/common_config/*, default=10001) => 10001
08:29:33 DEBUG| PARAMS (key=allow_insecure, path=/run/server_config/transport_config/*, default=False) => False
08:29:33 DEBUG| PARAMS (key=ca_cert, path=/run/server_config/transport_config/*, default=daosCA.crt) => 'daosCA.crt'
08:29:33 DEBUG| Added the directory: ca_cert => /etc/daos/certs/daosCA.crt
08:29:33 DEBUG| PARAMS (key=cert, path=/run/server_config/transport_config/*, default=server.crt) => 'server.crt'
08:29:33 DEBUG| Added the directory: cert => /etc/daos/certs/server.crt
08:29:33 DEBUG| PARAMS (key=client_cert_dir, path=/run/server_config/transport_config/*, default=clients) => 'clients'
08:29:33 DEBUG| Added the directory: client_cert_dir => /etc/daos/certs/clients
08:29:33 DEBUG| PARAMS (key=key, path=/run/server_config/transport_config/*, default=server.key) => 'server.key'
08:29:33 DEBUG| Added the directory: key => /etc/daos/certs/server.key
08:29:33 DEBUG| PARAMS (key=device, path=/run/server_config/control_metadata/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=path, path=/run/server_config/control_metadata/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=crt_ctx_share_addr, path=/run/server_config/engines/0/*, default=0) => 0
08:29:33 DEBUG| PARAMS (key=env_vars, path=/run/server_config/engines/0/*, default=['ABT_ENV_MAX_NUM_XSTREAMS=100', 'ABT_MAX_NUM_XSTREAMS=100', 'DAOS_MD_CAP=1024', 'DAOS_SCHED_WATCHDOG_ALL=1', 'DD_MASK=mgmt,io,md,epc,rebuild', 'D_LOG_FILE_APPEND_PID=1', 'COVFILE=/tmp/test.cov', 'FI_OFI_RXM_USE_SRX=1']) => ['ABT_ENV_MAX_NUM_XSTREAMS=100', 'ABT_MAX_NUM_XSTREAMS=100', 'DAOS_MD_CAP=1024', 'DAOS_SCHED_WATCHDOG_ALL=1', 'DD_MASK=mgmt,io,md,epc,rebuild', 'D_LOG_FILE_APPEND_PID=1', 'COVFILE=/tmp/test.cov', 'FI_OFI_RXM_USE_SRX=1']
08:29:33 DEBUG| PARAMS (key=fabric_iface, path=/run/server_config/engines/0/*, default=ib0) => 'ib0'
08:29:33 DEBUG| PARAMS (key=fabric_iface_port, path=/run/server_config/engines/0/*, default=31416) => 31416
08:29:33 DEBUG| PARAMS (key=first_core, path=/run/server_config/engines/0/*, default=0) => 0
08:29:33 DEBUG| PARAMS (key=log_file, path=/run/server_config/engines/0/*, default=daos_server.log) => 'daos_server.log'
08:29:33 DEBUG| Added the directory: log_file => /var/tmp/daos_testing/daos_server.log
08:29:33 DEBUG| PARAMS (key=log_mask, path=/run/server_config/engines/0/*, default=INFO) => 'INFO'
08:29:33 DEBUG| PARAMS (key=nr_xs_helpers, path=/run/server_config/engines/0/*, default=4) => 4
08:29:33 DEBUG| PARAMS (key=pinned_numa_node, path=/run/server_config/engines/0/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=targets, path=/run/server_config/engines/0/*, default=8) => 8
08:29:33 DEBUG| Checking for storage tier in /run/server_config/engines/0/storage/0/*
08:29:33 DEBUG| PARAMS (key=class, path=/run/server_config/engines/0/storage/0/*, default=None) => 'dcpm'
08:29:33 DEBUG| PARAMS (key=scm_list, path=/run/server_config/engines/0/storage/0/*, default=None) => ['/dev/pmem0']
08:29:33 DEBUG| PARAMS (key=scm_mount, path=/run/server_config/engines/0/storage/0/*, default=None) => '/mnt/daos0'
08:29:33 DEBUG| PARAMS (key=scm_size, path=/run/server_config/engines/0/storage/0/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=scm_hugepages_disabled, path=/run/server_config/engines/0/storage/0/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=bdev_list, path=/run/server_config/engines/0/storage/0/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=bdev_number, path=/run/server_config/engines/0/storage/0/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=bdev_size, path=/run/server_config/engines/0/storage/0/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=bdev_roles, path=/run/server_config/engines/0/storage/0/*, default=None) => None
08:29:33 DEBUG| Checking for storage tier in /run/server_config/engines/0/storage/1/*
08:29:33 DEBUG| PARAMS (key=class, path=/run/server_config/engines/0/storage/1/*, default=None) => 'nvme'
08:29:33 DEBUG| PARAMS (key=scm_list, path=/run/server_config/engines/0/storage/1/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=scm_mount, path=/run/server_config/engines/0/storage/1/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=scm_size, path=/run/server_config/engines/0/storage/1/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=scm_hugepages_disabled, path=/run/server_config/engines/0/storage/1/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=bdev_list, path=/run/server_config/engines/0/storage/1/*, default=None) => ['0000:85:05.5', '0000:d7:05.5']
08:29:33 DEBUG| PARAMS (key=bdev_number, path=/run/server_config/engines/0/storage/1/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=bdev_size, path=/run/server_config/engines/0/storage/1/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=bdev_roles, path=/run/server_config/engines/0/storage/1/*, default=None) => None
08:29:33 DEBUG| Checking for storage tier in /run/server_config/engines/0/storage/2/*
08:29:33 DEBUG| PARAMS (key=class, path=/run/server_config/engines/0/storage/2/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=_hostlist, path=/run/dmg/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=configpath, path=/run/dmg/*, default=/etc/daos/daos_control.yml) => '/etc/daos/daos_control.yml'
08:29:33 DEBUG| PARAMS (key=debug, path=/run/dmg/*, default=True) => True
08:29:33 DEBUG| PARAMS (key=hostfile, path=/run/dmg/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=insecure, path=/run/dmg/*, default=False) => False
08:29:33 DEBUG| PARAMS (key=json, path=/run/dmg/*, default=False) => False
08:29:33 DEBUG| PARAMS (key=pattern_timeout, path=/run/dmg/*, default=10) => 10
08:29:33 DEBUG| PARAMS (key=dmg_sub_command, path=/run/dmg/*, default=None) => None
08:29:33 DEBUG| PARAMS (key=env_vars, path=/run/client/*, default=[]) => []
08:29:33 DEBUG| PARAMS (key=env_vars, path=/run/dmg/*, default=[]) => []
08:29:33 DEBUG| PARAMS (key=hostlist, path=/run/dmg/*, default=localhost) => 'localhost'
08:29:33 DEBUG| PARAMS (key=name, path=/run/dmg/*, default=daos_server) => 'daos_server'
08:29:33 DEBUG| PARAMS (key=port, path=/run/dmg/*, default=10001) => 10001
08:29:33 DEBUG| PARAMS (key=allow_insecure, path=/run/dmg/transport_config/*, default=False) => False
08:29:33 DEBUG| PARAMS (key=ca_cert, path=/run/dmg/transport_config/*, default=daosCA.crt) => 'daosCA.crt'
08:29:33 DEBUG| Added the directory: ca_cert => /etc/daos/certs/daosCA.crt
08:29:33 DEBUG| PARAMS (key=cert, path=/run/dmg/transport_config/*, default=admin.crt) => 'admin.crt'
08:29:33 DEBUG| Added the directory: cert => /etc/daos/certs/admin.crt
08:29:33 DEBUG| PARAMS (key=key, path=/run/dmg/transport_config/*, default=admin.key) => 'admin.key'
08:29:33 DEBUG| Added the directory: key => /etc/daos/certs/admin.key
08:29:33 DEBUG| Updated param access_points => ['wolf-138']
08:29:33 INFO | ----------------------------------------------------------------------------------------------------
08:29:33 INFO | --- VERIFYING STATES OF 1 SERVERS GROUP ---
08:29:33 DEBUG| Copying certificates for dmg:
08:29:34 DEBUG| /var/tmp/daos_testing/daosCA/certs/daosCA.crt -> /etc/daos/certs/daosCA.crt
08:29:35 DEBUG| /var/tmp/daos_testing/daosCA/certs/admin.crt -> /etc/daos/certs/admin.crt
08:29:35 DEBUG| /var/tmp/daos_testing/daosCA/certs/admin.key -> /etc/daos/certs/admin.key
08:29:36 DEBUG| Copied certificates for dmg (in /etc/daos/certs):
08:29:37 DEBUG| wolf-137: total 40
08:29:37 DEBUG| wolf-137: drwxr-xr-x 3 root root 4096 Feb 22 06:47 .
08:29:37 DEBUG| wolf-137: drwxr-xr-x 3 root root 4096 Feb 22 08:29 ..
08:29:37 DEBUG| wolf-137: -rw-r--r-- 1 jenkins jenkins 5238 Feb 22 08:29 admin.crt
08:29:37 DEBUG| wolf-137: -r-------- 1 jenkins jenkins 2459 Feb 22 08:29 admin.key
08:29:37 DEBUG| wolf-137: -rw-r--r-- 1 daos_agent daos_agent 5238 Feb 22 08:29 agent.crt
08:29:37 DEBUG| wolf-137: -r-------- 1 daos_agent daos_agent 2459 Feb 22 08:29 agent.key
08:29:37 DEBUG| wolf-137: drwx------ 2 daos_server daos_server 4096 Feb 21 21:29 clients
08:29:37 DEBUG| wolf-137: -rw-r--r-- 1 jenkins jenkins 1436 Feb 22 08:29 daosCA.crt
08:29:37 DEBUG| Updated param dmg.yaml.hostlist => ['wolf-138', 'wolf-139']
08:29:37 DEBUG| Updated param ranks => None
08:29:37 DEBUG| Updated param verbose => True
08:29:37 INFO | Writing yaml configuration file /var/tmp/daos_testing/test_telemetry_list_nvme/test_daos_server_dmg.yaml
08:29:37 INFO | Copying /var/tmp/daos_testing/test_telemetry_list_nvme/test_daos_server_dmg.yaml yaml configuration file to /etc/daos/daos_control.yml on wolf-137
08:29:37 INFO | Command environment vars:
{}
08:29:37 INFO | Running '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j system query --verbose'
08:29:37 DEBUG| [stderr] DEBUG 08:29:37.830042 main.go:209: debug output enabled
08:29:37 DEBUG| [stderr] DEBUG 08:29:37.830469 main.go:241: control config loaded from /etc/daos/daos_control.yml
08:29:37 DEBUG| [stderr] DEBUG 08:29:37.848655 system.go:279: DAOS system query request: *mgmt.SystemQueryReq (sys:"daos_server-2.5.100" state_mask:65535)
08:29:37 DEBUG| [stderr] DEBUG 08:29:37.849309 rpc.go:278: request hosts: [wolf-138:10001 wolf-139:10001]
08:29:37 DEBUG| [stderr] DEBUG 08:29:37.952867 response.go:168: wolf-138:10001: *mgmt.SystemQueryResp@5 joined:0-1
08:29:37 DEBUG| [stdout] {
08:29:37 DEBUG| [stdout] "response": {
08:29:37 DEBUG| [stdout] "members": [
08:29:37 DEBUG| [stdout] {
08:29:37 DEBUG| [stdout] "addr": "10.8.1.138:10001",
08:29:37 DEBUG| [stdout] "state": "joined",
08:29:37 DEBUG| [stdout] "fault_domain": "/wolf-138.wolf.hpdd.intel.com",
08:29:37 DEBUG| [stdout] "rank": 0,
08:29:37 DEBUG| [stdout] "incarnation": 1586101373098000384,
08:29:37 DEBUG| [stdout] "uuid": "4b34991b-1f40-4294-8461-e238334ebb64",
08:29:37 DEBUG| [stdout] "fabric_uri": "ofi+verbs;ofi_rxm://192.168.100.138:31416",
08:29:37 DEBUG| [stdout] "fabric_contexts": 14,
08:29:37 DEBUG| [stdout] "info": "",
08:29:37 DEBUG| [stdout] "last_update": "2024-02-22T08:28:57.613312472Z"
08:29:37 DEBUG| [stdout] },
08:29:37 DEBUG| [stdout] {
08:29:37 DEBUG| [stdout] "addr": "10.8.1.139:10001",
08:29:37 DEBUG| [stdout] "state": "joined",
08:29:37 DEBUG| [stdout] "fault_domain": "/wolf-139.wolf.hpdd.intel.com",
08:29:37 DEBUG| [stdout] "rank": 1,
08:29:37 DEBUG| [stdout] "incarnation": 1586101373121069056,
08:29:37 DEBUG| [stdout] "uuid": "357f9bfc-dd72-4e5b-a8f4-0f92c396faa3",
08:29:37 DEBUG| [stdout] "fabric_uri": "ofi+verbs;ofi_rxm://192.168.100.139:31416",
08:29:37 DEBUG| [stdout] "fabric_contexts": 14,
08:29:37 DEBUG| [stdout] "info": "",
08:29:37 DEBUG| [stdout] "last_update": "2024-02-22T08:28:57.610914194Z"
08:29:37 DEBUG| [stdout] }
08:29:37 DEBUG| [stdout] ]
08:29:37 DEBUG| [stdout] },
08:29:37 DEBUG| [stdout] "error": null,
08:29:37 DEBUG| [stdout] "status": 0
08:29:37 DEBUG| [stdout] }
08:29:38 INFO | Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j system query --verbose' finished with 0 after 1.148012399673462s
08:29:38 INFO | <SERVER> Assigning expected server states: {0: {'uuid': '4b34991b-1f40-4294-8461-e238334ebb64', 'host': 'wolf-138', 'state': 'joined'}, 1: {'uuid': '357f9bfc-dd72-4e5b-a8f4-0f92c396faa3', 'host': 'wolf-139', 'state': 'joined'}}
08:29:38 INFO | <SERVER> Verifying server states: group=daos_server, hosts=wolf-[138-139]
08:29:38 INFO | Rank Host UUID Expected State Current State Result
08:29:38 INFO | ---- --------------- ------------------------------------ ---------------------- -------------- ------
08:29:38 INFO | 0 wolf-138 4b34991b-1f40-4294-8461-e238334ebb64 joined joined PASS
08:29:38 INFO | 1 wolf-139 357f9bfc-dd72-4e5b-a8f4-0f92c396faa3 joined joined PASS
08:29:38 INFO | All 1 groups(s) of servers currently running
08:29:38 INFO | ----------------------------------------------------------------------------------------------------
08:29:38 INFO | --- SERVER INFORMATION ---
08:29:38 DEBUG| Determining the current daos_server log files
08:29:39 DEBUG| Engine log files per host
08:29:39 DEBUG| wolf-138:
08:29:39 DEBUG| /var/tmp/daos_testing/daos_server.log.192825
08:29:39 DEBUG| wolf-139:
08:29:39 DEBUG| /var/tmp/daos_testing/daos_server.log.453468
08:29:39 DEBUG| Updated param rank => None
08:29:39 DEBUG| Updated param health => False
08:29:39 INFO | Command environment vars:
{}
08:29:39 INFO | Running '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j storage query list-devices'
08:29:39 DEBUG| [stderr] DEBUG 08:29:39.572220 main.go:209: debug output enabled
08:29:39 DEBUG| [stderr] DEBUG 08:29:39.572690 main.go:241: control config loaded from /etc/daos/daos_control.yml
08:29:39 DEBUG| [stderr] DEBUG 08:29:39.590508 server_meta.go:152: SmdQuery() called with request &{unaryRequest:{request:{timeout:0 deadline:{wall:0 ext:0 loc:<nil>} Sys: HostList:[]} rpc:<nil>} OmitDevices:false OmitPools:true IncludeBioHealth:false UUID: Rank:4294967295 FaultyDevsOnly:false}
08:29:39 DEBUG| [stderr] DEBUG 08:29:39.591145 rpc.go:278: request hosts: [wolf-138:10001 wolf-139:10001]
08:29:39 DEBUG| [stdout] {
08:29:39 DEBUG| [stdout] "response": {
08:29:39 DEBUG| [stdout] "host_errors": {},
08:29:39 DEBUG| [stdout] "host_storage_map": {
08:29:39 DEBUG| [stdout] "12730372067871850930": {
08:29:39 DEBUG| [stdout] "storage": {
08:29:39 DEBUG| [stdout] "nvme_devices": null,
08:29:39 DEBUG| [stdout] "scm_modules": null,
08:29:39 DEBUG| [stdout] "scm_namespaces": null,
08:29:39 DEBUG| [stdout] "scm_mount_points": null,
08:29:39 DEBUG| [stdout] "smd_info": {
08:29:39 DEBUG| [stdout] "devices": [
08:29:39 DEBUG| [stdout] {
08:29:39 DEBUG| [stdout] "role_bits": 0,
08:29:39 DEBUG| [stdout] "uuid": "32ccc33b-09a8-431a-821e-ebf90afbb20b",
08:29:39 DEBUG| [stdout] "tgt_ids": [
08:29:39 DEBUG| [stdout] 0,
08:29:39 DEBUG| [stdout] 2,
08:29:39 DEBUG| [stdout] 4,
08:29:39 DEBUG| [stdout] 6
08:29:39 DEBUG| [stdout] ],
08:29:39 DEBUG| [stdout] "rank": 0,
08:29:39 DEBUG| [stdout] "total_bytes": 0,
08:29:39 DEBUG| [stdout] "avail_bytes": 0,
08:29:39 DEBUG| [stdout] "usable_bytes": 0,
08:29:39 DEBUG| [stdout] "cluster_size": 0,
08:29:39 DEBUG| [stdout] "meta_size": 0,
08:29:39 DEBUG| [stdout] "meta_wal_size": 0,
08:29:39 DEBUG| [stdout] "rdb_size": 0,
08:29:39 DEBUG| [stdout] "rdb_wal_size": 0,
08:29:39 DEBUG| [stdout] "roles": "NA",
08:29:39 DEBUG| [stdout] "has_sys_xs": false,
08:29:39 DEBUG| [stdout] "ctrlr": {
08:29:39 DEBUG| [stdout] "info": "",
08:29:39 DEBUG| [stdout] "model": "INTEL SSDPED1K750GA",
08:29:39 DEBUG| [stdout] "serial": "PHKS750500HU750BGN",
08:29:39 DEBUG| [stdout] "pci_addr": "850505:01:00.0",
08:29:39 DEBUG| [stdout] "fw_rev": "E2010475",
08:29:39 DEBUG| [stdout] "vendor_id": "0x8086",
08:29:39 DEBUG| [stdout] "pci_type": "",
08:29:39 DEBUG| [stdout] "socket_id": 1,
08:29:39 DEBUG| [stdout] "health_stats": null,
08:29:39 DEBUG| [stdout] "namespaces": [
08:29:39 DEBUG| [stdout] {
08:29:39 DEBUG| [stdout] "id": 1,
08:29:39 DEBUG| [stdout] "size": 750156374016
08:29:39 DEBUG| [stdout] }
08:29:39 DEBUG| [stdout] ],
08:29:39 DEBUG| [stdout] "smd_devices": null,
08:29:39 DEBUG| [stdout] "dev_state": "NORMAL",
08:29:39 DEBUG| [stdout] "led_state": "OFF"
08:29:39 DEBUG| [stdout] },
08:29:39 DEBUG| [stdout] "ctrlr_namespace_id": 1
08:29:39 DEBUG| [stdout] },
08:29:39 DEBUG| [stdout] {
08:29:39 DEBUG| [stdout] "role_bits": 0,
08:29:39 DEBUG| [stdout] "uuid": "b19a8088-068b-4d1a-a0b3-589645d93416",
08:29:39 DEBUG| [stdout] "tgt_ids": [
08:29:39 DEBUG| [stdout] 1,
08:29:39 DEBUG| [stdout] 3,
08:29:39 DEBUG| [stdout] 5,
08:29:39 DEBUG| [stdout] 7
08:29:39 DEBUG| [stdout] ],
08:29:39 DEBUG| [stdout] "rank": 0,
08:29:39 DEBUG| [stdout] "total_bytes": 0,
08:29:39 DEBUG| [stdout] "avail_bytes": 0,
08:29:39 DEBUG| [stdout] "usable_bytes": 0,
08:29:39 DEBUG| [stdout] "cluster_size": 0,
08:29:39 DEBUG| [stdout] "meta_size": 0,
08:29:39 DEBUG| [stdout] "meta_wal_size": 0,
08:29:39 DEBUG| [stdout] "rdb_size": 0,
08:29:39 DEBUG| [stdout] "rdb_wal_size": 0,
08:29:39 DEBUG| [stdout] "roles": "NA",
08:29:39 DEBUG| [stdout] "has_sys_xs": false,
08:29:39 DEBUG| [stdout] "ctrlr": {
08:29:39 DEBUG| [stdout] "info": "",
08:29:39 DEBUG| [stdout] "model": "INTEL SSDPED1K750GA",
08:29:39 DEBUG| [stdout] "serial": "PHKS750500HC750BGN",
08:29:39 DEBUG| [stdout] "pci_addr": "d70505:05:00.0",
08:29:39 DEBUG| [stdout] "fw_rev": "E2010475",
08:29:39 DEBUG| [stdout] "vendor_id": "0x8086",
08:29:39 DEBUG| [stdout] "pci_type": "",
08:29:39 DEBUG| [stdout] "socket_id": 1,
08:29:39 DEBUG| [stdout] "health_stats": null,
08:29:39 DEBUG| [stdout] "namespaces": [
08:29:39 DEBUG| [stdout] {
08:29:39 DEBUG| [stdout] "id": 1,
08:29:39 DEBUG| [stdout] "size": 750156374016
08:29:39 DEBUG| [stdout] }
08:29:39 DEBUG| [stdout] ],
08:29:39 DEBUG| [stdout] "smd_devices": null,
08:29:39 DEBUG| [stdout] "dev_state": "NORMAL",
08:29:39 DEBUG| [stdout] "led_state": "OFF"
08:29:39 DEBUG| [stdout] },
08:29:39 DEBUG| [stdout] "ctrlr_namespace_id": 1
08:29:39 DEBUG| [stdout] }
08:29:39 DEBUG| [stdout] ],
08:29:39 DEBUG| [stdout] "pools": {}
08:29:39 DEBUG| [stdout] },
08:29:39 DEBUG| [stdout] "reboot_required": false,
08:29:39 DEBUG| [stdout] "mem_info": null
08:29:39 DEBUG| [stdout] },
08:29:39 DEBUG| [stdout] "hosts": "wolf-138:10001"
08:29:39 DEBUG| [stdout] },
08:29:39 DEBUG| [stdout] "4696307292159348612": {
08:29:39 DEBUG| [stdout] "storage": {
08:29:39 DEBUG| [stdout] "nvme_devices": null,
08:29:39 DEBUG| [stdout] "scm_modules": null,
08:29:39 DEBUG| [stdout] "scm_namespaces": null,
08:29:39 DEBUG| [stdout] "scm_mount_points": null,
08:29:39 DEBUG| [stdout] "smd_info": {
08:29:39 DEBUG| [stdout] "devices": [
08:29:39 DEBUG| [stdout] {
08:29:39 DEBUG| [stdout] "role_bits": 0,
08:29:39 DEBUG| [stdout] "uuid": "f525ff2a-ce9a-4e16-b17a-9a0802004ebf",
08:29:39 DEBUG| [stdout] "tgt_ids": [
08:29:39 DEBUG| [stdout] 0,
08:29:39 DEBUG| [stdout] 2,
08:29:39 DEBUG| [stdout] 4,
08:29:39 DEBUG| [stdout] 6
08:29:39 DEBUG| [stdout] ],
08:29:39 DEBUG| [stdout] "rank": 1,
08:29:39 DEBUG| [stdout] "total_bytes": 0,
08:29:39 DEBUG| [stdout] "avail_bytes": 0,
08:29:39 DEBUG| [stdout] "usable_bytes": 0,
08:29:39 DEBUG| [stdout] "cluster_size": 0,
08:29:39 DEBUG| [stdout] "meta_size": 0,
08:29:39 DEBUG| [stdout] "meta_wal_size": 0,
08:29:39 DEBUG| [stdout] "rdb_size": 0,
08:29:39 DEBUG| [stdout] "rdb_wal_size": 0,
08:29:39 DEBUG| [stdout] "roles": "NA",
08:29:39 DEBUG| [stdout] "has_sys_xs": false,
08:29:39 DEBUG| [stdout] "ctrlr": {
08:29:39 DEBUG| [stdout] "info": "",
08:29:39 DEBUG| [stdout] "model": "INTEL SSDPED1K750GA",
08:29:39 DEBUG| [stdout] "serial": "PHKS750500AJ750BGN",
08:29:39 DEBUG| [stdout] "pci_addr": "850505:01:00.0",
08:29:39 DEBUG| [stdout] "fw_rev": "E2010475",
08:29:39 DEBUG| [stdout] "vendor_id": "0x8086",
08:29:39 DEBUG| [stdout] "pci_type": "",
08:29:39 DEBUG| [stdout] "socket_id": 1,
08:29:39 DEBUG| [stdout] "health_stats": null,
08:29:39 DEBUG| [stdout] "namespaces": [
08:29:39 DEBUG| [stdout] {
08:29:39 DEBUG| [stdout] "id": 1,
08:29:39 DEBUG| [stdout] "size": 750156374016
08:29:39 DEBUG| [stdout] }
08:29:39 DEBUG| [stdout] ],
08:29:39 DEBUG| [stdout] "smd_devices": null,
08:29:39 DEBUG| [stdout] "dev_state": "NORMAL",
08:29:39 DEBUG| [stdout] "led_state": "OFF"
08:29:39 DEBUG| [stdout] },
08:29:39 DEBUG| [stdout] "ctrlr_namespace_id": 1
08:29:39 DEBUG| [stdout] },
08:29:39 DEBUG| [stdout] {
08:29:39 DEBUG| [stdout] "role_bits": 0,
08:29:39 DEBUG| [stdout] "uuid": "e8270c4b-75ff-41d8-b34c-ae75a52cb496",
08:29:39 DEBUG| [stdout] "tgt_ids": [
08:29:39 DEBUG| [stdout] 1,
08:29:39 DEBUG| [stdout] 3,
08:29:39 DEBUG| [stdout] … {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_cycle_duration_max",
08:29:44 DEBUG| [stdout] "description": "Schedule cycle duration (max value)",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_cycle_duration_mean",
08:29:44 DEBUG| [stdout] "description": "Schedule cycle duration (mean)",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_cycle_duration_min",
08:29:44 DEBUG| [stdout] "description": "Schedule cycle duration (min value)",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_cycle_duration_samples",
08:29:44 DEBUG| [stdout] "description": "Schedule cycle duration (samples)",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_cycle_duration_stddev",
08:29:44 DEBUG| [stdout] "description": "Schedule cycle duration (std dev)",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_cycle_size",
08:29:44 DEBUG| [stdout] "description": "Schedule cycle size",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_cycle_size_max",
08:29:44 DEBUG| [stdout] "description": "Schedule cycle size (max value)",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_cycle_size_mean",
08:29:44 DEBUG| [stdout] "description": "Schedule cycle size (mean)",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_cycle_size_min",
08:29:44 DEBUG| [stdout] "description": "Schedule cycle size (min value)",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_cycle_size_samples",
08:29:44 DEBUG| [stdout] "description": "Schedule cycle size (samples)",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_cycle_size_stddev",
08:29:44 DEBUG| [stdout] "description": "Schedule cycle size (std dev)",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "counter",
08:29:44 DEBUG| [stdout] "name": "engine_sched_relax_time",
08:29:44 DEBUG| [stdout] "description": "Total relaxing time",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_sleep_queue",
08:29:44 DEBUG| [stdout] "description": "Sleep queue length",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "counter",
08:29:44 DEBUG| [stdout] "name": "engine_sched_total_reject",
08:29:44 DEBUG| [stdout] "description": "Total rejected requests",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "counter",
08:29:44 DEBUG| [stdout] "name": "engine_sched_total_time",
08:29:44 DEBUG| [stdout] "description": "Total running time",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_sched_wait_queue",
08:29:44 DEBUG| [stdout] "description": "Wait queue length",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_servicing_at",
08:29:44 DEBUG| [stdout] "description": "Timestamp when the engine became ready",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "engine_started_at",
08:29:44 DEBUG| [stdout] "description": "Timestamp of last engine startup",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "summary",
08:29:44 DEBUG| [stdout] "name": "go_gc_duration_seconds",
08:29:44 DEBUG| [stdout] "description": "A summary of the pause duration of garbage collection cycles.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_goroutines",
08:29:44 DEBUG| [stdout] "description": "Number of goroutines that currently exist.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_info",
08:29:44 DEBUG| [stdout] "description": "Information about the Go environment.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_alloc_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of bytes allocated and still in use.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "counter",
08:29:44 DEBUG| [stdout] "name": "go_memstats_alloc_bytes_total",
08:29:44 DEBUG| [stdout] "description": "Total number of bytes allocated, even if freed.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_buck_hash_sys_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of bytes used by the profiling bucket hash table.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "counter",
08:29:44 DEBUG| [stdout] "name": "go_memstats_frees_total",
08:29:44 DEBUG| [stdout] "description": "Total number of frees.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_gc_sys_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of bytes used for garbage collection system metadata.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_heap_alloc_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of heap bytes allocated and still in use.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_heap_idle_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of heap bytes waiting to be used.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_heap_inuse_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of heap bytes that are in use.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_heap_objects",
08:29:44 DEBUG| [stdout] "description": "Number of allocated objects.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_heap_released_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of heap bytes released to OS.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_heap_sys_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of heap bytes obtained from system.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_last_gc_time_seconds",
08:29:44 DEBUG| [stdout] "description": "Number of seconds since 1970 of last garbage collection.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "counter",
08:29:44 DEBUG| [stdout] "name": "go_memstats_lookups_total",
08:29:44 DEBUG| [stdout] "description": "Total number of pointer lookups.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "counter",
08:29:44 DEBUG| [stdout] "name": "go_memstats_mallocs_total",
08:29:44 DEBUG| [stdout] "description": "Total number of mallocs.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_mcache_inuse_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of bytes in use by mcache structures.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_mcache_sys_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of bytes used for mcache structures obtained from system.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_mspan_inuse_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of bytes in use by mspan structures.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_mspan_sys_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of bytes used for mspan structures obtained from system.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_next_gc_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of heap bytes when next garbage collection will take place.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_other_sys_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of bytes used for other system allocations.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_stack_inuse_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of bytes in use by the stack allocator.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_me
08:29:44 DEBUG| [stdout] mstats_stack_sys_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of bytes obtained from system for stack allocator.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_memstats_sys_bytes",
08:29:44 DEBUG| [stdout] "description": "Number of bytes obtained from system.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "go_threads",
08:29:44 DEBUG| [stdout] "description": "Number of OS threads created.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "counter",
08:29:44 DEBUG| [stdout] "name": "process_cpu_seconds_total",
08:29:44 DEBUG| [stdout] "description": "Total user and system CPU time spent in seconds.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "process_max_fds",
08:29:44 DEBUG| [stdout] "description": "Maximum number of open file descriptors.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "process_open_fds",
08:29:44 DEBUG| [stdout] "description": "Number of open file descriptors.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "process_resident_memory_bytes",
08:29:44 DEBUG| [stdout] "description": "Resident memory size in bytes.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "process_start_time_seconds",
08:29:44 DEBUG| [stdout] "description": "Start time of the process since unix epoch in seconds.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "process_virtual_memory_bytes",
08:29:44 DEBUG| [stdout] "description": "Virtual memory size in bytes.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] {
08:29:44 DEBUG| [stdout] "type": "gauge",
08:29:44 DEBUG| [stdout] "name": "process_virtual_memory_max_bytes",
08:29:44 DEBUG| [stdout] "description": "Maximum amount of virtual memory available in bytes.",
08:29:44 DEBUG| [stdout] "metrics": null
08:29:44 DEBUG| [stdout] }
08:29:44 DEBUG| [stdout] ]
08:29:44 DEBUG| [stdout] },
08:29:44 DEBUG| [stdout] "error": null,
08:29:44 DEBUG| [stdout] "status": 0
08:29:44 DEBUG| [stdout] }
08:29:45 INFO | Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j telemetry metrics list --host-list=wolf-139' finished with 0 after 1.6572105884552002s
08:29:45 INFO | Detected 2/2 telemetry metrics list hosts
08:29:45 INFO | Difference between expected and actual: set()
08:29:45 INFO | Symmetric difference between expected and actual: set()
08:29:45 INFO | wolf-138: detected 425/366 telemetry metric names
08:29:45 INFO | wolf-138: difference between expected and actual: set()
08:29:45 INFO | wolf-138: symmetric difference between expected and actual: {'engine_io_ops_akey_enum_active_samples', 'engine_sched_cycle_size_samples', 'engine_io_ops_obj_punch_latency_samples', 'engine_io_ops_obj_enum_latency_samples', 'engine_io_ops_ec_agg_latency_samples', 'engine_sched_cycle_duration_samples', 'engine_io_ops_recx_enum_latency_samples', 'engine_io_ops_obj_sync_active_samples', 'engine_io_ops_compound_active_samples', 'engine_io_latency_update_samples', 'engine_io_ops_ec_rep_latency_samples', 'engine_io_ops_obj_coll_punch_latency_samples', 'engine_io_ops_fetch_active_samples', 'engine_io_ops_tgt_punch_active_samples', 'engine_io_ops_update_active_samples', 'engine_dmabuff_wal_waiters_samples', 'engine_io_ops_dkey_enum_active_samples', 'engine_io_ops_key_query_active_samples', 'engine_io_latency_vos_fetch_samples', 'engine_io_ops_akey_punch_latency_samples', 'engine_io_ops_obj_coll_punch_active_samples', 'engine_io_dtx_committed_samples', 'engine_io_ops_dkey_punch_latency_samples', 'engine_io_ops_compound_latency_samples', 'engine_io_ops_obj_coll_query_latency_samples', 'engine_io_ops_recx_enum_active_samples', 'engine_io_latency_bio_update_samples', 'engine_io_ops_key2anchor_active_samples', 'engine_io_ops_obj_enum_active_samples', 'engine_io_ops_obj_punch_active_samples', 'engine_io_latency_bio_fetch_samples', 'engine_io_latency_tgt_update_samples', 'engine_io_ops_obj_sync_latency_samples', 'engine_io_ops_migrate_active_samples', 'engine_io_ops_tgt_dkey_punch_latency_samples', 'engine_io_ops_akey_enum_latency_samples', 'engine_io_ops_key2anchor_latency_samples', 'engine_io_latency_bulk_fetch_samples', 'engine_io_dtx_committable_samples', 'engine_io_ops_tgt_punch_latency_samples', 'engine_io_ops_tgt_akey_punch_active_samples', 'engine_io_ops_ec_agg_active_samples', 'engine_io_ops_tgt_akey_punch_latency_samples', 'engine_io_ops_tgt_update_active_samples', 'engine_net_swim_delay_samples', 'engine_io_latency_bulk_update_samples', 'engine_io_ops_tgt_dkey_punch_active_samples', 'engine_io_ops_dkey_punch_active_samples', 'engine_io_latency_vos_update_samples', 'engine_io_ops_key_query_latency_samples', 'engine_io_ops_migrate_latency_samples', 'engine_dmabuff_wal_qd_samples', 'engine_io_ops_akey_punch_active_samples', 'engine_io_ops_obj_coll_query_active_samples', 'engine_dmabuff_wal_sz_samples', 'engine_dmabuff_grab_retries_samples', 'engine_io_ops_dkey_enum_latency_samples', 'engine_io_ops_ec_rep_active_samples', 'engine_io_latency_fetch_samples'}
08:29:45 INFO | wolf-139: detected 425/366 telemetry metric names
08:29:45 INFO | wolf-139: difference between expected and actual: set()
08:29:45 INFO | wolf-139: symmetric difference between expected and actual: {'engine_io_ops_akey_enum_active_samples', 'engine_sched_cycle_size_samples', 'engine_io_ops_obj_punch_latency_samples', 'engine_io_ops_obj_enum_latency_samples', 'engine_io_ops_ec_agg_latency_samples', 'engine_sched_cycle_duration_samples', 'engine_io_ops_recx_enum_latency_samples', 'engine_io_ops_obj_sync_active_samples', 'engine_io_ops_compound_active_samples', 'engine_io_latency_update_samples', 'engine_io_ops_ec_rep_latency_samples', 'engine_io_ops_obj_coll_punch_latency_samples', 'engine_io_ops_fetch_active_samples', 'engine_io_ops_tgt_punch_active_samples', 'engine_io_ops_update_active_samples', 'engine_dmabuff_wal_waiters_samples', 'engine_io_ops_dkey_enum_active_samples', 'engine_io_ops_key_query_active_samples', 'engine_io_latency_vos_fetch_samples', 'engine_io_ops_akey_punch_latency_samples', 'engine_io_ops_obj_coll_punch_active_samples', 'engine_io_dtx_committed_samples', 'engine_io_ops_dkey_punch_latency_samples', 'engine_io_ops_compound_latency_samples', 'engine_io_ops_obj_coll_query_latency_samples', 'engine_io_ops_recx_enum_active_samples', 'engine_io_latency_bio_update_samples', 'engine_io_ops_key2anchor_active_samples', 'engine_io_ops_obj_enum_active_samples', 'engine_io_ops_obj_punch_active_samples', 'engine_io_latency_bio_fetch_samples', 'engine_io_latency_tgt_update_samples', 'engine_io_ops_obj_sync_latency_samples', 'engine_io_ops_migrate_active_samples', 'engine_io_ops_tgt_dkey_punch_latency_samples', 'engine_io_ops_akey_enum_latency_samples', 'engine_io_ops_key2anchor_latency_samples', 'engine_io_latency_bulk_fetch_samples', 'engine_io_dtx_committable_samples', 'engine_io_ops_tgt_punch_latency_samples', 'engine_io_ops_tgt_akey_punch_active_samples', 'engine_io_ops_ec_agg_active_samples', 'engine_io_ops_tgt_akey_punch_latency_samples', 'engine_io_ops_tgt_update_active_samples', 'engine_net_swim_delay_samples', 'engine_io_latency_bulk_update_samples', 'engine_io_ops_tgt_dkey_punch_active_samples', 'engine_io_ops_dkey_punch_active_samples', 'engine_io_latency_vos_update_samples', 'engine_io_ops_key_query_latency_samples', 'engine_io_ops_migrate_latency_samples', 'engine_dmabuff_wal_qd_samples', 'engine_io_ops_akey_punch_active_samples', 'engine_io_ops_obj_coll_query_active_samples', 'engine_dmabuff_wal_sz_samples', 'engine_dmabuff_grab_retries_samples', 'engine_io_ops_dkey_enum_latency_samples', 'engine_io_ops_ec_rep_active_samples', 'engine_io_latency_fetch_samples'}
08:29:45 INFO | Test has failed, dumping ULT stacks
08:29:45 INFO | Dumping ULT stacks of engines on wolf-[138-139]
08:30:15 INFO | Command: rc=0; if /usr/bin/pgrep --list-full daos_engine; then rc=1; sudo pkill --signal USR2 daos_engine; sleep 30; fi; exit $rc
Results:
wolf-138: exit_status=1, interrupted=False: 192825 /usr/bin/daos_engine -t 8 -x 4 -g daos_server -d /var/run/daos_server -T 2 -n /mnt/daos0/daos_nvme.conf -I 0 -r 8192 -H 2 -s /mnt/daos0
wolf-139: exit_status=1, interrupted=False: 453468 /usr/bin/daos_engine -t 8 -x 4 -g daos_server -d /var/run/daos_server -T 2 -n /mnt/daos0/daos_nvme.conf -I 0 -r 8192 -H 2 -s /mnt/daos0
08:30:15 ERROR|
08:30:15 ERROR| Reproduced traceback from: /usr/lib/python3.6/site-packages/avocado/core/test.py:767
08:30:15 ERROR| Traceback (most recent call last):
08:30:15 ERROR| File "/usr/lib/daos/TESTING/ftest/control/dmg_telemetry_nvme.py", line 75, in test_telemetry_list_nvme
08:30:15 ERROR| self.verify_telemetry_list()
08:30:15 ERROR| File "/usr/lib/daos/TESTING/ftest/util/telemetry_test_base.py", line 82, in verify_telemetry_list
08:30:15 ERROR| self.fail("\n".join(errors))
08:30:15 ERROR| File "/usr/lib/daos/TESTING/ftest/util/apricot/apricot/test.py", line 1375, in fail
08:30:15 ERROR| super().fail(message)
08:30:15 ERROR| File "/usr/lib/python3.6/site-packages/avocado/core/test.py", line 953, in fail
08:30:15 ERROR| raise exceptions.TestFail(message)
08:30:15 ERROR| avocado.core.exceptions.TestFail: Symmetric difference found in telemetry metric names on wolf-138
08:30:15 ERROR| Symmetric difference found in telemetry metric names on wolf-139
08:30:15 ERROR|
08:30:15 DEBUG| Local variables:
08:30:15 DEBUG| -> self <class 'dmg_telemetry_nvme.TestWithTelemetryNvme'>: 2-./control/dmg_telemetry_nvme.py:TestWithTelemetryNvme.test_telemetry_list_nvme;run-hosts-server_config-engines-0-storage-0-1-telemetry_metrics-bdb4
08:30:15 INFO | ====================================================================================================
08:30:15 INFO | *** TEARDOWN called after test completion: elapsed time: 42.26406741142273 seconds ***
08:30:15 INFO | Amount of time left in test timeout: 77.73593258857727 seconds
08:30:15 DEBUG| no pre-teardown steps defined
08:30:15 INFO | ----------------------------------------------------------------------------------------------------
08:30:15 INFO | --- STOPPING AGENTS ---
08:30:15 INFO | ----------------------------------------------------------------------------------------------------
08:30:15 INFO | --- VERIFYING STATES OF 1 AGENTS GROUP ---
08:30:16 INFO | Command: systemctl is-active daos_agent.service
Results:
wolf-137: exit_status=0, interrupted=False: active
08:30:16 INFO | <AGENT> Verifying agent states: group=daos_server, hosts=wolf-137
08:30:16 INFO | Rank Host UUID Expected State Current State Result
08:30:16 INFO | ---- --------------- ------------------------------------ ---------------------- -------------- ------
08:30:16 INFO | 0 wolf-137 - active active PASS
08:30:16 INFO | Agents are configured to run across multiple test variants, not stopping
08:30:16 INFO | ----------------------------------------------------------------------------------------------------
08:30:16 INFO | --- STOPPING SERVERS ---
08:30:16 INFO | ----------------------------------------------------------------------------------------------------
08:30:16 INFO | --- VERIFYING STATES OF 1 SERVERS GROUP ---
08:30:16 DEBUG| Updated param ranks => None
08:30:16 DEBUG| Updated param verbose => True
08:30:16 INFO | Command environment vars:
{}
08:30:16 INFO | Running '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j system query --verbose'
08:30:16 DEBUG| [stderr] DEBUG 08:30:16.043442 main.go:209: debug output enabled
08:30:16 DEBUG| [stderr] DEBUG 08:30:16.043881 main.go:241: control config loaded from /etc/daos/daos_control.yml
08:30:16 DEBUG| [stderr] DEBUG 08:30:16.062065 system.go:279: DAOS system query request: *mgmt.SystemQueryReq (sys:"daos_server-2.5.100" state_mask:65535)
08:30:16 DEBUG| [stderr] DEBUG 08:30:16.062571 rpc.go:278: request hosts: [wolf-138:10001 wolf-139:10001]
08:30:16 DEBUG| [stderr] DEBUG 08:30:16.165720 response.go:168: wolf-138:10001: *mgmt.SystemQueryResp@5 joined:0-1
08:30:16 DEBUG| [stdout] {
08:30:16 DEBUG| [stdout] "response": {
08:30:16 DEBUG| [stdout] "members": [
08:30:16 DEBUG| [stdout] {
08:30:16 DEBUG| [stdout] "addr": "10.8.1.138:10001",
08:30:16 DEBUG| [stdout] "state": "joined",
08:30:16 DEBUG| [stdout] "fault_domain": "/wolf-138.wolf.hpdd.intel.com",
08:30:16 DEBUG| [stdout] "rank": 0,
08:30:16 DEBUG| [stdout] "incarnation": 1586101373098000384,
08:30:16 DEBUG| [stdout] "uuid": "4b34991b-1f40-4294-8461-e238334ebb64",
08:30:16 DEBUG| [stdout] "fabric_uri": "ofi+verbs;ofi_rxm://192.168.100.138:31416",
08:30:16 DEBUG| [stdout] "fabric_contexts": 14,
08:30:16 DEBUG| [stdout] "info": "",
08:30:16 DEBUG| [stdout] "last_update": "2024-02-22T08:28:57.613312472Z"
08:30:16 DEBUG| [stdout] },
08:30:16 DEBUG| [stdout] {
08:30:16 DEBUG| [stdout] "addr": "10.8.1.139:10001",
08:30:16 DEBUG| [stdout] "state": "joined",
08:30:16 DEBUG| [stdout] "fault_domain": "/wolf-139.wolf.hpdd.intel.com",
08:30:16 DEBUG| [stdout] "rank": 1,
08:30:16 DEBUG| [stdout] "incarnation": 1586101373121069056,
08:30:16 DEBUG| [stdout] "uuid": "357f9bfc-dd72-4e5b-a8f4-0f92c396faa3",
08:30:16 DEBUG| [stdout] "fabric_uri": "ofi+verbs;ofi_rxm://192.168.100.139:31416",
08:30:16 DEBUG| [stdout] "fabric_contexts": 14,
08:30:16 DEBUG| [stdout] "info": "",
08:30:16 DEBUG| [stdout] "last_update": "2024-02-22T08:28:57.610914194Z"
08:30:16 DEBUG| [stdout] }
08:30:16 DEBUG| [stdout] ]
08:30:16 DEBUG| [stdout] },
08:30:16 DEBUG| [stdout] "error": null,
08:30:16 DEBUG| [stdout] "status": 0
08:30:16 DEBUG| [stdout] }
08:30:17 INFO | Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j system query --verbose' finished with 0 after 1.1484229564666748s
08:30:17 INFO | <SERVER> Verifying server states: group=daos_server, hosts=wolf-[138-139]
08:30:17 INFO | Rank Host UUID Expected State Current State Result
08:30:17 INFO | ---- --------------- ------------------------------------ ---------------------- -------------- ------
08:30:17 INFO | 0 wolf-138 4b34991b-1f40-4294-8461-e238334ebb64 joined joined PASS
08:30:17 INFO | 1 wolf-139 357f9bfc-dd72-4e5b-a8f4-0f92c396faa3 joined joined PASS
08:30:17 INFO | Searching for any existing pools
08:30:17 DEBUG| Updated param no_query => False
08:30:17 DEBUG| Updated param verbose => False
08:30:17 INFO | Command environment vars:
{}
08:30:17 INFO | Running '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j pool list'
08:30:17 DEBUG| [stderr] DEBUG 08:30:17.194721 main.go:209: debug output enabled
08:30:17 DEBUG| [stderr] DEBUG 08:30:17.195171 main.go:241: control config loaded from /etc/daos/daos_control.yml
08:30:17 DEBUG| [stderr] DEBUG 08:30:17.213439 pool.go:1251: DAOS system list-pools request: *mgmt.ListPoolsReq (sys:"daos_server-2.5.100")
08:30:17 DEBUG| [stderr] DEBUG 08:30:17.214199 rpc.go:278: request hosts: [wolf-138:10001 wolf-139:10001]
08:30:17 DEBUG| [stderr] DEBUG 08:30:17.317246 response.go:168: wolf-138:10001: *mgmt.ListPoolsResp5 0 pools:
08:30:17 DEBUG| [stdout] {
08:30:17 DEBUG| [stdout] "response": {
08:30:17 DEBUG| [stdout] "status": 0,
08:30:17 DEBUG| [stdout] "pools": null
08:30:17 DEBUG| [stdout] },
08:30:17 DEBUG| [stdout] "error": null,
08:30:17 DEBUG| [stdout] "status": 0
08:30:17 DEBUG| [stdout] }
08:30:18 INFO | Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -d -j pool list' finished with 0 after 1.145552396774292s
08:30:18 INFO | Servers are configured to run across multiple test variants, not stopping
08:30:18 INFO | Removing temporary test files in /var/tmp/daos_testing/test_telemetry_list_nvme from wolf-[137-139]
08:30:18 DEBUG| Running on wolf-[137-139] with a 120 second timeout: sudo -n rm -fr /var/tmp/daos_testing/test_telemetry_list_nvme
08:30:18 DEBUG| wolf-[137-139] (rc=0): <no output>
08:30:18 DEBUG| DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
08:30:18 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data sources: variant, test, file)
08:30:18 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data sources: variant, test, file)
08:30:18 ERROR| FAIL 2-./control/dmg_telemetry_nvme.py:TestWithTelemetryNvme.test_telemetry_list_nvme;run-hosts-server_config-engines-0-storage-0-1-telemetry_metrics-bdb4 -> TestFail: Symmetric difference found in telemetry metric names on wolf-138
Symmetric difference found in telemetry metric names on wolf-139
08:30:18 INFO |
Loading