On Void Linux: QEMU failed to run feature checks / Failed creating instance record.

[mwyvr]

While creating a PR for the Void Linux project to update incus on void-packagesto v0.5.1, I discovered an issue that has been present in lxd and in incus from the current shipping version on Void, 0.4.0. I note a few similar reports on other distributions over the last few months.

Required information

• Distribution: Void Linux (glibc)
• Distribution version: Rolling
• The output of "incus info" or if that fails:
  • Kernel version: 6.6.15_1
  • LXC version: lxc-5.0.1_3
  • Incus version: Client version: 0.5.1, Server version: 0.5.1
  • Storage backend in use: zfs

Issue description

Attempt to create a VM instance - any VM - fails.

incus launch images:ubuntu/22.04 foobvm  --vm

Error: Failed instance creation: Failed creating instance record: Instance type "virtual-machine" is \
  not supported on this server: QEMU failed to run feature checks

No detailed logging information is provided beyond that, even in --debug mode or in incus monitor. It would help if the
actual error were reported. It appears that OVMF is being located, btw.

Steps to reproduce

1. Working incus 0.4.0 or 0.5.1 installation with proven ability to create container (not VM) instances
2. Check that the Void Linux host has the necessary bits to start a VM directly through command line startup via qemu (x86) secure boot and without.
3. incus launch per the issue description; multiple images/different distributions checked.

Information to attach

• Any relevant kernel output (dmesg)

None available.

• Container log (incus info NAME --show-log)

None created

• Container configuration (incus config show NAME --expanded)

None created

• Main daemon log (at /var/log/incus/incusd.log)

None available.

• Output of the client with --debug

# incus launch images:ubuntu/22.04 foobvm  --vm --debug
DEBUG  [2024-02-08T14:59:49-08:00] Connecting to a local Incus over a Unix socket
DEBUG  [2024-02-08T14:59:49-08:00] Sending request to Incus                      etag= method=GET url="http://unix.socket/1.0"
DEBUG  [2024-02-08T14:59:49-08:00] Got response struct from Incus
DEBUG  [2024-02-08T14:59:49-08:00]
	{
		"config": {},
		"api_extensions": [
			"storage_zfs_remove_snapshots",
			"container_host_shutdown_timeout",
			"container_stop_priority",
			"container_syscall_filtering",
			"auth_pki",
			"container_last_used_at",
			"etag",
			"patch",
			"usb_devices",
			"https_allowed_credentials",
			"image_compression_algorithm",
			"directory_manipulation",
			"container_cpu_time",
			"storage_zfs_use_refquota",
			"storage_lvm_mount_options",
			"network",
			"profile_usedby",
			"container_push",
			"container_exec_recording",
			"certificate_update",
			"container_exec_signal_handling",
			"gpu_devices",
			"container_image_properties",
			"migration_progress",
			"id_map",
			"network_firewall_filtering",
			"network_routes",
			"storage",
			"file_delete",
			"file_append",
			"network_dhcp_expiry",
			"storage_lvm_vg_rename",
			"storage_lvm_thinpool_rename",
			"network_vlan",
			"image_create_aliases",
			"container_stateless_copy",
			"container_only_migration",
			"storage_zfs_clone_copy",
			"unix_device_rename",
			"storage_lvm_use_thinpool",
			"storage_rsync_bwlimit",
			"network_vxlan_interface",
			"storage_btrfs_mount_options",
			"entity_description",
			"image_force_refresh",
			"storage_lvm_lv_resizing",
			"id_map_base",
			"file_symlinks",
			"container_push_target",
			"network_vlan_physical",
			"storage_images_delete",
			"container_edit_metadata",
			"container_snapshot_stateful_migration",
			"storage_driver_ceph",
			"storage_ceph_user_name",
			"resource_limits",
			"storage_volatile_initial_source",
			"storage_ceph_force_osd_reuse",
			"storage_block_filesystem_btrfs",
			"resources",
			"kernel_limits",
			"storage_api_volume_rename",
			"network_sriov",
			"console",
			"restrict_dev_incus",
			"migration_pre_copy",
			"infiniband",
			"dev_incus_events",
			"proxy",
			"network_dhcp_gateway",
			"file_get_symlink",
			"network_leases",
			"unix_device_hotplug",
			"storage_api_local_volume_handling",
			"operation_description",
			"clustering",
			"event_lifecycle",
			"storage_api_remote_volume_handling",
			"nvidia_runtime",
			"container_mount_propagation",
			"container_backup",
			"dev_incus_images",
			"container_local_cross_pool_handling",
			"proxy_unix",
			"proxy_udp",
			"clustering_join",
			"proxy_tcp_udp_multi_port_handling",
			"network_state",
			"proxy_unix_dac_properties",
			"container_protection_delete",
			"unix_priv_drop",
			"pprof_http",
			"proxy_haproxy_protocol",
			"network_hwaddr",
			"proxy_nat",
			"network_nat_order",
			"container_full",
			"backup_compression",
			"nvidia_runtime_config",
			"storage_api_volume_snapshots",
			"storage_unmapped",
			"projects",
			"network_vxlan_ttl",
			"container_incremental_copy",
			"usb_optional_vendorid",
			"snapshot_scheduling",
			"snapshot_schedule_aliases",
			"container_copy_project",
			"clustering_server_address",
			"clustering_image_replication",
			"container_protection_shift",
			"snapshot_expiry",
			"container_backup_override_pool",
			"snapshot_expiry_creation",
			"network_leases_location",
			"resources_cpu_socket",
			"resources_gpu",
			"resources_numa",
			"kernel_features",
			"id_map_current",
			"event_location",
			"storage_api_remote_volume_snapshots",
			"network_nat_address",
			"container_nic_routes",
			"cluster_internal_copy",
			"seccomp_notify",
			"lxc_features",
			"container_nic_ipvlan",
			"network_vlan_sriov",
			"storage_cephfs",
			"container_nic_ipfilter",
			"resources_v2",
			"container_exec_user_group_cwd",
			"container_syscall_intercept",
			"container_disk_shift",
			"storage_shifted",
			"resources_infiniband",
			"daemon_storage",
			"instances",
			"image_types",
			"resources_disk_sata",
			"clustering_roles",
			"images_expiry",
			"resources_network_firmware",
			"backup_compression_algorithm",
			"ceph_data_pool_name",
			"container_syscall_intercept_mount",
			"compression_squashfs",
			"container_raw_mount",
			"container_nic_routed",
			"container_syscall_intercept_mount_fuse",
			"container_disk_ceph",
			"virtual-machines",
			"image_profiles",
			"clustering_architecture",
			"resources_disk_id",
			"storage_lvm_stripes",
			"vm_boot_priority",
			"unix_hotplug_devices",
			"api_filtering",
			"instance_nic_network",
			"clustering_sizing",
			"firewall_driver",
			"projects_limits",
			"container_syscall_intercept_hugetlbfs",
			"limits_hugepages",
			"container_nic_routed_gateway",
			"projects_restrictions",
			"custom_volume_snapshot_expiry",
			"volume_snapshot_scheduling",
			"trust_ca_certificates",
			"snapshot_disk_usage",
			"clustering_edit_roles",
			"container_nic_routed_host_address",
			"container_nic_ipvlan_gateway",
			"resources_usb_pci",
			"resources_cpu_threads_numa",
			"resources_cpu_core_die",
			"api_os",
			"container_nic_routed_host_table",
			"container_nic_ipvlan_host_table",
			"container_nic_ipvlan_mode",
			"resources_system",
			"images_push_relay",
			"network_dns_search",
			"container_nic_routed_limits",
			"instance_nic_bridged_vlan",
			"network_state_bond_bridge",
			"usedby_consistency",
			"custom_block_volumes",
			"clustering_failure_domains",
			"resources_gpu_mdev",
			"console_vga_type",
			"projects_limits_disk",
			"network_type_macvlan",
			"network_type_sriov",
			"container_syscall_intercept_bpf_devices",
			"network_type_ovn",
			"projects_networks",
			"projects_networks_restricted_uplinks",
			"custom_volume_backup",
			"backup_override_name",
			"storage_rsync_compression",
			"network_type_physical",
			"network_ovn_external_subnets",
			"network_ovn_nat",
			"network_ovn_external_routes_remove",
			"tpm_device_type",
			"storage_zfs_clone_copy_rebase",
			"gpu_mdev",
			"resources_pci_iommu",
			"resources_network_usb",
			"resources_disk_address",
			"network_physical_ovn_ingress_mode",
			"network_ovn_dhcp",
			"network_physical_routes_anycast",
			"projects_limits_instances",
			"network_state_vlan",
			"instance_nic_bridged_port_isolation",
			"instance_bulk_state_change",
			"network_gvrp",
			"instance_pool_move",
			"gpu_sriov",
			"pci_device_type",
			"storage_volume_state",
			"network_acl",
			"migration_stateful",
			"disk_state_quota",
			"storage_ceph_features",
			"projects_compression",
			"projects_images_remote_cache_expiry",
			"certificate_project",
			"network_ovn_acl",
			"projects_images_auto_update",
			"projects_restricted_cluster_target",
			"images_default_architecture",
			"network_ovn_acl_defaults",
			"gpu_mig",
			"project_usage",
			"network_bridge_acl",
			"warnings",
			"projects_restricted_backups_and_snapshots",
			"clustering_join_token",
			"clustering_description",
			"server_trusted_proxy",
			"clustering_update_cert",
			"storage_api_project",
			"server_instance_driver_operational",
			"server_supported_storage_drivers",
			"event_lifecycle_requestor_address",
			"resources_gpu_usb",
			"clustering_evacuation",
			"network_ovn_nat_address",
			"network_bgp",
			"network_forward",
			"custom_volume_refresh",
			"network_counters_errors_dropped",
			"metrics",
			"image_source_project",
			"clustering_config",
			"network_peer",
			"linux_sysctl",
			"network_dns",
			"ovn_nic_acceleration",
			"certificate_self_renewal",
			"instance_project_move",
			"storage_volume_project_move",
			"cloud_init",
			"network_dns_nat",
			"database_leader",
			"instance_all_projects",
			"clustering_groups",
			"ceph_rbd_du",
			"instance_get_full",
			"qemu_metrics",
			"gpu_mig_uuid",
			"event_project",
			"clustering_evacuation_live",
			"instance_allow_inconsistent_copy",
			"network_state_ovn",
			"storage_volume_api_filtering",
			"image_restrictions",
			"storage_zfs_export",
			"network_dns_records",
			"storage_zfs_reserve_space",
			"network_acl_log",
			"storage_zfs_blocksize",
			"metrics_cpu_seconds",
			"instance_snapshot_never",
			"certificate_token",
			"instance_nic_routed_neighbor_probe",
			"event_hub",
			"agent_nic_config",
			"projects_restricted_intercept",
			"metrics_authentication",
			"images_target_project",
			"cluster_migration_inconsistent_copy",
			"cluster_ovn_chassis",
			"container_syscall_intercept_sched_setscheduler",
			"storage_lvm_thinpool_metadata_size",
			"storage_volume_state_total",
			"instance_file_head",
			"instances_nic_host_name",
			"image_copy_profile",
			"container_syscall_intercept_sysinfo",
			"clustering_evacuation_mode",
			"resources_pci_vpd",
			"qemu_raw_conf",
			"storage_cephfs_fscache",
			"network_load_balancer",
			"vsock_api",
			"instance_ready_state",
			"network_bgp_holdtime",
			"storage_volumes_all_projects",
			"metrics_memory_oom_total",
			"storage_buckets",
			"storage_buckets_create_credentials",
			"metrics_cpu_effective_total",
			"projects_networks_restricted_access",
			"storage_buckets_local",
			"loki",
			"acme",
			"internal_metrics",
			"cluster_join_token_expiry",
			"remote_token_expiry",
			"init_preseed",
			"storage_volumes_created_at",
			"cpu_hotplug",
			"projects_networks_zones",
			"network_txqueuelen",
			"cluster_member_state",
			"instances_placement_scriptlet",
			"storage_pool_source_wipe",
			"zfs_block_mode",
			"instance_generation_id",
			"disk_io_cache",
			"amd_sev",
			"storage_pool_loop_resize",
			"migration_vm_live",
			"ovn_nic_nesting",
			"oidc",
			"network_ovn_l3only",
			"ovn_nic_acceleration_vdpa",
			"cluster_healing",
			"instances_state_total",
			"auth_user",
			"security_csm",
			"instances_rebuild",
			"numa_cpu_placement",
			"custom_volume_iso",
			"network_allocations",
			"zfs_delegate",
			"storage_api_remote_volume_snapshot_copy",
			"operations_get_query_all_projects",
			"metadata_configuration",
			"syslog_socket",
			"event_lifecycle_name_and_project",
			"instances_nic_limits_priority",
			"disk_initial_volume_configuration",
			"operation_wait",
			"image_restriction_privileged",
			"cluster_internal_custom_volume_copy",
			"disk_io_bus",
			"storage_cephfs_create_missing",
			"instance_move_config",
			"ovn_ssl_config",
			"certificate_description",
			"disk_io_bus_virtio_blk",
			"loki_config_instance",
			"instance_create_start",
			"clustering_evacuation_stop_options",
			"boot_host_shutdown_action",
			"agent_config_drive"
		],
		"api_status": "stable",
		"api_version": "1.0",
		"auth": "trusted",
		"public": false,
		"auth_methods": [
			"tls"
		],
		"auth_user_name": "root",
		"auth_user_method": "unix",
		"environment": {
			"addresses": [],
			"architectures": [
				"x86_64",
				"i686"
			],
			"certificate": "-----BEGIN CERTIFICATE-----\nMIIB/TCCAYOgAwIBAgIQQv/mknU8SbiEJpZImecaHzAKBggqhkjOPQQDAzAxMRkw\nFwYDVQQKExBMaW51eCBDb250YWluZXJzMRQwEgYDVQQDDAtyb290QHdhYmJpdDAe\nFw0yNDAyMDcwNDM0MjRaFw0zNDAyMDQwNDM0MjRaMDExGTAXBgNVBAoTEExpbnV4\nIENvbnRhaW5lcnMxFDASBgNVBAMMC3Jvb3RAd2FiYml0MHYwEAYHKoZIzj0CAQYF\nK4EEACIDYgAEM1IM5PEPqkpW2GzJLw5oPRf39pard8e+4CK/l5Mql12BHrb3AIhQ\nO6vzikm1IbEsF9BjLha7/LjlIvN1PYEFSCcuSV08d1tdKDfTDg7qdMfJKekx83oe\nbEokoIN3nIxJo2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH\nAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZ3YWJiaXSHBH8AAAGHEAAAAAAA\nAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaAAwZQIxAJFy2bNOIsHIOegV+JzUcLXi\nzGh/nmMWO75i5pKTy2S/Z5uPtXhI7iuwzwNsjJfOJwIwKo2xLnlqherTxhaB2V1D\nctDqfih8e0yeHtyl3dvaOhaeyMcaa18pYFtDvMNI1z0t\n-----END CERTIFICATE-----\n",
			"certificate_fingerprint": "9fc743cf92db7303ee2dfd22d074897efe55ec4ed23f536cf474d841576726ff",
			"driver": "lxc",
			"driver_version": "5.0.1",
			"firewall": "xtables",
			"kernel": "Linux",
			"kernel_architecture": "x86_64",
			"kernel_features": {
				"idmapped_mounts": "true",
				"netnsid_getifaddrs": "true",
				"seccomp_listener": "true",
				"seccomp_listener_continue": "true",
				"uevent_injection": "true",
				"unpriv_fscaps": "true"
			},
			"kernel_version": "6.6.15_1",
			"lxc_features": {
				"cgroup2": "true",
				"core_scheduling": "true",
				"devpts_fd": "true",
				"idmapped_mounts_v2": "true",
				"mount_injection_file": "true",
				"network_gateway_device_route": "true",
				"network_ipvlan": "true",
				"network_l2proxy": "true",
				"network_phys_macvlan_mtu": "true",
				"network_veth_router": "true",
				"pidfd": "true",
				"seccomp_allow_deny_syntax": "true",
				"seccomp_notify": "true",
				"seccomp_proxy_send_notify_fd": "true"
			},
			"os_name": "Void",
			"os_version": "",
			"project": "default",
			"server": "incus",
			"server_clustered": false,
			"server_event_mode": "full-mesh",
			"server_name": "wabbit",
			"server_pid": 18833,
			"server_version": "0.5.1",
			"storage": "zfs",
			"storage_version": "2.2.2-1",
			"storage_supported_drivers": [
				{
					"Name": "dir",
					"Version": "1",
					"Remote": false
				},
				{
					"Name": "lvm",
					"Version": "2.03.22(2) (2023-08-02) / 1.02.196 (2023-08-02) / 4.48.0",
					"Remote": false
				},
				{
					"Name": "zfs",
					"Version": "2.2.2-1",
					"Remote": false
				},
				{
					"Name": "btrfs",
					"Version": "6.5.1",
					"Remote": false
				}
			]
		}
	}
Launching foobvm
DEBUG  [2024-02-08T14:59:49-08:00] Connecting to a remote simplestreams server   URL="https://images.linuxcontainers.org"
DEBUG  [2024-02-08T14:59:49-08:00] Connected to the websocket: ws://unix.socket/1.0/events
DEBUG  [2024-02-08T14:59:49-08:00] Sending request to Incus                      etag= method=POST url="http://unix.socket/1.0/instances"
DEBUG  [2024-02-08T14:59:49-08:00]
	{
		"architecture": "",
		"config": {},
		"devices": {},
		"ephemeral": false,
		"profiles": null,
		"stateful": false,
		"description": "",
		"name": "foobvm",
		"source": {
			"type": "image",
			"certificate": "",
			"alias": "ubuntu/22.04",
			"server": "https://images.linuxcontainers.org",
			"protocol": "simplestreams",
			"mode": "pull",
			"allow_inconsistent": false
		},
		"instance_type": "",
		"type": "virtual-machine",
		"start": true
	}
DEBUG  [2024-02-08T14:59:49-08:00] Got operation from Incus
DEBUG  [2024-02-08T14:59:49-08:00]
	{
		"id": "5ad8bdaf-0294-4a65-96f7-8848154f12a6",
		"class": "task",
		"description": "Creating instance",
		"created_at": "2024-02-08T14:59:49.591118997-08:00",
		"updated_at": "2024-02-08T14:59:49.591118997-08:00",
		"status": "Running",
		"status_code": 103,
		"resources": {
			"instances": [
				"/1.0/instances/foobvm"
			]
		},
		"metadata": null,
		"may_cancel": false,
		"err": "",
		"location": "none"
	}
DEBUG  [2024-02-08T14:59:49-08:00] Sending request to Incus                      etag= method=GET url="http://unix.socket/1.0/operations/5ad8bdaf-0294-4a65-96f7-8848154f12a6"
DEBUG  [2024-02-08T14:59:49-08:00] Got response struct from Incus
DEBUG  [2024-02-08T14:59:49-08:00]
	{
		"id": "5ad8bdaf-0294-4a65-96f7-8848154f12a6",
		"class": "task",
		"description": "Creating instance",
		"created_at": "2024-02-08T14:59:49.591118997-08:00",
		"updated_at": "2024-02-08T14:59:49.591118997-08:00",
		"status": "Running",
		"status_code": 103,
		"resources": {
			"instances": [
				"/1.0/instances/foobvm"
			]
		},
		"metadata": null,
		"may_cancel": false,
		"err": "",
		"location": "none"
	}
Error: Failed instance creation: Failed creating instance record: Instance type "virtual-machine" is not supported on this server: QEMU failed to run feature checks

• Output of the daemon with --debug (alternatively output of incus monitor --pretty while reproducing the issue)

$ incus monitor --pretty
DEBUG  [2024-02-08T14:58:19-08:00] Event listener server handler started         id=1d2011e8-d963-4a4b-a041-006fffd1da91 local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-02-08T14:58:26-08:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
DEBUG  [2024-02-08T14:58:26-08:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=root
DEBUG  [2024-02-08T14:58:26-08:00] Event listener server handler started         id=38c10ed3-fe7b-4544-8787-9f36971321a4 local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-02-08T14:58:26-08:00] Responding to instance create
DEBUG  [2024-02-08T14:58:26-08:00] Handling API request                          ip=@ method=POST protocol=unix url=/1.0/instances username=root
DEBUG  [2024-02-08T14:58:26-08:00] New operation                                 class=task description="Creating instance" operation=c4ad89a3-49e8-4afb-93c2-f04d98f091cd project=default
DEBUG  [2024-02-08T14:58:26-08:00] Started operation                             class=task description="Creating instance" operation=c4ad89a3-49e8-4afb-93c2-f04d98f091cd project=default
INFO   [2024-02-08T14:58:26-08:00] ID: c4ad89a3-49e8-4afb-93c2-f04d98f091cd, Class: task, Description: Creating instance  CreatedAt="2024-02-08 14:58:26.055037084 -0800 PST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/foobvm]]" Status=Pending StatusCode=Pending UpdatedAt="2024-02-08 14:58:26.055037084 -0800 PST"
INFO   [2024-02-08T14:58:26-08:00] ID: c4ad89a3-49e8-4afb-93c2-f04d98f091cd, Class: task, Description: Creating instance  CreatedAt="2024-02-08 14:58:26.055037084 -0800 PST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/foobvm]]" Status=Running StatusCode=Running UpdatedAt="2024-02-08 14:58:26.055037084 -0800 PST"
DEBUG  [2024-02-08T14:58:26-08:00] Connecting to a remote simplestreams server   URL="https://images.linuxcontainers.org"
DEBUG  [2024-02-08T14:58:26-08:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/c4ad89a3-49e8-4afb-93c2-f04d98f091cd username=root
DEBUG  [2024-02-08T14:58:26-08:00] Lock acquired for image                       fingerprint=0f706a9f09b346c73ffef18f93212fd5aac4424e0783bcca99371577ab329528
DEBUG  [2024-02-08T14:58:26-08:00] Acquiring lock for image                      fingerprint=0f706a9f09b346c73ffef18f93212fd5aac4424e0783bcca99371577ab329528
DEBUG  [2024-02-08T14:58:26-08:00] Image already exists in the DB                fingerprint=0f706a9f09b346c73ffef18f93212fd5aac4424e0783bcca99371577ab329528
DEBUG  [2024-02-08T14:58:26-08:00] Failure for operation                         class=task description="Creating instance" err="Failed creating instance record: Instance type \"virtual-machine\" is not supported on this server: QEMU failed to run feature checks" operation=c4ad89a3-49e8-4afb-93c2-f04d98f091cd project=default
INFO   [2024-02-08T14:58:26-08:00] ID: c4ad89a3-49e8-4afb-93c2-f04d98f091cd, Class: task, Description: Creating instance  CreatedAt="2024-02-08 14:58:26.055037084 -0800 PST" Err="Failed creating instance record: Instance type \"virtual-machine\" is not supported on this server: QEMU failed to run feature checks" Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/foobvm]]" Status=Failure StatusCode=Failure UpdatedAt="2024-02-08 14:58:26.055037084 -0800 PST"
DEBUG  [2024-02-08T14:58:26-08:00] Event listener server handler stopped         listener=38c10ed3-fe7b-4544-8787-9f36971321a4 local=/var/lib/incus/unix.socket remote=@

[stgraber]

Just got to do some quick tests. To get VMs working on Void, I had to:

• Install incus and incus-client
• Install qemu and edk2-ovmf
• Manually add INCUS_OVMF_PATH=/usr/share/edk2/x64/ to incusd environment
• Make sure to disable security.secureboot on VMs as void doesn't provide adequate firmware+config for UEFI Secureboot

With that done,

[stgraber]

Worth noting that I figured out what was going on by looking at the Incus log file as that's where we log early startup errors.

In this case, we were showing:

Unable to run feature checks during QEMU initialization: Unable to locate a UEFI firmware

[mwyvr]

Thanks, Stéphane, for tracking that down. I was looking into edk2 as a source but overlooked putting --debug on the incus daemon.

Is that logging output only available from incusd --debug? ; would it be possible to include it in the client --debug or --verbose? May help resolve things for others down the road.

I'll update the Void incus package accordingly.

[stgraber]

I didn't have to run incusd with --debug or even --verbose to get that message.
As it's a warning, it's shown by default, you just need to have something actually catching it, whether that's --logfile /var/log/incus/incusd.log or your init system recording the stderr output (as systemd does with journald).

[mwyvr]

We do indeed log to --syslog; not sure how I missed the UEFI hint when I went looking for it, thank you.

Regarding INCUS_OVMF_PATH that does solve the issue for those only launching one arch-firmware; I don't know the Void history behind splitting the OVMF files up by arch but perhaps other distributions are in the same boat.

Would it be possible to add a configuration option for this so instances can be specified with different paths to the OVMF files?

[stgraber]

Incus only supports running the native architecture anyway, we don't do support architecture emulation. So as long as you set INCUS_OVMF_PATH to point to the host architecture path, it'll be fine.

[trebestie]

Hi all,
manually add INCUS_OVMF_PATH=/usr/share/edk2/x64/ to incusd environment works in x64 system.

That fix can't be applied (or I do not know how) on a RaspberryPi4 since VoidLinux doesn't have edk2-ovmf package for aarch64 platform, so I switched to Fedora which provides such package and it looks like the right environment in incusd.service as well.

To setup it I followed the section 'Start your first virtual machine' here, but i got the the same error.

This is a very low priority issue, I was dealing with my RPI4,

Let me know if i can help anyway

My regards

[dkwo]

@stgraber Sorry for bumping this, but in updating incus to 6.4 on Void, now for me incus exec v1 bash errors with "VM agent isn't currently running", even though status is running, and core at 100% (where instead you get a shell above). Do you mind repeating your test above?

[dkwo]

Also, for secure boot, the files are named like

/usr/share/edk2/ia32/OVMF_CODE.secure.4m.fd
/usr/share/edk2/ia32/OVMF_CODE.secure.fd
/usr/share/edk2/x64/OVMF_CODE.secure.4m.fd
/usr/share/edk2/x64/OVMF_CODE.secure.fd

Is there a way to make incus use them?

[stgraber]

Probably won't get to look into this much until October, I've got a bunch of things to do for 6.5 next week and then traveling most of September.

You should look at the console incus start --console NAME to see what's going on.

For the paths, you can send a PR to update https://github.com/lxc/incus/blob/main/internal/server/instance/drivers/edk2/driver_edk2.go#L36 to match the paths on Void. Note that you need both the firmware (CODE) and data (VARS).

[dkwo]

incus start --console v1 is stuck at

BdsDxe: loading Boot0001 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x1)
BdsDxe: starting Boot0001 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x1)

Note that in this case incus stop v1 does not work (errors out after a while), and I have to manually kill the qemu instance, as well as possibly clean /run/incus by hand. (Also notice that the dnsmasq --keep-in-foreground process is also usually kept around after stopping incus services, not sure if it's desirable, and ip r has the additional routes

xxx/24 dev incusbr0 proto kernel scope link src xxx linkdown 
yyy/24 dev incusbr-1000 proto kernel scope link src yyy linkdown 

until reboot.)

Not sure it is relevant, but the same command (without daemonize) gives

$ doas qemu-system-x86_64 -S -name v1 -uuid d7919b95-4d57-4ccb-9f6e-723254710111 -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /run/incus/user-1000_v1/qemu.conf -spice unix=on,disable-ticketing=on,addr=/run/incus/user-1000_v1/qemu.spice -pidfile /run/incus/user-1000_v1/qemu.pid -D /var/log/incus/user-1000_v1/qemu.log -smbios type=2,manufacturer=LinuxContainers,product=Incus -runas nobody 
qemu-system-x86_64:/run/incus/user-1000_v1/qemu.conf:53: Could not open '/dev/fd/4': No such device or address

From /run/incus/user-1000_v1/qemu.conf

# Firmware settings (writable)
[drive]
file = "/dev/fd/4"
if = "pflash"
format = "raw"
unit = "1"

[stgraber]

incus stop performs a clean shutdown which requires a working OS, incus stop -f would most likely have worked fine.

incusbr0 keeps running when Incus is stopped as otherwise restarting incus would break networking for all instances.

So looks like everything is fine on the Incus side, we started QEMU with the firmware from your distribution, things then got stuck there which could indicate a broken EDK2 firmware or something wrong with the VM image.

[dkwo]

Thanks for clarifying. Indeed incus stop -f works fine. This is with alpine/edge vm: BdsDxe: loading Boot0001 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x1) BdsDxe: starting Boot0001 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x1) Booting `Alpine Linux v3.19' Loading Linux virt ... Loading initial ramdisk ... EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path I can try downgrading the ed2k package, but is it possible that something changed in incus? if so, do you have a guess which version of incus to try and downgrade to? or can you still get to exec a shell with v6.4?

[dkwo]

Ah, downgrading edk2-ovmf to 202311 lets the vm start.

[dkwo]

Will inquire with Void then.

[dkwo]

I'm trying to patch incus to find void edk2 stuff.

[dkwo]

see #1170