CA trust certificate not working for 0.2

[wociscz]

Required information

• Distribution: Ubuntu
• Distribution version: Ubuntu 22.04.3 LTS
• The output of "inc info" or if that fails:
  • Kernel version: 6.5.9-zabbly+
  • LXC version: ?
  • Incus version: 0.2
  • Storage backend in use: zfs

Issue description

After upgrading to 0.2 (pkg 0.2-202310282127-ubuntu22.04) CA trust stopped working - I mean adding remotes without tokens and api communication, just plain PKI with core.trust_ca_certificates: "true" configure.

Steps to reproduce

1. install 0.1 incus and configure pki with server/client certificates and server.ca/client.ca
2. incus remote add name hostname works without asking password
3. upgrade to 0.2 incus
4. remove any remote and try to add it again

incus remote add name hostname --debug
DEBUG  [2023-11-04T16:03:28Z] Connecting to a remote Incus over HTTPS       url="https://hostname:8443"
DEBUG  [2023-11-04T16:03:28Z] Sending request to Incus                      etag= method=GET url="https://hostname:8443/1.0"
Certificate fingerprint: f8adadd08bac1069ecb6317e0285b16d70ca0b472e50b8ae27cb09d1a41eb482
ok (y/n/[fingerprint])? y
DEBUG  [2023-11-04T16:03:37Z] Connecting to a remote Incus over HTTPS       url="https://hostname:8443"
DEBUG  [2023-11-04T16:03:37Z] Sending request to Incus                      etag= method=GET url="https://hostname:8443/1.0"
Error: Client certificate not found

5. downgrade to 0.1-202310210536-ubuntu22.04 everything back in working state

incus monitor on the server side:

location: none
metadata:
  context:
    ip: 192.168.30.104:41138
    method: GET
    protocol: tls
    url: /1.0
    username: 8d69ec0820ad7f0bfda61c50719c49b22003ebd49ec740e1ae3eb6fc84e46435
  level: debug
  message: Handling API request
timestamp: "2023-11-04T16:04:53.303719244Z"
type: logging


location: none
metadata:
  context:
    ip: 192.168.30.104:41150
    method: GET
    protocol: tls
    url: /1.0
    username: 8d69ec0820ad7f0bfda61c50719c49b22003ebd49ec740e1ae3eb6fc84e46435
  level: debug
  message: Handling API request
timestamp: "2023-11-04T16:04:54.257944385Z"
type: logging

incus info (config: part)

config:
  core.https_address: :8443
  core.metrics_address: :8444
  core.trust_ca_certificates: "true"
  storage.images_volume: default/kitchen

other notes

Both certificates (client and server) are trusted - tested via openssl verify. Our internal CA is baked into system CA "trust store".
not working table:

client	server	state
0.1	0.2	not working
0.2	0.2	not working
0.2	0.1	ok
0.1	0.1	ok

[stgraber]

I'm looking at this one now.

[stgraber]

Reproduced the issue, starting a bisect now.

[stgraber]

So looks like it's in the commits we imported from LXD which are related to the addition of OpenFGA. I'm looking at a fix now.