Add policy support to peer-pods #1607

bpradipt · 2023-12-01T06:39:26Z

This series add policy support to peer-pods.
~~Only packer built podvm images will work for now. A follow on commit will enable addition of opa and related deps via mkosi build~~

bpradipt · 2023-12-14T17:30:21Z

For testing, new Kata shim (from CCv0) is needed. A quick way is to use the following daemonset to replace the existing shim with the new one - https://gist.githubusercontent.com/bpradipt/971b2423e183839ae2f73e3bfa3670eb/raw/0db094e6bc85337b252a0456ea8c751f51543751/kata-ds.yaml

CAA image: quay.io/bpradipt/cloud-api-adaptor:latest

bpradipt · 2023-12-15T17:59:34Z

Tested a scenario where the default policy only allows SetPolicy API

root@ip-192-168-87-191:/etc/kata-opa# ls -ltr
total 12
-rw-rw-r-- 1 ubuntu ubuntu 1361 Dec 14 17:21 allow-all.rego
-rw-rw-r-- 1 ubuntu ubuntu 1361 Dec 14 17:21 allow-all-except-exec-process.rego
-rw-rw-r-- 1 ubuntu ubuntu 1395 Dec 15 16:06 disallow-all-except-setpolicy.rego
lrwxrwxrwx 1 ubuntu ubuntu   34 Dec 15 16:07 default-policy.rego -> disallow-all-except-setpolicy.rego

And policy is made available via annotation

 io.katacontainers.config.agent.policy: cGFja2FnZSBhZ2VudF9wb2xpY3kKCmRlZmF1bHQgQWRkQVJQTmVpZ2hib3JzUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgQWRkU3dhcFJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IENsb3NlU3RkaW5SZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBDb3B5RmlsZVJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IENyZWF0ZUNvbnRhaW5lclJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IENyZWF0ZVNhbmRib3hSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBEZXN0cm95U2FuZGJveFJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IEdldE1ldHJpY3NSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBHZXRPT01FdmVudFJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IEd1ZXN0RGV0YWlsc1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IExpc3RJbnRlcmZhY2VzUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgTGlzdFJvdXRlc1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IE1lbUhvdHBsdWdCeVByb2JlUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgT25saW5lQ1BVTWVtUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgUGF1c2VDb250YWluZXJSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBQdWxsSW1hZ2VSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBSZWFkU3RyZWFtUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgUmVtb3ZlQ29udGFpbmVyUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgUmVtb3ZlU3RhbGVWaXJ0aW9mc1NoYXJlTW91bnRzUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgUmVzZWVkUmFuZG9tRGV2UmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgUmVzdW1lQ29udGFpbmVyUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgU2V0R3Vlc3REYXRlVGltZVJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFNldFBvbGljeVJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFNpZ25hbFByb2Nlc3NSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBTdGFydENvbnRhaW5lclJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFN0YXJ0VHJhY2luZ1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFN0YXRzQ29udGFpbmVyUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgU3RvcFRyYWNpbmdSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBUdHlXaW5SZXNpemVSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBVcGRhdGVDb250YWluZXJSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBVcGRhdGVFcGhlbWVyYWxNb3VudHNSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBVcGRhdGVJbnRlcmZhY2VSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBVcGRhdGVSb3V0ZXNSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBXYWl0UHJvY2Vzc1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFdyaXRlU3RyZWFtUmVxdWVzdCA6PSB0cnVlCgpkZWZhdWx0IEV4ZWNQcm9jZXNzUmVxdWVzdCA6PSBmYWxzZQo=

CAA logs

...
2023/12/15 16:15:44 [adaptor/proxy] SetPolicy: policy:package agent_policy

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := true
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SetPolicyRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default WriteStreamRequest := true

default ExecProcessRequest := false
2023/12/15 16:15:58 [adaptor/proxy] CreateSandbox: hostname:bp-test sandboxId:bbddbf5c1fe49f27b82461aef98f7af681ea8bf13bffcf03c5ce96768c1313b9
2023/12/15 16:15:58 [adaptor/proxy]     storages:
2023/12/15 16:15:58 [adaptor/proxy]         mountpoint:/run/kata-containers/sandbox/shm source:shm fstype:tmpfs driver:ephemeral
2023/12/15 16:15:58 [adaptor/proxy] CreateContainer: containerID:bbddbf5c1fe49f27b82461aef98f7af681ea8bf13bffcf03c5ce96768c1313b9
...

bpradipt · 2023-12-16T02:35:53Z

In order to test with mkosi image, apply the following patch

diff --git a/podvm-mkosi/Makefile b/podvm-mkosi/Makefile
index 20ab680..7ac8a71 100644
--- a/podvm-mkosi/Makefile
+++ b/podvm-mkosi/Makefile
@@ -13,6 +13,8 @@ fedora-binaries-builder:
        docker buildx build \
                -t fedora-binaries-builder \
                --load \
+               --build-arg CAA_SRC="https://github.com/bpradipt/cloud-api-adaptor" \
+               --build-arg CAA_SRC_REF="policy" \
                - < ../podvm/Dockerfile.podvm_builder.fedora

 PHONY: binaries

I had to add a commit to fix the missing agent-config.toml in /run/peerpod for packer built image. Discussions happening in slack - https://cloud-native.slack.com/archives/C04A2EJ70BX/p1702557858190549

bpradipt · 2023-12-16T02:38:58Z

@mkulke @katexochen @stevenhorsman ptal
For testing you'll need the latest CCv0 kata shim with policy support.
You can use a daemonset to replace the existing shim from the coco operator installed system as mentioned in the following message - #1607 (comment)

The default policy is to allow all api. There are sample policies to "disable only exec api" and "disable all except setpolicy".

Here is a sample pod yaml which disables exec

apiVersion: v1
kind: Pod
metadata:
  name: test
  labels:
    app: test
  annotations:
    io.containerd.cri.runtime-handler: kata-remote
    io.katacontainers.config.agent.policy: cGFja2FnZSBhZ2VudF9wb2xpY3kKCmRlZmF1bHQgQWRkQVJQTmVpZ2hib3JzUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgQWRkU3dhcFJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IENsb3NlU3RkaW5SZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBDb3B5RmlsZVJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IENyZWF0ZUNvbnRhaW5lclJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IENyZWF0ZVNhbmRib3hSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBEZXN0cm95U2FuZGJveFJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IEdldE1ldHJpY3NSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBHZXRPT01FdmVudFJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IEd1ZXN0RGV0YWlsc1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IExpc3RJbnRlcmZhY2VzUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgTGlzdFJvdXRlc1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IE1lbUhvdHBsdWdCeVByb2JlUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgT25saW5lQ1BVTWVtUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgUGF1c2VDb250YWluZXJSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBQdWxsSW1hZ2VSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBSZWFkU3RyZWFtUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgUmVtb3ZlQ29udGFpbmVyUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgUmVtb3ZlU3RhbGVWaXJ0aW9mc1NoYXJlTW91bnRzUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgUmVzZWVkUmFuZG9tRGV2UmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgUmVzdW1lQ29udGFpbmVyUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgU2V0R3Vlc3REYXRlVGltZVJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFNldFBvbGljeVJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFNpZ25hbFByb2Nlc3NSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBTdGFydENvbnRhaW5lclJlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFN0YXJ0VHJhY2luZ1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFN0YXRzQ29udGFpbmVyUmVxdWVzdCA6PSB0cnVlCmRlZmF1bHQgU3RvcFRyYWNpbmdSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBUdHlXaW5SZXNpemVSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBVcGRhdGVDb250YWluZXJSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBVcGRhdGVFcGhlbWVyYWxNb3VudHNSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBVcGRhdGVJbnRlcmZhY2VSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBVcGRhdGVSb3V0ZXNSZXF1ZXN0IDo9IHRydWUKZGVmYXVsdCBXYWl0UHJvY2Vzc1JlcXVlc3QgOj0gdHJ1ZQpkZWZhdWx0IFdyaXRlU3RyZWFtUmVxdWVzdCA6PSB0cnVlCgpkZWZhdWx0IEV4ZWNQcm9jZXNzUmVxdWVzdCA6PSBmYWxzZQo=           
spec:
  runtimeClassName: kata-remote
  containers:
    - name: ubuntu
      image: ubuntu
      command: ["sleep"]
      args: ["infinity"]
      ports:
        - containerPort: 8888

kubectl exec -it test -c ubuntu -- bash
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "50a9c3a93edf60bca47023c496ab3b41168d53df9360af97fa06f76ec493bd60": cannot enter container d954a556fc517276a4ce51a1661b429e3c2c4b65d211902699b2078b6d612d71, with err rpc error: code = PermissionDenied desc = "ExecProcessRequest is blocked by policy": unknown

bpradipt · 2023-12-16T02:47:43Z

@stevenhorsman I couldn't find s390x binary for opa from the release page, however the opa support code is written in a way that it should be easy to extend and include opa binary for s390x podvm images.

stevenhorsman · 2023-12-18T09:16:47Z

@stevenhorsman I couldn't find s390x binary for opa from the release page, however the opa support code is written in a way that it should be easy to extend and include opa binary for s390x podvm images.

Good point - in kata-containers (https://github.com/kata-containers/kata-containers/pull/7769/files) we built it from source, but I'm not sure if there is any plan to get upstream to create s390x & ppc64le releases, or if we'll need to self-build in peer pods too. @BbolroC - might know ~~move~~ more about that?

BbolroC · 2023-12-18T09:56:47Z

@BbolroC - might know move about that?

I haven't raised any discussion internally yet. I will bring up this topic soon and update you guys on the plan. Thanks!

bpradipt · 2023-12-18T10:15:31Z

@stevenhorsman I couldn't find s390x binary for opa from the release page, however the opa support code is written in a way that it should be easy to extend and include opa binary for s390x podvm images.

Good point - in kata-containers (https://github.com/kata-containers/kata-containers/pull/7769/files) we built it from source, but I'm not sure if there is any plan to get upstream to create s390x & ppc64le releases, or if we'll need to self-build in peer pods too. @BbolroC - might know ~~move~~ more about that?

I can also add support to build the opa binary from source.
@stevenhorsman @BbolroC would that be fine ?

bpradipt · 2023-12-19T05:27:31Z

@stevenhorsman @BbolroC I have switched to building OPA from source. Ptal.

katexochen · 2023-12-19T06:12:12Z

Good point - in kata-containers (https://github.com/kata-containers/kata-containers/pull/7769/files) we built it from source, but I'm not sure if there is any plan to get upstream to create s390x & ppc64le releases, or if we'll need to self-build in peer pods too. @BbolroC - might know ~~move~~ more about that?

see open-policy-agent/opa#5838

podvm/files/etc/systemd/system/kata-opa.service

podvm/files/etc/kata-opa/allow-all.rego

podvm/files/etc/kata-opa/disallow-all-except-setpolicy.rego

podvm/Makefile.inc

stevenhorsman · 2024-01-08T11:25:38Z

Is there any intention to add an e2e test for this? I think without it and considering that a user can't run it with standard operator deploy as they need a custom payload we should add at the start of the docs that this is an untested, experimental feature and mention the manual adjustment required?

bpradipt · 2024-01-08T11:36:33Z

I added a Docker build arg to specify the agent policy file during building the binaries.

I'm not sure if my comment didn't make sense, but I don't think I understand this reply. I ran DEFAULT_AGENT_POLICY_FILE=allow-all-except-exec-process.rego make podvm-builder podvm-binaries podvm-image I think you would need a change in https://github.com/bpradipt/cloud-api-adaptor/blob/a2c636f4663fd0102c662392188a5f99848da135/Makefile#L218 to optionally pass through the DEFAULT_AGENT_POLICY_FILE to get that to work?

I'll add it. My intention was to avoid additional Makefile options if possible. So only added it in Dockerfile allowing anyone to use explicit docker build to use a different policy file than what is shipped and tested as part of upstream.

bpradipt · 2024-01-08T11:38:14Z

Is there any intention to add an e2e test for this? I think without it and considering that a user can't run it with standard operator deploy as they need a custom payload we should add at the start of the docs that this is an untested, experimental feature and mention the manual adjustment required?

e2e is definitely needed. But I would prefer to do it as a separate PR. Also for users, we expect them to deploy released version and main is anyway unstable. Or am I missing something ?

stevenhorsman · 2024-01-08T11:44:39Z

I'll add it. My intention was to avoid additional Makefile options if possible. So only added it in Dockerfile allowing anyone to use explicit docker build to use a different policy file than what is shipped and tested as part of upstream.

Sorry if I confused things further. I didn't necessarily mean for it to get added now (though we need to make our docs consistent long term), which is why I said we might want to look into, but not a big deal, I was just trying to explain the process I did as the docker build arg reply suggested that I hadn't explained properly

stevenhorsman · 2024-01-08T11:46:54Z

e2e is definitely needed. But I would prefer to do it as a separate PR. Also for users, we expect them to deploy released version and main is anyway unstable. Or am I missing something ?

I think that's fair, I didn't really mean users, more developers who want to play around with policy to let them know it doesn't work from the main branch at the moment?

bpradipt · 2024-01-08T12:03:58Z

e2e is definitely needed. But I would prefer to do it as a separate PR. Also for users, we expect them to deploy released version and main is anyway unstable. Or am I missing something ?

I think that's fair, I didn't really mean users, more developers who want to play around with policy to let them know it doesn't work from the main branch at the moment?

Personally my thinking is that operator deployment from main and kata payload are transient aspects.
I replaced the shim with latest CCv0 using a daemonset. But this is a hack and I don't think it makes sense to add it to the doc.
I can add a statement on requiring a shim with agent-policy support to the doc. Or if there is something else that would help, please suggest and I'll update this PR.
Or if you want to hold this PR till we have the right working payload, I'm fine with that as well :-)

stevenhorsman · 2024-01-08T12:09:19Z

I can add a statement on requiring a shim with agent-policy support to the doc

I think this is enough. I definitely appreciate the position you are coming from where this is dev code, not a release, so things aren't expected to work cleanly, and I don't want to hold up good work, but I'm aware that 0.9 is likely to be an extended release, so want to try and be sympathetic and let people try out the great new features as easily as possible. I think the answer though is doc that it needs a supporting shim and once we switch to the kata-containers main payload (hopefully not too far away) then we can remove that after testing.

bpradipt · 2024-01-08T13:43:20Z

@stevenhorsman the e2e failure looks unrelated

stevenhorsman · 2024-01-08T13:48:27Z

@stevenhorsman the e2e failure looks unrelated

I guess it is might be due to the problem fixed in #1651 ?

stevenhorsman · 2024-01-08T14:29:41Z

What tag of the CAA did you use in your testing? I'm just using the default quay.io/confidential-containers/cloud-api-adaptor:dev-d4496d008b65c979a4d24767979a77ed1ba21e76 with libvirt, and the CAA can't communicate with the peer pod VM (which I also saw last week on the latest CAA then, hence why I used the release one)? I just took the tag from 4 hours ago dev-aea376162b2ed59d81c4ae2fb0ad32b12073db19 and that worked :)

stevenhorsman · 2024-01-08T15:12:02Z

So when testing this - as soon as I add the io.katacontainers.config.agent.policy: cGFja2FnZSBh... annotation from the doc then I can't get the container started and the forwarder is shut down:

2024/01/08 15:04:10 [adaptor/cloud] agent proxy is ready
2024/01/08 15:04:10 [adaptor/proxy] shutting down socket forwarder
2024/01/08 15:04:10 [adaptor/cloud/libvirt] Deleting instance (20)

If I remove that annotation then the pod starts fine, but unfortunately with the instant shut-down I'm not sure how to debug what is going wrong with the system.
I've checked the worker node and it has the expected version of the kata-shim installed:

/opt/kata/bin/containerd-shim-kata-v2 --version
Kata Containers containerd shim (Golang): id: "io.containerd.kata.v2", version: 3.2.0-rc0, commit: d0df91935b8840036c2891b1f93dd8059ebe486a

I don't want to hold this up, so we can merge it based on the code (as long as the libvirt e2e regression tests pass, which hopefully they should after a rebase), but I'm not sure if anyone other than Pradipta, with a similarly hacked version has it working, so I that until we have an e2e test there is a big caveat over this feature (which isn't reflective of Pradipta's code, just that we are in an unstable state with the operator, peer-pods and kata-agent at the moment until everything can switch to main)

bpradipt · 2024-01-08T17:09:14Z

So when testing this - as soon as I add the io.katacontainers.config.agent.policy: cGFja2FnZSBh... annotation from the doc then I can't get the container started and the forwarder is shut down:
2024/01/08 15:04:10 [adaptor/cloud] agent proxy is ready
2024/01/08 15:04:10 [adaptor/proxy] shutting down socket forwarder
2024/01/08 15:04:10 [adaptor/cloud/libvirt] Deleting instance (20)
If I remove that annotation then the pod starts fine, but unfortunately with the instant shut-down I'm not sure how to debug what is going wrong with the system. I've checked the worker node and it has the expected version of the kata-shim installed:
/opt/kata/bin/containerd-shim-kata-v2 --version
Kata Containers containerd shim (Golang): id: "io.containerd.kata.v2", version: 3.2.0-rc0, commit: d0df91935b8840036c2891b1f93dd8059ebe486a
I don't want to hold this up, so we can merge it based on the code (as long as the libvirt e2e regression tests pass, which hopefully they should after a rebase), but I'm not sure if anyone other than Pradipta, with a similarly hacked version has it working, so I that until we have an e2e test there is a big caveat over this feature (which isn't reflective of Pradipta's code, just that we are in an unstable state with the operator, peer-pods and kata-agent at the moment until everything can switch to main)

We'll be in a state of flux for sometime. I feel as long as existing tests passes without adding a policy and also the default policy is to allow everything, we can continue working on the shim side.

This commit adds the required configurations for policy support in podvm based on OPA. The default policy is to allow all APIs. There are additional policy files included - disable exec api - allow only setpolicy api Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>

The following makefile options are added - AGENT_POLICY option for building the kata-agent with policy support. This is enabled by default - DEFAULT_AGENT_POLICY_FILE option to specify the policy file to be used as default. The default policy file is allow-all.rego Additionally the OPA binary is built from source since binaries for s390x is not available Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>

SetPolicy allows setting per-pod policy via annotations. This commit adds support for the same in cloud-api-adaptor The policy needs to be base64 encoded and provided as an annotation. Example ``` ... annotations: io.containerd.cri.runtime-handler: kata-remote io.katacontainers.config.agent.policy: <base64_rego_policy> ... ``` Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>

update unit test code to include SetPolicy Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>

This commit just enables the kata-opa service Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>

Provide details on policy configuration using OPA Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>

Agent policy file can be specified as part of containerised builds. The build arg is DEFAULT_AGENT_POLICY_FILE and it takes a policy file name kept under podvm/files/etc/kata-opa Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>

stevenhorsman · 2024-01-09T10:15:38Z

We're getting a failure in the kcli setup:

time="2024-01-08T20:03:06Z" level=info msg="Cluster provisioning"
Traceback (most recent call last):
  File "/usr/bin/kcli", line 33, in <module>
    sys.exit(load_entry_point('kcli==99.0', 'console_scripts', 'kcli')())
  File "/usr/lib/python3/dist-packages/kvirt/cli.py", line 5371, in cli
    args.func(args)
  File "/usr/lib/python3/dist-packages/kvirt/cli.py", line 131, in get_version
    git_version, git_date = get_git_version()
  File "/usr/lib/python3/dist-packages/kvirt/common/__init__.py", line 2112, in get_git_version
    git_version, git_date = open(git_file).read().rstrip().split(' ')
ValueError: not enough values to unpack (expected 2, got 1)
F0108 20:03:08.329146   27874 env.go:369] Setup failure: exit status 1
FAIL	github.com/confidential-containers/cloud-api-adaptor/test/e2e	1.793s
FAIL

I tried re-queuing it a few times last night and still got it and we saw it in the nightly run this morning: https://github.com/confidential-containers/cloud-api-adaptor/actions/runs/7456619508/job/20288166316. I had a quick look at https://github.com/karmab/kcli/blame/main/kvirt/common/__init__.py#L2107 and the code doesn't seem to have changed recently, but we'll need to dig in a bit more. I'll raise a separate issue for this though...

#1655 - issue raised

stevenhorsman

The code looks okay to me. I wasn't able to get it working myself, but we aren't in a very stable state wrt supported shim versions. Based on the discussions we've had about this being 'experimental'/'WIP', we'll need to do more work on this once we are using the main branch of the agent and shim e.g. adding e2e tests for the feature. However it doesn't impact the current e2e CI I think we can merge it to not block progress on this. Thanks for your patience and the updates Pradipta!

bpradipt mentioned this pull request Dec 1, 2023

packer: Enable support for adding nvidia gpu to podvm image #1602

Merged

bpradipt force-pushed the policy branch 4 times, most recently from 134f466 to 7466793 Compare December 14, 2023 17:24

bpradipt marked this pull request as ready for review December 14, 2023 17:24

bpradipt marked this pull request as draft December 15, 2023 08:22

bpradipt force-pushed the policy branch 4 times, most recently from 9051a5e to 69bf62a Compare December 15, 2023 17:53

bpradipt marked this pull request as ready for review December 16, 2023 02:36

bpradipt requested review from mkulke, stevenhorsman and katexochen and removed request for mkulke December 16, 2023 02:36

bpradipt force-pushed the policy branch from 69bf62a to 5104ec4 Compare December 19, 2023 05:22

bpradipt requested a review from snir911 December 19, 2023 05:28

mkulke reviewed Dec 19, 2023

View reviewed changes

bpradipt force-pushed the policy branch from a2c636f to 654b56b Compare January 8, 2024 11:41

bpradipt force-pushed the policy branch from 654b56b to 1027696 Compare January 8, 2024 12:18

bpradipt added 8 commits January 8, 2024 22:40

caa: Update tests to include SetPolicy

bbf2c9e

update unit test code to include SetPolicy Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>

podvm-mkosi: Enable opa for policy support

a92e87f

This commit just enables the kata-opa service Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>

doc: Add description of the Kata agent policy

5e62c86

Provide details on policy configuration using OPA Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>

bpradipt force-pushed the policy branch from 1027696 to ae9638a Compare January 8, 2024 17:12

stevenhorsman added test_e2e_libvirt Run Libvirt e2e tests and removed test_e2e_libvirt Run Libvirt e2e tests labels Jan 9, 2024

stevenhorsman approved these changes Jan 9, 2024

View reviewed changes

bpradipt merged commit 592c646 into confidential-containers:main Jan 10, 2024
38 of 39 checks passed

bpradipt deleted the policy branch January 10, 2024 04:06

stevenhorsman mentioned this pull request Jun 19, 2024

Get policies working on main based peer pods #1874

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add policy support to peer-pods #1607

Add policy support to peer-pods #1607

bpradipt commented Dec 1, 2023 •

edited

Loading

bpradipt commented Dec 14, 2023

bpradipt commented Dec 15, 2023

bpradipt commented Dec 16, 2023

bpradipt commented Dec 16, 2023 •

edited

Loading

bpradipt commented Dec 16, 2023

stevenhorsman commented Dec 18, 2023 •

edited

Loading

BbolroC commented Dec 18, 2023 •

edited

Loading

bpradipt commented Dec 18, 2023

bpradipt commented Dec 19, 2023

katexochen commented Dec 19, 2023

stevenhorsman commented Jan 8, 2024

bpradipt commented Jan 8, 2024

bpradipt commented Jan 8, 2024

stevenhorsman commented Jan 8, 2024

stevenhorsman commented Jan 8, 2024

bpradipt commented Jan 8, 2024

stevenhorsman commented Jan 8, 2024

bpradipt commented Jan 8, 2024

stevenhorsman commented Jan 8, 2024

stevenhorsman commented Jan 8, 2024 •

edited

Loading

stevenhorsman commented Jan 8, 2024 •

edited

Loading

bpradipt commented Jan 8, 2024

stevenhorsman commented Jan 9, 2024 •

edited

Loading

stevenhorsman left a comment

Add policy support to peer-pods #1607

Add policy support to peer-pods #1607

Conversation

bpradipt commented Dec 1, 2023 • edited Loading

bpradipt commented Dec 14, 2023

bpradipt commented Dec 15, 2023

bpradipt commented Dec 16, 2023

bpradipt commented Dec 16, 2023 • edited Loading

bpradipt commented Dec 16, 2023

stevenhorsman commented Dec 18, 2023 • edited Loading

BbolroC commented Dec 18, 2023 • edited Loading

bpradipt commented Dec 18, 2023

bpradipt commented Dec 19, 2023

katexochen commented Dec 19, 2023

stevenhorsman commented Jan 8, 2024

bpradipt commented Jan 8, 2024

bpradipt commented Jan 8, 2024

stevenhorsman commented Jan 8, 2024

stevenhorsman commented Jan 8, 2024

bpradipt commented Jan 8, 2024

stevenhorsman commented Jan 8, 2024

bpradipt commented Jan 8, 2024

stevenhorsman commented Jan 8, 2024

stevenhorsman commented Jan 8, 2024 • edited Loading

stevenhorsman commented Jan 8, 2024 • edited Loading

bpradipt commented Jan 8, 2024

stevenhorsman commented Jan 9, 2024 • edited Loading

stevenhorsman left a comment

Choose a reason for hiding this comment

bpradipt commented Dec 1, 2023 •

edited

Loading

bpradipt commented Dec 16, 2023 •

edited

Loading

stevenhorsman commented Dec 18, 2023 •

edited

Loading

BbolroC commented Dec 18, 2023 •

edited

Loading

stevenhorsman commented Jan 8, 2024 •

edited

Loading

stevenhorsman commented Jan 8, 2024 •

edited

Loading

stevenhorsman commented Jan 9, 2024 •

edited

Loading