Can't load CO-RE eBPF code that accesses enum values

[orishuss]

Describe the bug

When using bpf_core_enum_value_exists on an enum and within it bpf_core_enum_value on the same enum value, when the value doesn't exist, the output go binary fails to load.
When loading the compiled object file with libbpf (bpftool) it loads successfully.

The code is in the attached file, but the problem can be seen in:

if (bpf_core_enum_value_exists(enum cgroup_subsys_id, cpuset_cgrp_id_lublub)) {
    __attribute__((unused)) const volatile int val =
            bpf_core_enum_value(enum cgroup_subsys_id, cpuset_cgrp_id_lublub);
}

where the kernel enum cgroup_subsys_id is queried for type cpuset_cgrp_id_lublub (which doesn't really exist). The expected behaviour, which happens when loading the object with bpftool, is that the program passes the verifier's checks because the code inside the if shouldn't be evaluated.

The second value in the enum in the attached code, cpuset_cgrp_id, indeed exists, so using it instead of the fictitious value cpuset_cgrp_id_lublub works with both cilium and libbpf.

How to reproduce

The code:
enum-error.tar.gz

build with make and run with ./main.

Loading with bpftool (succeeds):

bpftool prog load bpf_bpf.o /sys/fs/bpf/bpf_bpf
rm /sys/fs/bpf/bpf_bpf # to remove

Version information

• github.com/cilium/ebpf v0.12.3
• go version go1.21.5 linux/amd64
• Linux 5.15.0

[dylandreimerink]

I was able to repo the issue and did some digging, found the following:

When I dump the program before CO-RE relocation logic the program looks like:

kprobe_tcp_close:
         ; void BPF_KPROBE(kprobe_tcp_close)
        0: LdImmDW dst: r1 imm: 1
         ; if (bpf_core_enum_value_exists(enum cgroup_subsys_id, cpuset_cgrp_id_lublub))
        2: JEqImm dst: r1 off: 3 imm: 0
        3: LdImmDW dst: r1 imm: 1
         ; __attribute__((unused)) const volatile int val =
        5: StXMemW dst: rfp src: r1 off: -4 imm: 0
         ; void BPF_KPROBE(kprobe_tcp_close)
        6: Exit

But afterwards, it looks like this:

kprobe_tcp_close:
         ; void BPF_KPROBE(kprobe_tcp_close)
        0: LdImmDW dst: r1 imm: 0
         ; if (bpf_core_enum_value_exists(enum cgroup_subsys_id, cpuset_cgrp_id_lublub))
        2: JEqImm dst: r1 off: 3 imm: 0
        3: Call BuiltinFunc(195896080)
         ; __attribute__((unused)) const volatile int val =
        4: StXMemW dst: rfp src: r1 off: -4 imm: 0
         ; void BPF_KPROBE(kprobe_tcp_close)
        5: Exit

Which in my case lead to a jump out of range from insn 2 to 6 error. This is because the LDIMM64 instruction which takes 2 instructions is replaced by a Call BuiltinFunc(195896080) which takes 1 instruction, this throws off offsets of the jumps.

The fixups are:

([]btf.COREFixup) (len=2 cap=2) {
 (btf.COREFixup) enumval_exists=1->0,
 (btf.COREFixup) enumval_value=poison
}

Tracing this, its this piece of code that is responsible

ebpf/btf/core.go

Lines 43 to 48 in ff37506

	if f.poison {
	const badRelo = 0xbad2310
	*ins = asm.BuiltinFunc(badRelo).Call()
	return nil
	}

[orishuss]

Cool! thanks
this also explains why when I used strace I saw BPF_PROG_LOAD with longer insn_cnt when using cilium than when using libbpf.