Skip to content

using BTF to verify userspace eBPF extensions

Here we will show how to use the type information of userspace application to verify the eBPF program which will access the userspace memory(include valid or invalid data structure memory access, pointer access, etc), and how to verify the resource allocation and deallocation of a psudo ufuncs(Userspace Function) which can be accessed by eBPF program in bpftime.

This is using kernel eBPF verifier to verify userspace eBPF extensions, the userspace eBPF extensions can be uprobe or other eBPF programs, it can be run in the kernel space or in bpftime.

Usage

The application developer or the extension user who wants to use the eBPF program for userspace application needs to provide:

  1. The BTF information of the userspace application. It can be generated by the compiler or from the DWARF information.
  2. The psudo ufunc information if the eBPF program will access the resource allocation and deallocation function of userspace application.

By reusing the kernel verifier, we can provide:

  1. CO-RE relocation, if the userspace application has different version, and the struct/function type is different.
  2. Type checking and variable access checking for the userspace extension.
  3. Memory access checking for the userspace extension, so no need to copy the data.
  4. Resource allocation and deallocation function access checking.

The BTF format

The BTF format is a binary format that contains the type information of the progam.

The based program can be found in the examples directory, they represent the different version of host userspace applications, include:

  • btf-base: the base version of btf-base-complete, which contain the type information of struct data the same as verify-failed-no-btf/uprobe.bpf.c.
  • btf-base-new: the new version of btf-base, which contains the type information of struct data different from btf-base and verify-failed-btf/uprobe.bpf.c. We can use it to test CO-RE relocation.
  • btf-base-complete: the complete version of btf-base, which contains all the types that are used in the eBPF program, and a resource allocation and deallocation function.

run make -C ../ in this directory to generate the BTF files.

Verify struct data access

The userspace application has data struct like

version 1:

struct data {
    int a;
    int c;
    int d;
};

or

version 2:

struct data {
    int a;
    int b;
    int c;
    int d;
};

1.1 Verify failed without BTF for userspace application

The eBPF program has

struct data {
    int a;
    int c;
    int d;
};

SEC("uprobe/examples/btf-base:add_test")
int BPF_UPROBE(add_test, struct data *d)
{
    int a = 0, c = 0;
    bpf_probe_read_user(&a, sizeof(a), &d->a);
    bpf_probe_read_user(&c, sizeof(c), &d->c);
    bpf_printk("add_test(&d) %d + %d = %d\n", a, c,  a + c);
    return a + c;
}

If no BTF for userspace application is provided, the verification will fail.

$ sudo ./verify-failed-no-btf/uprobe ../examples/btf-base 

libbpf: prog 'add_test': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int BPF_UPROBE(add_test, struct data *d) @ uprobe.bpf.c:23
0: (79) r6 = *(u64 *)(r1 +112)        ; R1=ctx() R6_w=scalar()
1: (b7) r7 = 0                        ; R7_w=0
; int a = 0, c = 0; @ uprobe.bpf.c:25
2: (63) *(u32 *)(r10 -4) = r7         ; R7_w=0 R10=fp0 fp-8=0000????
3: (63) *(u32 *)(r10 -8) = r7         ; R7_w=0 R10=fp0 fp-8=00000
4: <invalid CO-RE relocation>
failed to resolve CO-RE relocation <byte_off> [17] struct data.a (0:0 @ offset 0)
processed 5 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'add_test': failed to load: -22
libbpf: failed to load object 'uprobe_bpf'
libbpf: failed to load BPF skeleton 'uprobe_bpf': -22
Failed to load and verify BPF skeleton

1.2 Verify success with BTF for struct access and relocation

See ../README.md for more details.

$ sudo ./verify-failed-no-btf/uprobe ../examples/btf-base ../target-base.btf

libbpf: prog 'add_test': relo #0: <byte_off> [2] struct pt_regs.di (0:14 @ offset 112)
libbpf: prog 'add_test': relo #0: matching candidate #0 <byte_off> [88] struct pt_regs.di (0:14 @ offset 112)
libbpf: prog 'add_test': relo #0: patched insn #0 (LDX/ST/STX) off 112 -> 112
libbpf: CO-RE relocating [17] struct data: found target candidate [143229] struct data in [vmlinux]
libbpf: prog 'add_test': relo #1: <byte_off> [17] struct data.a (0:0 @ offset 0)
libbpf: prog 'add_test': relo #1: matching candidate #0 <byte_off> [143229] struct data.a (0:0 @ offset 0)
libbpf: prog 'add_test': relo #1: patched insn #4 (ALU/ALU64) imm 0 -> 0
libbpf: prog 'add_test': relo #2: <byte_off> [17] struct data.c (0:1 @ offset 4)
libbpf: prog 'add_test': relo #2: matching candidate #0 <byte_off> [143229] struct data.c (0:1 @ offset 4)
libbpf: prog 'add_test': relo #2: patched insn #11 (ALU/ALU64) imm 4 -> 4
libbpf: map '.rodata.str1.1': created successfully, fd=3
libbpf: elf: symbol address match for 'add_test' in '../examples/btf-base': 0x1140
Successfully started! Press Ctrl+C to stop.

It can successfully find the type information of struct data in the BTF file and perform the verification.

1.3 Verify failed with invalid userspace struct var access

The eBPF program is:

struct data {
    int a;
    int e;
    int d;
};

SEC("uprobe/examples/btf-base:add_test")
int BPF_UPROBE(add_test, struct data *d)
{
    int a = 0, e = 0;
    bpf_probe_read_user(&a, sizeof(a), &d->a);
    bpf_probe_read_user(&e, sizeof(e), &d->e);
    bpf_printk("add_test(&d) %d + %d = %d\n", a, e,  a + e);
    return a + e;
}

If the extension (BPF program) access the invalid userspace struct variable, the verification will fail.

Verify memory pointer access

See verify-memory-access directory.

The data struct is:

struct deep_memory_block {
    int a;
    char b[10];
};

struct inner_memory_block {
    int a;
    char b[10];
    struct deep_memory_block *deep;
};

struct data {
        int a;
        int b;
        int c;
        int d;
        // represent a pointer to a memory block
        struct inner_memory_block *inner;
};

This include a deep memory access through pointers.

Memory access success for inner data structs without deep copy

The eBPF program is:

SEC("uprobe/examples/btf-base:add_test")
int BPF_UPROBE(add_test, struct data *d)
{
    int inner_deep_a = BPF_CORE_READ_USER(d, inner, deep, a);
    bpf_printk("inner_deep_a = %d\n", inner_deep_a);
    char* inner_deep_b = BPF_CORE_READ_USER(d, inner, deep, b);
    bpf_printk("inner_deep_b[9] = %c\n", inner_deep_b[9]);
    return 0;
}

Run it:

$ sudo ./verify-memory-access/uprobe ../examples/btf-base-complete ../target-base-complete.btf 

libbpf: sec 'uprobe/examples/btf-base:add_test': found 7 CO-RE relocations
libbpf: CO-RE relocating [2] struct pt_regs: found target candidate [88] struct pt_regs in [vmlinux]
libbpf: prog 'add_test': relo #0: <byte_off> [2] struct pt_regs.di (0:14 @ offset 112)
libbpf: prog 'add_test': relo #0: matching candidate #0 <byte_off> [88] struct pt_regs.di (0:14 @ offset 112)
libbpf: prog 'add_test': relo #0: patched insn #0 (LDX/ST/STX) off 112 -> 112
libbpf: CO-RE relocating [17] struct data: found target candidate [143228] struct data in [vmlinux]
libbpf: prog 'add_test': relo #1: <byte_off> [17] struct data.a (0:0 @ offset 0)
libbpf: prog 'add_test': relo #1: matching candidate #0 <byte_off> [143228] struct data.a (0:0 @ offset 0)
libbpf: prog 'add_test': relo #1: patched insn #4 (ALU/ALU64) imm 0 -> 0
libbpf: prog 'add_test': relo #2: <byte_off> [17] struct data.c (0:2 @ offset 8)
libbpf: prog 'add_test': relo #2: matching candidate #0 <byte_off> [143228] struct data.c (0:2 @ offset 8)
libbpf: prog 'add_test': relo #2: patched insn #11 (ALU/ALU64) imm 8 -> 8
libbpf: prog 'add_test': relo #3: <byte_off> [17] struct data.inner (0:4 @ offset 16)
libbpf: prog 'add_test': relo #3: matching candidate #0 <byte_off> [143228] struct data.inner (0:4 @ offset 16)
libbpf: prog 'add_test': relo #3: patched insn #38 (ALU/ALU64) imm 16 -> 16
libbpf: CO-RE relocating [19] struct inner_memory_block: found target candidate [143231] struct inner_memory_block in [vmlinux]
libbpf: prog 'add_test': relo #4: <byte_off> [19] struct inner_memory_block.deep (0:2 @ offset 16)
libbpf: prog 'add_test': relo #4: matching candidate #0 <byte_off> [143231] struct inner_memory_block.deep (0:2 @ offset 16)
libbpf: prog 'add_test': relo #4: patched insn #46 (ALU/ALU64) imm 16 -> 16
libbpf: CO-RE relocating [24] struct deep_memory_block: found target candidate [143234] struct deep_memory_block in [vmlinux]
libbpf: prog 'add_test': relo #5: <byte_off> [24] struct deep_memory_block.a (0:0 @ offset 0)
libbpf: prog 'add_test': relo #5: matching candidate #0 <byte_off> [143234] struct deep_memory_block.a (0:0 @ offset 0)
libbpf: prog 'add_test': relo #5: patched insn #52 (ALU/ALU64) imm 0 -> 0
libbpf: prog 'add_test': relo #6: <byte_off> [24] struct deep_memory_block.b (0:1 @ offset 4)
libbpf: prog 'add_test': relo #6: matching candidate #0 <byte_off> [143234] struct deep_memory_block.b (0:1 @ offset 4)
libbpf: prog 'add_test': relo #6: patched insn #84 (ALU/ALU64) imm 4 -> 4
libbpf: map '.rodata.str1.1': created successfully, fd=3
libbpf: elf: symbol address match for 'add_test' in '../examples/btf-base-complete': 0x1160
Successfully started! Press Ctrl+C to stop.

Memory access failed for inner data structs because of out of bounds

If we change the eBPF program to access the memory out of bounds, the verification will fail.

SEC("uprobe/examples/btf-base:add_test")
int BPF_UPROBE(add_test, struct data *d)
{
    int inner_deep_a = BPF_CORE_READ_USER(d, inner, deep, a);
    bpf_printk("inner_deep_a = %d\n", inner_deep_a);
    char* inner_deep_b = BPF_CORE_READ_USER(d, inner, deep, b);
    // memory access out of bounds
    bpf_printk("inner_deep_b[11] = %c\n", inner_deep_b[100]);
    return 0;
}

test:

$ sudo ./verify-memory-access/uprobe_failed ../examples/btf-base-complete ../target-base-complete.btf 
...
; bpf_printk("inner_deep_b[11] = %c\n", inner_deep_b[100]); @ uprobe_failed.bpf.c:44
57: (6b) *(u16 *)(r10 -12) = r1       ; R1_w=2659 R10=fp0 fp-16=??mm?mmm
58: (b7) r1 = 622869792               ; R1_w=0x25203d20
59: (63) *(u32 *)(r10 -16) = r1       ; R1_w=0x25203d20 R10=fp0 fp-16=??mm0x25203d20
60: (18) r1 = 0x5d31315b625f7065      ; R1_w=0x5d31315b625f7065
62: (7b) *(u64 *)(r10 -24) = r1       ; R1_w=0x5d31315b625f7065 R10=fp0 fp-24_w=0x5d31315b625f7065
63: (7b) *(u64 *)(r10 -32) = r8       ; R8=0x65645f72656e6e69 R10=fp0 fp-32_w=0x65645f72656e6e69
64: (73) *(u8 *)(r10 -10) = r9        ; R9=0 R10=fp0 fp-16=?0mmmmmm
65: (71) r3 = *(u8 *)(r10 +68)
invalid read from stack R10 off=68 size=1
processed 63 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 2
-- END PROG LOAD LOG --
libbpf: prog 'add_test': failed to load: -13
libbpf: failed to load object 'uprobe_failed_bpf'
libbpf: failed to load BPF skeleton 'uprobe_failed_bpf': -13
Failed to load and verify BPF skeleton

Verify resource allocation and deallocation with psudo ufuncs

See verify-resource-allocation directory.

First, give kernel the psudo ufunc information through the kernel module:

cd module
make
sudo insmod hello.ko

Check the kernel output:

$ sudo dmesg | tail
[83824.574456] Hello, world!
[83824.574952] bpf_kfunc_example: Module loaded successfully

The psudo ufunc information is like:

__bpf_kfunc struct data *my_alloc_data(void)
{
    // here we only use it for verification
    return NULL;
}

__bpf_kfunc void my_free_data(struct data *d)
{
    // here we only use it for verification
    return;
}
/*Auto generated code end*/

__bpf_kfunc_end_defs();

BTF_KFUNCS_START(bpf_kfunc_example_ids_set)
BTF_ID_FLAGS(func, my_alloc_data, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, my_free_data, KF_RELEASE)
BTF_KFUNCS_END(bpf_kfunc_example_ids_set)

Note in previous approach, we just use the BTF information of userspace program to verify the eBPF program, so we don't need a additional kernel module. In this case, the kernel may try to link the psudo kfunc to the eBPF program. If it don't have a implementation in the module, the verification will fail.

And this can only be run in userspace, since run it in kernel is meaningless(The "ufunc" is just a placeholder kfunc, don't have a implementation in the kernel).

Verify resource allocation and deallocation success with psudo ufunc information

The eBPF progam is like:

SEC("uprobe/examples/btf-base:add_test")
int BPF_UPROBE(add_test, struct data *d)
{
    struct data *alloced = my_alloc_data();
    if (alloced == NULL) {
        bpf_printk("Failed to allocate data\n");
        return 0;
    }
    my_free_data(alloced);
    return 0;
}

Then you can run it successfuly:

$ sudo ./verify-resource-allocation/uprobe ../examples/btf-base-complete ../target-base-complete.btf 

libbpf: loaded kernel BTF from '/sys/kernel/btf/vmlinux'
libbpf: extern (func ksym) 'my_alloc_data': resolved to hello [143233]
libbpf: extern (func ksym) 'my_free_data': resolved to hello [143235]
libbpf: map '.rodata.str1.1': created successfully, fd=3
libbpf: elf: symbol address match for 'add_test' in '../examples/btf-base-complete': 0x1160
Successfully started! Press Ctrl+C to stop.

Verify resource allocation and deallocation failed

If don't release resource allocated by userspace application, the verification will fail.

code:

SEC("uprobe/examples/btf-base:add_test")
int BPF_UPROBE(add_test, struct data *d)
{
    struct data *alloced = my_alloc_data();
    if (alloced == NULL) {
        bpf_printk("Failed to allocate data\n");
        return 0;
    }
    return 0;
}

run it:

$ sudo ./verify-resource-allocation/uprobe_failed ../examples/btf-base-complete ../target-base-complete.btf 

0: R1=ctx() R10=fp0
; struct data *alloced = my_alloc_data(); @ uprobe_failed.bpf.c:43
0: (85) call my_alloc_data#143233     ; R0_w=ptr_or_null_data(id=2,ref_obj_id=2) refs=2
; if (alloced == NULL) { @ uprobe_failed.bpf.c:44
1: (55) if r0 != 0x0 goto pc+15 17: R0_w=ptr_data(ref_obj_id=2) R10=fp0 refs=2
17: (b7) r0 = 0                       ; R0_w=0 refs=2
18: (95) exit
Unreleased reference id=2 alloc_insn=0
processed 18 insns (limit 1000000) max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'add_test': failed to load: -22
libbpf: failed to load object 'uprobe_failed_bpf'
libbpf: failed to load BPF skeleton 'uprobe_failed_bpf': -22
Failed to load and verify BPF skeleton

Or if you don't check the return value of my_alloc_data, the verification will also fail, like:

SEC("uprobe/examples/btf-base:add_test")
int BPF_UPROBE(add_test, struct data *d)
{
    struct data *alloced = my_alloc_data();

    my_free_data(alloced);
    return 0;
}

run it:

$ sudo ./verify-resource-allocation/uprobe ../examples/btf-base-complete ../target-base-complete.btf 

libbpf: loaded kernel BTF from '/sys/kernel/btf/vmlinux'
libbpf: extern (func ksym) 'my_alloc_data': resolved to hello [143233]
libbpf: extern (func ksym) 'my_free_data': resolved to hello [143235]
libbpf: prog 'add_test': BPF program load failed: Permission denied
libbpf: prog 'add_test': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; struct data *alloced = my_alloc_data(); @ uprobe.bpf.c:43
0: (85) call my_alloc_data#143233     ; R0_w=ptr_or_null_data(id=2,ref_obj_id=2) refs=2
; my_free_data(alloced); @ uprobe.bpf.c:45
1: (bf) r1 = r0                       ; R0_w=ptr_or_null_data(id=2,ref_obj_id=2) R1_w=ptr_or_null_data(id=2,ref_obj_id=2) refs=2
2: (85) call my_free_data#143235
Possibly NULL pointer passed to trusted arg0
processed 3 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'add_test': failed to load: -13
libbpf: failed to load object 'uprobe_bpf'
libbpf: failed to load BPF skeleton 'uprobe_bpf': -13
Failed to load and verify BPF skeleton

Run ufunc in userspace

Since it's meaningless to run ufunc in kernel, so we can only run it in userspace.

load eBPF program with ufunc:

LD_PRELOAD=/home/yunwei37/.bpftime/libbpftime-syscall-server.so ./verify-resource-allocation/uprobe ../examples/btf-base-complete ../target-base-complete.btf

And run the userspace program:

# LD_PRELOAD=/home/yunwei37/.bpftime/libbpftime-agent.so ../examples/btf-base-complete

my_alloc_data
my_free_data

Share on Share on