Testing

[kloenk]

Create some way to run rust test.

Bindgen creates tests, for allignment. maybe we can test them somehow.

[stappersg]

Few days after this issue was created, was merge request #52, "use rustc directly, created".

With this comment are now those two connected. How helpfull this comment (this connection) is something that can be told in a next comment.

Thing I'm aim for is a comment that tells which how helpful cargo test is, when builds are done without cargo. Or even a rename of this issue from cargo test to rust test.

[ojeda]

Yeah, we can use this issue for testing in general. It's something we should probably take a look at relatively soon.

[adamrk]

Seems like there was a sporadic test failure here: https://github.com/Rust-for-Linux/linux/runs/1921879052 (the only change from the previous commit which passed was a comment). Not sure if it's something to worry about.

[ojeda]

Hmm... It doesn't look like it is us, since it happens before we run any Rust code, but we could have broken some invariant somewhere nevertheless. Do you want to post it to the LKML? (It would be best to reproduce it first without the Rust stuff compiled in, though).

Related: perhaps we should add a CI run to compile things without enabling Rust but with our changes, and start using fail-fast: false to keep all the CI results.

[ojeda]

As I mentioned elsewhere, I will be splitting rust_example.rs into different kernel modules as poor man's tests.

Later on, after the RFC most likely, I will be working on setting up a proper test framework so that we can write tests as easy as in normal Rust.

[alastairreid]

[@wedsonaf pointed me at this issue because I am looking at the closely related question of 'how can we use formal verification tools with Rust?' - and Rust-for-Linux might be a really good use for the tools. In particular, I want to create a verification harness that can also be used as a test-harness or fuzzing harness.]

When looking at the current samples, it looks as if there are two important decisions about testing (or verification):

1. Should a test for the Rust code call into any of the C parts of the Linux kernel or should parts of rust/kernel/*.rs be replaced by a mocking library that mocks memory allocation, printk, device registration, mutexes, etc.?
2. Should a test for a device driver access the driver by invoking something that looks like Linux syscalls or should it access the driver by invoking Rust methods?

I am currently exploring the No/No choice. That is, I am modifying rust/kernel/.rs so that it does not rely on bindings:: types/functions and I am trying to write tests that use Rust types like UserSlicePtr, File, etc. and invoke Rust methods to manipulate those objects instead of trying to invoke drivers by creating inodes with the right major/minor device numbers and invoking the standard C-shaped system calls.

I'm making the No/No choice because I specifically want to focus on the Rust code and because I want to be able to write generic tests that benefit from Rust's traits, etc.
But note that other choices in this design space would also make sense. e.g., Tests that invoke both the Rust and the the C parts of the Linux kernel, have the potential to detect mismatches/misunderstandings between the Rust code and the C code whereas the tests I am writing could never find those because they ignore all the C code.

So far I have exactly one test working. It checks that samples/rust_chrdev.rs rejects attempts to read from the file. Not a particularly strenuous test for now: my focus is still on mocking enough of rust/kernel/*.rs that I can tackle more tricky code.

Once I have more of the mocking layer implemented, my plan is to use the KLEE verification tool to look for bugs such as integer overflows, array bounds violations, etc. in the Rust code. I will also use the same test harness with a fuzzer as an alternative way of looking for bugs. I am hoping that being able to verify/fuzz for these low level bugs would make it less painful/worrying to disable runtime checks in production builds.

Questions:

• Is anybody else working on tests?
• Does the No/No choice I am making fit with your thoughts on testing or do you think I should shift my focus?

[kloenk]

Just a quick idea. You could use no mangling and extern fn to create a mock library which shadows every C function used by the kernel rust crate. So you use the normal rust/kernel but that links against some custom kernel with shadows functions

[alastairreid]

Thanks. I think there's several places where I could cut the dependencies and insert my mock. The C interface has the huge advantage of being (I think) fairly stable. But it's also a bit low level.

Some places where I think it would be easier to modify rust/kernel/*.rs are as follows. (Where modify probably means 'refactor to expose the right interfaces and then use conditional compilation.)

First: to call FileOperations::read, you need to pass a &File. The File struct contains a pointer to bindings::file so I need to create a pointer to a file somehow. I think that the only way to create a file object is to invoke C code in the kernel. At the moment, I am YOLOing it and passing a null pointer but this only works because (AFAIK) the code I am testing does not use the two public methods on File both of which dereference it - but that is not going to work in the long term.
The cleanest solution that I can see seems to be to have an alternative implementation of File. (The underlying data structure has about 15-20 fields, lots of invariants, etc. and I think that replicating that code in Rust would be a mistake and that trying to invoke the C code would pull in a lot of dependencies.

Second: There are Rust functions that I think can be simplified down to no-ops (for testing purposes). For example, the Registration::register function in rust/kernel/miscdev.rs jumps through a bunch of hoops to initialize a C device object, construct a VTABLE, etc. so that it can call bindings::misc_register. If I understand this correctly, the only purpose for this is so that the C part of the Linux kernel can use the major/minor device numbers on an inode to decide which device driver to invoke. Since my plan for testing sidesteps that mechanism, this is not needed. (This is a bit less clear cut - and, in this case, your suggestion also works and is simpler in some ways.)

Third: In part because formal verification of Rust is in its infancy, there don't seem to be any verifiers that support concurrency at the moment. So the entire content of rust/kernel/sync is a problem because the verifier does not support threads, mutexes, semaphores, etc. Getting the right behaviour (or, at least, a sufficiently good approximation) here will almost certainly require some major surgery. This won't necessarily affect conventional testing and fuzzing - but it is a major issue for me.

At the moment, my feeling is that I am going to have to use several different approaches here: sometimes making the cut at the Rust-C interface as you suggest, sometimes making the cut in the Rust code. But it's early days and I expect that I will end up having to discard and redo my first attempt or two...

[ojeda]

@alastairreid

Is anybody else working on tests?

My plan is to work on supporting unit tests. I am not aware of anybody working on anything else.

At the moment, we only have a few sanity functional tests for the CI.

Does the No/No choice I am making fit with your thoughts on testing or do you think I should shift my focus?

Hard to answer :) Perhaps having a meeting for this would be best.

My gut feeling is that userspace functional tests are currently the most valuable given it is still early days. They are likely the easiest to write and the most stable ones.

Some places where I think it would be easier to modify rust/kernel/*.rs are as follows. (Where modify probably means 'refactor to expose the right interfaces and then use conditional compilation.)

In general, I would be cautious about changing interfaces to accommodate testing/mocking.

This is not to say we should not do it: if we can show it has a big impact, we should definitely push for it. However, we first need to get Rust proven within the kernel.

I am hoping that being able to verify/fuzz for these low level bugs would make it less painful/worrying to disable runtime checks in production builds.

There may be some particular places where we want to do unchecked operations due to performance reasons, and the policy is to justify every unsafe block in the form of a comment (and, of course, ideally they should be proven/tested/fuzzed/etc.). But, in general, I do not think we should encourage users to run kernels with checks disabled.

The C interface has the huge advantage of being (I think) fairly stable.

Please note that the kernel internal APIs are not stable. However, it is true that, at the moment, the Rust ones are likely to change more than the C ones (by virtue of being new). But longer-term there would be nothing that would make the Rust ones more or less stable than the C ones.

my plan is to use the KLEE verification tool to look for bugs such as integer overflows, array bounds violations, etc. in the Rust code.

That sounds great. Has there been any project on using KLEE for the C side? Any showstoppers?

Another thing I would like to have is Miri/KASAN/UBSAN etc. for the Rust side.