Startup initialized statics #1674
Conversation
| ``` | ||
| Remember that SI statics are never placed in read-only memory, which makes the transmute to `&mut` not be undefined behaviour (I think). | ||
|
|
||
| In order to both catch circular dependencies between SI functions and to be able to determine the order in which SI functions should be called at the beginning of main, the compiler must keep track of which SI statics each function and each static item (through a reference) may potentially access. An SI static must not be accessed (neither read nor written to) before its associated SI function is called (and while said SI function call is running, we allow all functions to read and write to the SI static if they take it as a `&mut` argument as long as they don't store a long-lived mutable reference to it, but I think Rust's borrow checking rules prevent this from happening). |
There was a problem hiding this comment.
This seems like a lot of nonlocal analysis that is very different than anything that currently exists in the compiler or language.
There was a problem hiding this comment.
@sfackler I don't really know what non-local analysis means, but just based on how it sounds, it seems to me that the static analysis required here is quite local being pretty well encapsulated to each function, because the analysis relies on the meta-data (dependency sets) that were determined earlier (for the functions being called and statics being accessed) and cached somewhere.
There was a problem hiding this comment.
@tommit Changes in unrelated places can make the initializer fail to compile because it suddenly accesses the static
There was a problem hiding this comment.
@jonas-schievink Do you mean that by changing some code in one place can introduce a cyclic dependency and an initializer in another place would fail to compile? I wouldn't call those places unrelated then. In fact, I mention in the RFC that the error message in such a situation should mention which functions are responsible in causing the circular dependency, and therefore the function you just modified would probably be mentioned in the error message.
There was a problem hiding this comment.
Maybe "distant" would be a better word. Still, I don't think calling a function should impose restrictions on the callee's body.
There was a problem hiding this comment.
@jonas-schievink If by "restriction" you refer to the fact that dependency cycles are not allowed during the initialization of static data, then I wouldn't call that so much a restriction but rather "catching a bug at compile-time", which is a good thing considering the alternative today (if you wish to have zero-overhead access time to statics and also nice ergonomics), where you have to use mutable statics, the dependency cycles will compile just fine and maybe you'll get a runtime error (if you're lucky and didn't make errors in trying to catch them). And let's not forget that the resulting library is safe only if the user reads the documentation and remembers to call your initialization function before doing anything else.
nrc
added
the
T-lang
|
This seems like a very complex addition for the little benefit it gives us. 2 new attributes, paths in attributes, a non-local inter-function analysis, magically inserted code at the beginning of |
|
I haven't had a need for statics yet, but I nevertheless don't like this either. First, it should be possible to take the restrictions this RFC proposes and simply apply them to the expression used to initialize the static. As you're stating it, I have to put two attributes in, then define the initialization in another function, possibly even in another source file. Why can't I just put it with the static? Rust already makes blocks expressions, so it's not like it's going to really be a big syntax extension to do it. Second, this RFC appears to make no allowance for code which calls into Rust from outside. One use case of Rust is to b used as a DLL/shared library from a C FFI. At the very least, you'd need to call it from dllMain or the equivalent on non-windows platforms. Third, what if I statically link Rust into some C code and that C code provides the main? I'm not sure if this is currently possible, but I think it is. If it isn't, it might be a nice thing we'd want in the future: large, dependencyless binaries are becoming popular. As a Windows developer, I can say that it's a big deal to me personally, as well. In this case, where do the statics get initialized? Fourth, what's the big deal with unsafe initialization of statics? I know that Rust's goal is to make everything safe, but it's not very hard to design your interface as follows:
And finally, what's the use of a static that's initialized in this manner, if I can't unsafely mutate it elsewhere? As I understand things, I'm still going to have to either use unsafe or make all my statics mutexes if I want to mutate them. The former case is unsafe and thus this RFC doesn't gain anything. The latter case is still a really large amount of overhead and can probably be made ergonomic in a library, possibly through some sort of static-specific mutex. If I'm putting things that have interior mutability into statics, the implication to me at least is that they're encapsulating some sort of I/O. Such things don't belong in statics because they involve operations which might fail, and the only way to signal failure at the time statics have to be initialized is with a panic. This seems like a lot for nothing, or pretty close to it. I think there are better ways to solve the problem this is solving, and I don't think it solves the problems that need to be solved. |
|
@camlorn
I didn't think it would be too much of a hassle to put in two attributes, and I took my cue from Rust being explicit rather than implicit. But if it is too much work, then we could simply get rid of the I can't think of any reason why you would/should ever want to put the startup initialization function to a different module from the associated startup initialized statics. Your second & third point: Your fourth point: Your final point: |
|
Why do I care which type of initialization is being used? in both cases, it's initialized before main. As I understand it, statics have to be allocated in read-write memory anyway because we're allowed to mutate them in unsafe blocks. From my point of view, being able to initialize it by putting something in the binary that's not code is just an optimization, not something fundamentally different. When Miri and the constant stuff it brings comes along, it's possible we'll even be able to optimize away expressions that would now require this sort of initialization. Either way, the static gets initialized before I use it. I'm pretty sure that you can't safely display a progress bar. This would require doing UI initialization beforehand. The conditions that you have to add to make that kind of thing work get as bad as the darker corners of C++. If you add this, you've taken this RFC from somewhat complicated land to very scary and black magic land. What is a use case for a static that is initialized in this way, but which I never modify later? I don't mean one that you could use it for; I mean one where it's absolutely necessary. I can think of lots of ways you could use it, but they all involve things like opening I/O devices with interior mutability, and as I said you can't signal failure. As I see it, this RFC solves the easiest 1% of the safety problem around statics while leaving the other 99% out in the cold. I really don't see a use case where this RFC is the best tool and I don't have to use unsafe anywhere else. the design I proposed to hide the unsafe initialization yourself is a bit ugly. I won't deny that. But I think most people here will agree that it's a good one. In addition to making static initialization safe through the interface, it's also a design which can let you just avoid them to start with. It also allows you to change your mind later without breaking your public API. From a philosophical point of view, I think that statics should be hard to use mutably. Everything I've ever seen, been taught, or done has backed this up. I can think of exactly one place where a mutable global variable was the answer, and that was a very special case to do with getting a C++ library to have a C API that could talk to the Python/any other language with finalizers GC. If I'm being extra pessimistic, there's also a newbie concern along these lines. I come along from Python or something. I learn Rust. I see the part of the book that we add on this RFC. I structure my code to use a static because how convenient! Half an hour or a day of coding later, I need to modify it. But restructuring my code is expensive. So I pull out unsafe all over the place. And now you've got a project that would otherwise have been safe rust ruined by more unsafety than it would have been without it. My point about putting the initializer in a separate file isn't that I would want to. It's that someone will. A language should do the maximum it can to encourage good design by default. having a statics.rs is good. Having a static_initializers.rs and statics all over your project is both bad and totally possible, and the rule with both bad and totally possible things is that some programmer somewhere will always, always do them and not understand why they shouldn't. Whether or not you agree with me, I think it's my practical point about static linking into C executables that's going to bring this down. There may be a way to deal with this, but I've been using C/C++ for a very long time and haven't ever heard of one. It's the kind of thing where we need someone familiar with the very darkest corners of linking on all three major compilers to chime in, and I still think that fails to solve all the concerns around embedded systems. But embedded systems are way outside my area of expertise. |
| static DATA: Data = [1,2,3]; | ||
|
|
||
| #[startup_initialization] | ||
| fn init_my_data() { |
There was a problem hiding this comment.
Why not make init_my_data take a &mut Data? Then you don't need to go through all those unsafe transformation.
#[startup_initialization]
fn init_my_data(data: &mut Data) {
data[0] = 11;
mutate(data);
}There was a problem hiding this comment.
@kennytm This wouldn't get rid of the unsafe transformation (i.e. transmute to &mut) but only hoist it to the caller. And like the RFC said, inside the body of a startup initialization function, all of its associated startup initialized statics are made mutable. So, you might need more than one argument. Also, it would obfuscate what exactly is being mutated:
type Data = [i8; 3];
#[startup_initialized(init_my_data)]
static DATA_1: Data = [1,2,3];
#[startup_initialized(init_my_data)]
static DATA_2: Data = [4,5,6];
#[startup_initialization]
fn init_my_data(x: &mut Data, y: &mut Data) {
x[0] = 11; // what does x refer to?
y[0] = 22; // what does y refer to?
}There was a problem hiding this comment.
@tommit I would expect you call different functions for each static. If you annotate #[startup_initialized(init_my_data)] twice wouldn't init_my_data be called twice? That doesn't make sense.
There was a problem hiding this comment.
@kennytm You can have different functions for each static, but sometimes you may want to initialize multiple statics inside the same loop for example. And init_my_data would be called only once. I admit that the way the RFC says this is not the clearest sentence: "At the beginning of the main function, all SI functions of the current crate and all SI functions of all the extern crates it depends upon are called implicitly by the compiler, once each".
|
@camlorn
I think you should be at least somewhat wary of the possibility of cyclic dependencies among runtime-initializers. I admit that it wouldn't be that big of a problem given my RFC, since cycles would fail to compile.
That's not exactly true. The Rust Reference has this to say: "Static items may be placed in read-only memory if they do not contain any interior mutability". I did try that out (transmute a non-mut static array of ints to &mut and then try to mutate it) and it resulted in undefined behaviour.
That sounds interesting. I didn't know that (nor about Miri in general).
I can't think of a use case where this is "absolutely necessary", because there are those alternatives which have been mentioned. But each alternative has some drawback (either lowered ergonomics or performance) that this RFC doesn't have.
Can you elaborate a bit on why it is bad to have statics all over your project? I'm asking because that's what I currently have (non-mut statics that is).
Couldn't we just require the C/C++ side to explicitly call the initialization functions in the right order in the beginning of their main? |
|
I just realized that there's a bug in this RFC's "Possible implementation" part as it's currently written. The problem is that taking a reference to data is not considered "accessing" the data. One possible way to fix this bug is to change it so that taking a pointer or a reference to a startup initialized static (or a part of it) would be considered "accessing" the static. The example is silly but it does demonstrate the illegal access that's currently possible: const WHATEVER: i8 = 111;
#[startup_initialized(init_x)]
static DATA_X: &'static i8 = &WHATEVER;
#[startup_initialized(init_y)]
static DATA_Y: i8 = 123;
#[startup_initialized(init_z)]
static DATA_Z: i8 = 0;
#[startup_initialization]
fn init_x() {
DATA_X = &DATA_Z; // Taking a reference is not considered "accessing" data
}
#[startup_initialization]
fn init_y() {
let x = *DATA_X; // Whoops, reading "uninitialized" data
DATA_Y = x + 1;
}
#[startup_initialization]
fn init_z() {
DATA_Z = DATA_Y + 100;
}
fn main() {
// The implicit initializer calls:
init_x(); // Dependency set: {}
init_y(); // Dependency set: {init_x}
init_z(); // Dependency set: {init_y}
} |
|
On top of my other problems with this, I just figured out a safety concern. What stops me starting some threads in these functions, and then having a data race? The compiler is obviously letting me write to statics without unsafe. I don't see anything here about not being allowed to call other functions or being allowed to use lambdas. So doing that is totally possible. And wanting threads when I initialize my statics is totally a thing. That's the fastest way to precompute very large tables, assuming you can't use a If we want to go here, we should do what I discussed and make it the return value of a non-const expression and disallow any access to other statics. I think this would be safe. I'm trying to think if there's a way that a thread could escape from that environment, carrying the special magic I can modify statics power with it. If this turns into some sort of mini-Rust where you can't use half the features for safety, you will not be able to convince me that this solution is a good idea. Likewise if the analysis that has to be performed means that using this RFC is going to send my compile times through the roof. At the moment, I still have an open mind. I don't like it, but I presume that someone might be able to provide a motivating example I haven't thought of. Assuming that we can't initialize statics properly if we're embedding into a C program providing main, we have three choices, each with a problem: We can provide a compiler generated We can do what you suggested and make the C component call the functions in the right order. But this is going to be possibly tens of functions, and the only one who knows what the right order is is the compiler's algorithm. It also drags in anything from an upstream crate. In addition, you still have possible name clashes. We can insert special code on all functions that can be called from C that checks a flag on every call. if the flag isn't set, it initializes the statics in a threadsafe manner. This would work and work transparently, but a goal of this RFC is performance. Since Firefox is now shipping Rust components, this is important and not a hypothetical problem. Someone who knows a lot about the dark corners of C/C++ linking is going to have to chime in to tell me if I'm concerned for no reason or not. No solution I can think of other than making Rust insert before-main code into the C app without my intervention is even close to ideal. |
|
@camlorn
I don't doubt your word, but could you describe a bit how a data race might manifest itself in safe code (given the changes this RFC proposes), or even better, provide a small code snippet showing the issue? |
|
@tommit The compiler may also need to check and prevent all moves out of the static as well. It might be better to make the static which the function is initializing visible inside the function as a mutable borrow of the static only. Alternatively, make the function return the value for the static instead and then make the compiler smart enough to get rid of the move (but this puts us back at just allowing the expressions to be non-const). I think there is work going on to perform move elision. Another clarification point. The RFC does not appear to say whether or not an initialization function is free to mutate any static. Presumably it's only allowed to mutate the static it is assigned to, but this implies that every static must have a unique function. I missed the bit about disallowing dynamic calls on my first read. I understand why this is necessary, but it makes it potentially hard to know which libraries I can use. If |
|
@camlorn
The meaning of "considered mutable" is clarified in the beginning of the "A possible implementation" chapter. The sentence "Inside the body of an SI function, all of its associated SI statics are made mutable. It is as if the following kind of transformation happened:" was meant to be taken literally, i.e. the compiler could actually implement the feature by performing the proposed code transformation. And to clarify, the proposed code transformation was this: Given an SI function 1: Write the following in the beginning of the body of // anon_x and anon_y represent some unique identifiers
let anon_x: &mut DataX = unsafe { std::mem::transmute(&DATA_X) };
let anon_y: &mut DataY = unsafe { std::mem::transmute(&DATA_Y) };2: Replace all other uses of And the same logic would apply to any number of associated SI statics.
I thought that it's not possible to move out of a static today.
It's not mentioned explicitly in the RFC, but you're right in that an SI function is not free to mutate whatever static it wants to, but rather only those SI statics the said SI function is associated with (or as you said, assigned to). But why would that imply that every SI static must have a unique SI function? The RFC says quite clearly that "Multiple different startup initialized statics may have the same startup initialization function" (better wording would have been: "...statics may be associated with the same..."). |
|
@tommit If a SI function is shared between multiple statics, then all the statics which it initializes need to be declared mutable in the body of the function and the function needs to be called only once for all the statics for which it is used, I think. Is the once-only constraint specified somewhere already? |
|
@camlorn
It's buried in this not so clear a sentence: "At the beginning of the main function, all SI functions of the current crate and all SI functions of all the extern crates it depends upon are called implicitly by the compiler, once each". While writing the RFC, I thought that SI functions would never need to be user-callable, but if that's not the case, I think the compiler should also add some code to all SI functions to protect against the possibility of being called more than once (think |
|
I'd like to propose we move this RFC to final comment period (note that there is a change to the RFC acceptance process here to get more consensus from the team before moving to FCP). Summary: this RFC proposes a new mechanism for initialising statics with code run at startup. This allows some flexibilty in the use of statics, however, it introduces life before main which has always been avoided in Rust and which is rightly terrifying to any C++ programmer. This RFC also proposes adding a somewhat complicated, non-local analysis to ensure safety. A partial alternative is the ongoing work on const functions (which allow initialising statics with functions run at compile-time, rather than startup). Reaction has been mostly negative. I propose moving to FCP with an inclination to close. @rust-lang/lang members, please check off your name to signal agreement. Leave a comment with concerns or objections. Others, please leave comments. Thanks!
|
|
I'd think const functions should cover many use cases much better. There are oddball use cases that say depends upon the runtime environment, like say a SPRNG, or take considerable ram but can be generated faster than loaded from disk, but those commonly benefit from an explicit initialization function. You might wait to initialize some huge array once you launch your thread pool to improve user perception, for example. Also, if a warning to call an initialization routine in the docs does not suffice, then one could maybe add |
|
@burdges Life-before-main is a tricky one. On one hand, it is incredibly useful. On the other hand, it has major disadvantages. I don't this particular proposal is especially elegant in that light. Another (less special case) proposal is a magic macro, say There are other problems which are not addressed by this rfc as well, such as linking with dylibs, or even just linking in general. You cannot rely on static analysis or other fanciness on link time. |
|
An improvement over using Among the many possible uses for the prefix invariant assertion, one could simply check to see if a static was zeroed, meaning the caller had failed to initialize it. If zeroed is a valid value for your static, then simply add another static that should not be zero, maybe one that only exists in debug builds. I'm assuming here that debug and release build binaries only link against debug and release build libraries, respectively, as I have hardly used |
|
FCP proposed with disposition to close. Review requested from: No concerns currently listed. |
|
@nrc I'm slightly confused about going through RFC bot again -- it seemed we were already ready to go into FCP? |
|
All relevant subteam members have reviewed. No concerns remain. |
|
It has been one week since all blocks to the FCP were resolved. |
|
The rfcbot isn't automatically applying the final-comment-period tag yet, but this RFC has been vocally heading toward closure for long enough that I'm going to go ahead and close it out. |
Rendered