System organization overhaul and the road to a stageless schedule

[alice-i-cecile]

The Context

This post is an attempt to capture conversation between @Ratysz, @TheRawMeatball, @cart and myself around more ergonomic ways to organize systems. Credit for the original seed of this idea belongs to @Ratysz.

The approach discussed here shouldn't be seriously attempted until 0.5 has been released, in order to make sure the fundamentals of #1144 and Cart's ECS storage overhaul are solidified (see further discussion here).

The Problem

As Bevy's needs grow, there's been an increasing demand for ways to elegantly add "metadata" to systems (which describe how and when a system should execute, rather than what it does). As of 0.4, that includes:

1. The stage they should run in
2. Whether they're a startup system or not
3. Whether they have any run criteria attached
4. Whether they're an exclusive ("thread-local" in 0.4 parlance) system or not
5. Whether they are only functional during a certain State or not.

This was initially handled by a proliferation of AppBuilder methods, such as add_startup_system_to_stage, but this has obvious limitations due to the exponential number of combinations as we add metadata. See #1060 for a previous discussion of this problem. I believe @cart has a PR in the works to address this for single systems, by using the builder pattern to specify metadata.

Some of this metadata was also handled on a per-stage basis, as seen with SystemStage::serial.

Explicit system ordering was introduced in #1144, to resolve issues around implicit magical behavior that violate least-surprise and make refactoring code bases very dangerous in non-obvious ways. This added another critical, but relatively complex piece of metadata to our systems: dependencies (which specify that a system must run before)

This increases the severity of this problem, and, as part of those changes, SystemSets were introduced. These allow you to group systems together, then specify metadata that applies to all the systems within that set. This is a solid improvement over having to specify them one-at-a-time, or tying this data only to a Stage (see the issues around StateStage and run_criteria).

It is my view that the following difficulties still remain:

1. There is serious conceptual overlap between stages and system sets, which causes confusion for both users and feature development.
2. Stages have four responsibilities: setting the Schedule, determining when 'hard sync' points should be to allow for Commands to be processed, organizing systems to apply metadata, and ensuring that all of the systems in the previous stage have completed before other systems can proceed. This complex mixture of concerns makes it very hard to reason about when they should be used and what the distinction should be.
3. The 'hard sync' nature of stages means that they come with a significant parallelization cost.
4. Specifying and maintaining complex behavior is painful because each system belongs to at most one system set. If you want behavior that is almost the same, you need to make a completely new system set.
5. SystemSets encourage you to scatter your organizational logic within separate modules / plugins of your code, making it hard to get a big-picture view of how everything fits together.

DIGRESSION: Somewhat orthogonal to these concerns, allowing diverse Schedules, and especially nesting Schedules adds a good deal of complexity to our design (and use cases) without a clear benefit. I have yet to see a compelling use of this, either internally or in user code. There should be a way to replace the scheduler with your own, but this doesn't seem to be the way and IMO can be cut from consideration in terms of how to reimplement existing functionality.

[alice-i-cecile]

The Proposal

1. Remove stages.
2. Create "labelled system sets", which gather and collect information about when and if a system should run in a single place.
3. Add these sets to systems using the builder pattern to specify their behavior. Systems can belong to multiple labelled system sets, and each labelled system set can contain multiple systems.
4. Create a minimal idiom that allows you to add the collection of sets to each system within them.
5. As a best practice: record exactly what each of these system sets do in a single place in the code base (or do so fractally). This avoids the logic fragmentation encouraged by SystemSets + Stages.

The existing functionality of stages would be covered by:

1. Setting the Schedule: just remove this, and set it globally as needed.
2. Specifying hard-sync points: explicitly insert an exclusive system that only processes commands. By default, one of these would exist at the end of each pass of the main game loop ('tick').
3. Organizing systems: obviated by the system labels described above.
4. Ensuring dependencies at a coarse scale: obviated by system labels + explicit dependencies.

The Benefits

1. Much clearer separation of concerns due to the dismantling of stages.
2. No need to maintain parallel APIs for stages and system sets.
3. More visibility and control over hard sync points.
4. Easier to specify complex behavior by updating the definition of a label in one place.
5. More explicit system behavior by removing metadata modification via stages.
6. Possibility of better parallelization due to reduced use of hard sync points.
7. Clearer mental model for users and developers.
8. The scheduler can operate on the basis of an entire pass through the main game loop, rather than having to operate on a per-stage basis.

The Costs

1. This would be a seriously breaking change. This is lots of work for both fixing the DefaultPlugins and for our users.
2. Understanding broad execution order from reading code alone would be harder. This would make a scheduler visualizer even more sorely needed.
3. I don't see a good way to segment off different parts of the schedule without stages, so Schedule specification would likely be lost. Explicit system dependencies probably go a long way to addressing these use cases however.
4. This approach is substantially more expressive, and makes it easy to apply complex requirements to systems. This increases the risk of conflicting requirements, either through directly clashing on a metadata field or through creating circular dependencies.

[Ratysz]

As a best practice: record exactly what each of these system sets do in a single place in the code base (or do so fractally). This avoids the logic fragmentation encouraged by SystemSets + Stages.

I don't understand this one. (There is no "logic fragmentation" encouragement.)

Setting the Schedule: just remove this, and set it globally as needed.

I'm not sure what you mean by that.

Specifying hard-sync points: explicitly insert an exclusive system that only processes commands. By default, one of these would exist at the end of each pass of the main game loop ('tick').

That's not how commands work. Commands are system-owned constructs, only the system that issued the commands can apply them.

Much clearer separation of concerns due to the dismantling of stages.

Separation of concerns implies segregating code and abstractions of unrelated mechanisms; all of the hierarchy has one concern: running systems in correct order. We're not separating anything, we're flattening the hierarchy by removing at least one level of it.

No need to maintain parallel APIs for stages and system sets.

There is no parallel API between the two, they do different things.

More visibility and control over hard sync points.

I'd argue they're plenty visible now - anything before and after a block of parallel systems is essentially synchronous.

The scheduler can operate on the basis of an entire pass through the main game loop, rather than having to operate on a per-stage basis.

I'm assuming you mean the executor. As I've mentioned in the corrections post, that's not exactly how it works. It'll still necessarily have to operate between hard sync points - right now that equates to "per-stage basis".

I don't see a good way to segment off different parts of the schedule without stages, so Schedule specification would likely be lost.

That just means the hierarchy will be even flatter than initially thought. Which is a good thing - more parallelization opportunities.

[alice-i-cecile]

Open Questions

1. Is this proposal an improvement over the existing patterns (once System sets and parallel executor v2 #1144 lands)?
2. What do we want this idiom to be called? The original term was "multiple labels", but I think "overlapping sets" might be a bit clearer? More discussion here.
3. What unforeseen complications would this proposal have?
4. How do the details of multiple labels + ordering dependencies work?
5. Should you be able to nest SystemSets to recursively apply labels?
6. Should you be able to remove labels to work better with SystemSets?
7. What exactly should the user-facing syntax look like?
8. Should you be allowed to specify system metadata without using a label at all?
9. Does this solve the issues around ergonomically specifying system dependencies?
10. Should SystemSets be replaced under this proposal with more standard functional idioms like map? (Relates to 5).
11. What exactly is "system metadata" in this sense, and where do the boundaries lie?
12. On a conceptual level, this pattern works very similarly to both Rust's traits and marker components. Can we use either for implementing this design?
13. Should we be rethinking how Commands works to enable 'soft sync' points?
14. Do we need better circular dependency tools to handle to go along with the increased expressivity?
15. Can / should we integrate this idea with "resources as config" mentioned here?

[tigregalis]

It might be worth calling out that a label need not be unique, i.e. some label e.g. "render" will be applied to multiple systems. This isn't explicit in your excellent write-up above. At least, I assume this is the case, otherwise the utility of multiple labels would be extremely limited.

A label per current bevy stage would just about serve as a drop-in replacement for stages as they are now. And for the default plugins, if Commands isn't used in a stage, we can omit that hard sync point. From what I can tell, there's only one place that use Commands: the parent_update_system, which is registered in POST_STARTUP and POST_UPDATE. However, there are many exclusive systems, but perhaps these can be rewritten (if appropriate).

Specifying hard-sync points: explicitly insert an exclusive system that only processes commands.

I think this is critical. However, I wonder if it would be possible for this not to be exclusive, e.g. allowing other systems to move forward, if the commands don't touch the same resources / archetypes, allowing for increased parallelism. That would be impossible to know at compile time or app build time however, with the current API.

DIGRESSION: Somewhat orthogonal to these concerns, allowing diverse Schedules, and especially nesting Schedules adds a good deal of complexity to our design (and use cases) without a clear benefit.
What is the alternative to nesting of schedules for a fixed timestep scenario*, or any kind of app with multiple "timelines" e.g. 1 render tick (the main loop) =/= 1 game update tick (the physics update loop)? Ideally these should run in parallel if the systems they contain do not conflict.

As has previously been mentioned, the schedule is essentially a directed acyclic graph. Having nested schedules/stages/sets is like boxing up a number of systems and treating it as one system, and making the implementation details opaque to you: the main loop just sees another system, and its predecessors and its dependents. Ideally the schedule would be "aware" of which resources/archetypes it touches (i.e. treating a "schedule" as being the sum of its systems), so that systems in the main schedule or another schedule can run if appropriate (Commands might limit this however).

On graphs, another thing to consider is that these labels may make circular dependency errors easier to make:
(#abc represents a label applied to a system (or set of systems), <abc represents a dependency)

system_a#label_a<label_b
system_b#label_b<label_a

How will this be handled?

[alice-i-cecile]

It might be worth calling out that a label need not be unique, i.e. some label e.g. "render" will be applied to multiple systems.

This is correct; let me make it clearer.

However, I wonder if it would be possible for this not to be exclusive, e.g. allowing other systems to move forward, if the commands don't touch the same resources / archetypes, allowing for increased parallelism. That would be impossible to know at compile time or app build time however, with the current API.

This idea of "soft syncs" is really interesting to me, but would require some rethinking of how commands work. I'm particularly interested in live insertion of marker components, but we'll see how @cart's sparse components feature lands.

another thing to consider is that these labels may make circular dependency errors easier to make

I agree with this. We're going to need better tools for detecting and untangling this sort of error. Being able to put all your metadata in one place in a natural fashion would help, but may not be sufficient.

@tigregalis I've updated The Costs and Open Questions to reflect your comments :)

[Ratysz]

Is this proposal an improvement over the existing patterns (once #1144 lands)?

Yes. It will flatten the hierarchy, which makes the API simpler and removes some obvious pitfalls (like overreliance on stages), and it will enable more flexible and ergonomic graph definition APIs.

What do we want this idiom to be called? The original term was "multiple labels", but I think "overlapping sets" might be a bit clearer? More discussion here.

I stand by my opinion: in comparison with current way, "multiple labels" or a synonym is distinct enough. After this is fully implemented, just "labels" is fine, because that's what it will look like in the external API.

What unforeseen complications would this proposal have?

How do the details of multiple labels + ordering dependencies work?

Should you be able to nest SystemSets to recursively apply labels?

Should you be able to remove labels to work better with SystemSets?

What exactly should the user-facing syntax look like?

Should you be allowed to specify system metadata without using a label at all?

Does this solve the issues around ergonomically specifying system dependencies?

These are impossible to answer before at least starting implementation.

Should SystemSets be replaced under this proposal with more standard functional idioms like map? (Relates to 5).

What does map mean in this context?

What exactly is "system metadata" in this sense, and where do the boundaries lie?

I don't understand the question. It's an implementation detail, a design proposal doesn't need to concern itself with that.

On a conceptual level, this pattern works very similarly to both Rust's traits and marker components. Can we use either for implementing this design?

What are "Rust's marker components"?

No, we can't use traits: those are static language constructs that don't exist once the program is compiled. We need our systems' graphs to be dynamic.

Should we be rethinking how Commands works to enable 'soft sync' points?

Commands have to require hard sync points. They can modify archetypes and insert/remove resources.

Do we need better circular dependency tools to handle to go along with the increased expressivity?

#1144 forbids circular dependencies, because a directed graph with a cycle cannot be ordered topologically and therefore is not executable. Unless you meant something else?

Can / should we integrate this idea with "resources as config" mentioned here?

I'm not sure what you mean. Use config resources to describe labels/relationships/graphs? Sounds like needless API nesting.

---

And for the default plugins, if Commands isn't used in a stage, we can omit that hard sync point. [...] However, I wonder if it would be possible for this not to be exclusive, e.g. allowing other systems to move forward, if the commands don't touch the same resources / archetypes, allowing for increased parallelism.

That's not how commands work. Each parallel system writes to its own commands buffer, and only that system may apply those commands to the world/resources. Commands may modify archtypes and insert/remove resources, so they must be applied during a hard sync point. That's the whole point of their existence: providing parallel systems a delayed way to perform necessarily synchronous operations, in an automated (from the user perspective) way.

[...] Ideally the schedule would be "aware" of which resources/archetypes it touches [...]

All nestable hierarchy blocks are synchronous.

[...] circular dependency errors [...] How will this be handled?

Right now this is a hard crash. A graph with a cycle is not executable - an irrecoverable error. When we have runtime graph editing (via the visual editor), the methods that alter the graph will be fallible, i.e., they will return a result with an appropriate error, so that it can be shown to the user without crashing.

---

[...] but would require some rethinking of how commands work.

Again, commands have to be hard sync points - I don't think sparse sets will let us fudge this in all cases.

We're going to need better tools for detecting and untangling this [circular dependency] sort of error.

We already do. Take a look at what the cyclic dependency tests in #1144 output to the console.

[Ratysz]

I'm sorry, too many concepts are misrepresented here. If I correct the mistakes line-by-line the whole thing stops making sense, so I'll just dump thoughts as they come.

Systems being exclusive or not is determined solely and automatically from their arguments.

AppBuilder method proliferation is a design choice - we don't have to do it. #1144 did not add any new methods to it. #1060 is implemented in #1144. Cart's current work has nothing to do with any of that, as far as I know.

The only thing system sets do is allow attaching run criteria to one or more systems within the same stage. Literally nothing else.

There is no overlap between stages and system sets - the former owns the latter. Schedule is not set by stages - it owns them. System sets don't encourage you to scatter systems - they own them.

Schedule > stage > set > system. A schedule can be coerced to be a stage. A stage can be stuffed into an exclusive system and be a part of a set. This is not overlap: this is putting things inside other things.

There is no "scheduler" - there is an order in which a schedule is executed, and there is no circumstance under which it would make sense to change it. There is no point to execute stages in an order different from the one and only order you've defined for them when building the schedule, that's the whole point of the abstraction: static order between big building blocks of the hierarchy.

The executor is an implementation detail of specifically SystemStage: it's a scheduling algorithm (I know, the terms are confusingly close, it's bad) that runs the parallel systems of that stage, that's it. We have two: parallel and single-threaded; the latter does the same thing as the former but uses no more than one thread to do so.

---

I think the discussion name should have "schedule" and not "scheduler".

[TheRawMeatball]

After much discussion on discord, I have an incomplete, but workable API proposal (also relevant to #1312):

App::build()
  // Stages replaced with a "command set" to reduce confusion: A command set is a system set
  // which groups systems whose commands get executed together
  // Other than setting hard sync points, they're mostly just ordering systems with a similar API
  .add_command_set("stage 1")
    // This must be a command set. (probably)
    // TODO: Clear up the semantics of how to handle multiple command sets specifying `.before("x")`
    // TODO: Consider the implications of letting people use regular ordering sets here
    .before("startup")
  // if a command set isn't explicitly stated, it defaults to the update stage
  // TODO: Consider trying to infer a better alternative if one exists. 
  .add_system(example_system)
  // Systems in this set will not trigger the ambiguity detector for being
  // ambiguous on the specified components. In release builds without logging,
  // this is a no-op.
  .add_ambiguous_set("ambiguous set")
    .for_component::<Component1>()
    .for_resource::<Resource1>()
  // All systems who are part of the set must be in the same command set.
  // The run condition gets evaluated once at the start of the stage, and then those who need to
  // be checked again run at the end, right before commands.
  // TODO: Maybe allow a `parent` conditional set
  .add_conditional_set("conditional set", condition_system)
  // Ordering sets are used to specify various order trees using `.before` and `.after`
  // Multiple, non-interfering set graphs can exist, and they can be as simple as a single node.
  // When a system is part of a ordering set, it must run during the execution of said set.
  .add_ordering_set("set 1")
  .add_ordering_set("set 2").after("set 1")
  .add_ordering_set("set 3").after("set 2")
  .add_system(
    // Inferring the type of system from the given key is done internally.
    // TODO: define the specifics of being part of multiple conditional sets (or remove it).
    system_1.in_set("conditional set")
      // .before and .after calls can only be made to ordering sets.
      // The other types, including command sets, should be handled directly using `in_set`
      // These should be equivalent to creating a ordering set with a single node,
      // setting the given constraining and calling `in_set()`
      .before("set 2")
      .in_set("different set")
  )

[tigregalis]

Does there need to be so many methods?

Since they are exclusive anyway, can we simply specify the type of the set as a variant of an enum?

App::build()
    .add_set("set 1", Set::Ordering)
    .add_set("conditional set", Set::Conditional(conditional_system));

[TheRawMeatball]

Absolutely not! This is way cleaner.

[Ratysz]

Here is one of my variations on the idea. This assumes some form of #1423 has landed.

Global rule: everything tends towards the start of the tick. An .after() is enough to make sure something runs between specific sync points - a matching .before() is not required; things without an .after() somewhere are assumed to be before any defined sync points.

There are multiple ways to define most relationships, some work best for fine grained definitions, some enable mass assignments; ideally, redundant constraints produce an informative warning. In this specific version, "ambiguity sets" are not type-wise disambiguated with "ordering sets" - the burden of that lies on the user.

#[derive(SystemLabel)]
struct Label(&'static str);
#[derive(SetLabel)]
struct Set(&'static str);
// "Conditions" is a snappier synonym for "run criteria". Since this is a chance to rename things.
#[derive(ConditionLabel)]
enum Condition {
    SixtyTimesPerSecond,
    EveryOtherTime,
}
// No point dancing around the real name of the concept.
#[derive(SyncPointLabel)]
enum SyncPoint {
    LogicDone,
    DrawingDone,
}
App::build()
    .add_system(physics.system().label(Label("physics")))
    .add_set(Set("commutative transform").commutative_over_component::<Transform>())
    .add_set(Set("nudge")
        // This can be a label of a set, a parallel system, or a sync point.
        .before(Label("physics"))
    )
    .add_system(
        nudge_up
            .system()
            // This too can be a label of a set, a parallel system, or a sync point.
            .before(Label("physics"))
            .in_set(Set("commutative transform")),
    )
    .add_systems(
        // All this does is apply the same properties to all systems within. It's a shorthand.
        Systems::new()
            .in_set(Set("commutative transform"))
            .in_set(Set("nudge"))
            .with_system(nudge_left.system().label(Label("nudge left")))
            .with_system(nudge_right.system().after(Label("nudge left"))),
    )
    .add_system(
        print_transform_after_nudging
            .system()
            .after(Set("nudge")),
    )
    .add_sync_point(SyncPoint::LogicDone)
    // This is `System<In = (), Out = ShouldRun>`.
    .add_condition(
        Condition::SixtyTimesPerSecond,
        FixedTimestep::steps_per_second(60.0),
    )
    .add_condition(
        Condition::EveryOtherTime,
        // This is `System<In = ShouldRun, Out = ShouldRun>`. Closure because wishful thinking.
        Condition::SixtyTimesPerSecond.then(
            |tps60: In<ShouldRun>, ran_last_time: Local<bool>| {
                match tps60.0 {
                    YesAndCheckAgain => {
                        *ran_last_time = !*ran_last_time;
                        // Returning `Yes` or `No` here will not evaluate this condition again this tick.
                        // `*AndCheckAgain` makes sure this is re-evaluated when `SixtyTimesPerSecond` is.
                        if *ran_last_time {
                            YesAndCheckAgain
                        } else {
                            NoAndCheckAgain
                        }
                    }
                    // Fixed timestep can only return `YesAndCheckAgain` or `No`.
                    // If "parent" condition evaluates to `No`, "child" conditions aren't
                    // evaluated, so anything other than `YesAndCheckAgain` is unreachable.
                    _ => unreachable!(),
                }
            },
        ),
    )
    .add_system(
        draw_stuff
            // Exclusive systems always run after the sync point and before parallel systems.
            .exclusive_system()
            // This can be a label of a sync point or an exclusive system (not shown; likely should be a separate trait).
            .after(SyncPoint::LogicDone)
            .with_condition(Condition::SixtyTimesPerSecond),
    )
    .add_system(
        flicker_idk
            .system()
            .after(SyncPoint::LogicDone)
            .with_condition(Condition::EveryOtherTime),
    )
    // Sync points can have relations with other sync points.
    .add_sync_point(SyncPoint::DrawingDone.after(SyncPoint::LogicDone))
    .add_system(log_stuff.exclusive_system().after(SyncPoint::DrawingDone))

[alice-i-cecile]

Automatic Labelling

I think it would be very useful to automatically apply labels to systems based on their type signatures (or similar information). This would allow us to apply labels to a large number of systems in a consistent, foolproof way using logic that's easily understood, and encode important rules into our schedule without requiring new primitives.

As a concrete application, rather than having to special case safety features like #1439, we can use labels to do the lifting in a customizable way.

In this case, define a label that is automatically applied to all systems with Transform, and create a rule that all such systems must run before our transformation propagation system. Do the same for GlobalTransform, and state that all systems with this second label must run after the transform propagation system.

[alice-i-cecile]

Very preliminary outline of a mad science idea to fully eliminate hard sync points, from Discord:

The basic outline goes something like this:

1. Commands which mutate the same data are stored together, like an enhanced version of Cart's ECS PR
2. Commands are processed one-type-at-a-time in a just-in-time fashion: automatically inserting a system that takes write-access to all of the data that they mutate immediately before the next system that reads any of the data that they would modify
3. We take advantage of component and resource registeration plus the dynamic system cosntruction

• insertion tech to replace anything left that still needs access to everything in the world

4. We remove exclusive systems and stages, eliminating all hard sync points, except at the end of the game loop.
5. At the end of the game loop, process any remaining outstanding commands then in parallel by creating a bunch of command-handling systems

I fully expect there to be some serious feasibility questions around it, but it would:
a) remove all hard sync points
b) simplify the conceptual model of how to use Bevy apps by making commands always take place instantly*
c) allow commands to be efficiently batch-processed
d) allow different types of commands to be safely processed in parallel
e) open the door to event-triggered systems that work in exactly the same way
f) give us the tech needed to do automatic index updating

*not actually instantly, but there would be no way to accidentally run your system before commands resolve

[alice-i-cecile]

Scripting is one of the most important and complex uses of exclusive systems. As an idea, you could give the scripting engine unlimited read access to World, have it emit arbitrary commands, and then hand off the mutation rights after you know what work needs doing.

As a potential compromise, exclusives systems could still exist, but can only run at the beginning or end of the game loop.

[cart]

Related discussion in #1512

[Ratysz]

Bears mentioning that there is an ECS project board card with a rough roadmap.