Allow appending instances to an existing batch

[kpreid]

Is your feature request related to a problem? Please describe.
I want to visualize progress in an algorithm which is incrementally building an output dataset. This means displaying not just a current state, but what has come before it. Currently, I do this by having my application log the entire dataset each time an element is added, which has the following disadvantages:

• Takes O(n²) time to build, and space in the recording.
• It's more difficult to maintain that copy of the data (in the place that's responsible logging to Rerun) than it would be to fire-and-forget log things.

Describe the solution you'd like
Add the ability to append a batch of component instances to the existing data, rather than replacing them.

Describe alternatives you've considered
It is possible that the “Visible time range” feature would make more sense here instead. I haven't explored its possibilities in detail since there is no Rust API for blueprint configuration yet. However, it seems to me that there should be support for efficiently logging data which is semantically an accumulation rather than only for display purposes. (And, for example, an application might want to use appends but then sometimes clear and rebuild the whole thing due to some change — which does not interact well with the visible time range feature.)

[teh-cmc]

Interesting. That does indeed sound like textbook visible time range to me.

I haven't explored its possibilities in detail since there is no Rust API for blueprint configuration yet

Shouldn't be long now ™️

However, it seems to me that there should be support for efficiently logging data which is semantically an accumulation rather than only for display purposes.

In effect, this is exactly what the "visible time range" does. Internally, there's nothing specifically "visible" about it, it's just a different kind of query that accumulates results across many time indices instead of focusing only on the latest index available.
Every system (rendering and otherwise) is then free to deal with those results as it pleases.

It's really made specifically for this, the naming is just unfortunate here.

(And, for example, an application might want to use appends but then sometimes Clear and rebuild the whole thing due to some change — which does not interact well with the visible time range feature.)

Interestingly, we already support that kind of Clear awareness today, but only for scalars (because we hacked the support directly into the plot view 🙈).

I agree we should have (configurable?) generalized Clear-handling behavior directly at the query layer for range queries. That'd be pretty neat.

[teh-cmc]

Interestingly, we already support that kind of Clear awareness today, but only for scalars (because we hacked the support directly into the plot view 🙈).

Here's what this looks like in practice:

[kpreid]

In effect, this is exactly what the "visible time range" does. …
I agree we should have (configurable?) generalized Clear-handling behavior directly at the query layer for range queries. That'd be pretty neat.

The reason I'm skeptical that "visible time range" will cover all of the use cases that appending and clearing might is: "visible time range" allows expressing "visualize these events strictly cumulatively" but it does not allow expressing “the current state at this instant contains fewer of these entities". For example, if the algorithm I mentioned above has multiple phases, it might be appropriate to completely clear the data from a certain phase at a certain point, and start accumulating new data in the same space. In a 2D time series plot, this is no problem because the horizontal axis allows keeping the two phases obviously separate, but in 3D there is no analogous way to avoid arbitrarily-positioned objects hiding others.

It could work, though, if the viewer could be instructed to respect Clears as "cut off the portion of the range before this; treat this query as if it starts at the clear point".

---

Also, “append” can be seen as a subset of “modify existing instances”, which might be useful for any situation where the process being visualized revisits old data and updates it.

[kpreid]

From chat I heard that I wasn't very clear about the “multiple phases” thing, so let me repost what I said there with sketches. The basic principle is that the application should be able to log stuff and forget it exists, but end up displaying more data over time. A chart of time vs. how much stuff is on the screen looks like this:

t=0 #
t=1 ###
t=2 ######

So far, a “from beginning to now” range query is perfect for that. But if there are multiple stages to the process being visualized,

t=0 #
t=1 ###
t=2 ######
t=3 @@
t=4 @@@@@
t=5 @@@@@@

such that you want the #s and the @s to not ever visually overlap each other, then a range query by itself cannot help any more. But if there was a way to say “for this entity, ranges extending backwards to t=3 should be truncated at that point”, then that would work for this purpose. I imagine that Clear could have an optional flag to have this behavior.

[teh-cmc]

Lots of interesting thoughts in this thread: https://discord.com/channels/1062300748202921994/1075873257124810852/1233429218441035879

[teh-cmc]

We had a long discussion that yielded a lot of ideas across a wide range of query-related matters (see discord thread linked above).

One of the more concrete ideas that came out of this is this issue:

• Configurable Clear-break semantics for range queries #6136