Codegen'd Rust/Arrow (de)ser 5: doc, definitions and regression tests…

… for combinatorial affixes (#2546)

**Best reviewed on a commit-by-commit basis; in particular the `rerun
codegen` commit is nothing but generated code.**

This PR documents what we call "affixes", why they're problematic when
contributing to the codegen stack and why we need to have regression
tests in place for them.

Quoting the newly added doc:
> ### Understanding the subtleties of affixes
> 
> So-called "affixes" are effects applied to objects defined with the
Rerun IDL and that affect
> the way these objects behave and interoperate with each other (so,
yes, monads. shhh.).
> 
> There are 3 distinct and very common affixes used when working with
Rerun's IDL: transparency,
> nullability and plurality.
> 
> Broadly, we can describe these affixes as follows:
> - Transparency allows for bypassing a single layer of typing (e.g. to
"extract" a field out of
>   a struct).
> - Nullability specifies whether a piece of data is allowed to be left
unspecified at runtime.
> - Plurality specifies whether a piece of data is actually a collection
of that same type.
> 
> We say "broadly" here because the way these affixes ultimately affect
objects in practice will
> actually depend on the kind of object that they are applied to, of
which there are 3: archetypes,
> components and datatypes.
> 
> Not only that, but objects defined in Rerun's IDL are materialized
into 3 distinct environments:
> IDL definitions, Arrow datatypes and native code (e.g. Rust & Python).
> 
> These environment have vastly different characteristics, quirks,
pitfalls and limitations,
> which once again lead to these affixes having different, sometimes
surprising behavior
> depending on the environment we're interested in.
> Also keep in mind that Flatbuffers and native code are generally
designed around arrays of
> structures, while Arrow is all about structures of arrays!
> 
> All in all, these interactions between affixes, object kinds and
environments lead to a
> combinatorial explosion of edge cases that can be very confusing when
it comes to (de)serialization
> code, and even API design.
> 
> When in doubt, check out the `rerun.testing.archetypes.AffixFuzzer`
IDL definitions, generated code and
> test suites for definitive answers.

At the moment, the test suite merely checks that the native objects that
are generated make sense.
Serialization roundtrips will be added in a couple PRs.
It also only focuses on the Rust backend, as I'm still unsure what
complex, deeply nested objects are gonna look like in the Python backend
(need to work on #2390 first to get a feel for it).

Also note that, for now, test-archetypes are mixed in with all other
user-facing archetypes, which is not ideal.
We'll work towards something better as the dust settle, as for now the
overall organization of things is still a moving target (#2537).

---

- #2484
- #2485 
- #2487 
- #2545
- #2546
- #2549
- #2554
- #2570
- #2571

---

<!-- This line will get updated when the PR build summary job finishes.
-->
PR Build Summary: https://build.rerun.io/pr/2487

<!-- pr-link-docs:start -->
Docs preview: https://rerun.io/preview/46068dd/docs
Examples preview: https://rerun.io/preview/46068dd/examples
<!-- pr-link-docs:end -->

crates/re_types/definitions/rerun/archetypes.fbs

@@ -1,3 +1,5 @@
 include "./archetypes/points2d.fbs";
 
+include "./testing/archetypes/fuzzy.fbs";
+
 namespace rerun.archetypes;

crates/re_types/definitions/rerun/testing/archetypes/fuzzy.fbs

@@ -0,0 +1,50 @@
+include "fbs/attributes.fbs";
+
+include "rerun/testing/components/fuzzy.fbs";
+
+namespace rerun.testing.archetypes;
+
+// ---
+
+// TODO(cmc): we actually have an opportunity to resolve the nullability situation here if we make
+// component_required vs. required do different things:
+// - component_required affects the nullability of the component itself at the archetype level (as it does today)
+// - required affects the nullability of the element values for the underlying array (both native and arrow)
+
+table AffixFuzzer1 (
+  "attr.rust.derive": "Debug, Clone, PartialEq",
+  order: 100
+) {
+  fuzz1001: rerun.testing.components.AffixFuzzer1 ("attr.rerun.component_required", required, order: 1001);
+  fuzz1002: rerun.testing.components.AffixFuzzer2 ("attr.rerun.component_required", required, order: 1002);
+  fuzz1003: rerun.testing.components.AffixFuzzer3 ("attr.rerun.component_required", required, order: 1003);
+  fuzz1004: rerun.testing.components.AffixFuzzer4 ("attr.rerun.component_required", required, order: 1004);
+  fuzz1005: rerun.testing.components.AffixFuzzer5 ("attr.rerun.component_required", required, order: 1005);
+  fuzz1006: rerun.testing.components.AffixFuzzer6 ("attr.rerun.component_required", required, order: 1006);
+  fuzz1007: rerun.testing.components.AffixFuzzer7 ("attr.rerun.component_required", required, order: 1007);
+
+  fuzz1101: [rerun.testing.components.AffixFuzzer1] ("attr.rerun.component_required", required, order: 1101);
+  fuzz1102: [rerun.testing.components.AffixFuzzer2] ("attr.rerun.component_required", required, order: 1102);
+  fuzz1103: [rerun.testing.components.AffixFuzzer3] ("attr.rerun.component_required", required, order: 1103);
+  fuzz1104: [rerun.testing.components.AffixFuzzer4] ("attr.rerun.component_required", required, order: 1104);
+  fuzz1105: [rerun.testing.components.AffixFuzzer5] ("attr.rerun.component_required", required, order: 1105);
+  fuzz1106: [rerun.testing.components.AffixFuzzer6] ("attr.rerun.component_required", required, order: 1106);
+  fuzz1107: [rerun.testing.components.AffixFuzzer7] ("attr.rerun.component_required", required, order: 1107);
+
+  fuzz2001: rerun.testing.components.AffixFuzzer1 ("attr.rerun.component_optional", order: 2001);
+  fuzz2002: rerun.testing.components.AffixFuzzer2 ("attr.rerun.component_optional", order: 2002);
+  fuzz2003: rerun.testing.components.AffixFuzzer3 ("attr.rerun.component_optional", order: 2003);
+  fuzz2004: rerun.testing.components.AffixFuzzer4 ("attr.rerun.component_optional", order: 2004);
+  fuzz2005: rerun.testing.components.AffixFuzzer5 ("attr.rerun.component_optional", order: 2005);
+  fuzz2006: rerun.testing.components.AffixFuzzer6 ("attr.rerun.component_optional", order: 2006);
+  fuzz2007: rerun.testing.components.AffixFuzzer7 ("attr.rerun.component_optional", order: 2007);
+
+  fuzz2101: [rerun.testing.components.AffixFuzzer1] ("attr.rerun.component_optional", order: 2101);
+  fuzz2102: [rerun.testing.components.AffixFuzzer2] ("attr.rerun.component_optional", order: 2102);
+  fuzz2103: [rerun.testing.components.AffixFuzzer3] ("attr.rerun.component_optional", order: 2103);
+  fuzz2104: [rerun.testing.components.AffixFuzzer4] ("attr.rerun.component_optional", order: 2104);
+  fuzz2105: [rerun.testing.components.AffixFuzzer5] ("attr.rerun.component_optional", order: 2105);
+  fuzz2106: [rerun.testing.components.AffixFuzzer6] ("attr.rerun.component_optional", order: 2106);
+  fuzz2107: [rerun.testing.components.AffixFuzzer7] ("attr.rerun.component_optional", order: 2107);
+}
+

crates/re_types/definitions/rerun/testing/components/fuzzy.fbs

@@ -0,0 +1,72 @@
+include "arrow/attributes.fbs";
+include "fbs/attributes.fbs";
+include "rust/attributes.fbs";
+
+include "rerun/testing/datatypes/fuzzy.fbs";
+
+namespace rerun.testing.components;
+
+// ---
+
+table AffixFuzzer1 (
+  "attr.arrow.transparent",
+  "attr.rust.derive": "Debug, Clone, PartialEq",
+  order: 100
+) {
+  single_required: rerun.testing.datatypes.AffixFuzzer1 (required);
+}
+
+table AffixFuzzer2 (
+  "attr.arrow.transparent",
+  "attr.rust.derive": "Debug, Clone, PartialEq",
+  "attr.rust.tuple_struct",
+  order: 200
+) {
+  single_required: rerun.testing.datatypes.AffixFuzzer1 (required);
+}
+
+table AffixFuzzer3 (
+  "attr.rust.derive": "Debug, Clone, PartialEq",
+  order: 300
+) {
+  single_required: rerun.testing.datatypes.AffixFuzzer1 (required);
+}
+
+table AffixFuzzer4 (
+  "attr.arrow.transparent",
+  "attr.rust.derive": "Debug, Clone, PartialEq",
+  order: 400
+) {
+  single_optional: rerun.testing.datatypes.AffixFuzzer1;
+}
+
+table AffixFuzzer5 (
+  "attr.arrow.transparent",
+  "attr.rust.derive": "Debug, Clone, PartialEq",
+  "attr.rust.tuple_struct",
+  order: 500
+) {
+  single_optional: rerun.testing.datatypes.AffixFuzzer1;
+}
+
+table AffixFuzzer6 (
+  "attr.rust.derive": "Debug, Clone, PartialEq",
+  order: 600
+) {
+  single_optional: rerun.testing.datatypes.AffixFuzzer1;
+}
+
+table AffixFuzzer7 (
+  "attr.rust.derive": "Debug, Clone, PartialEq",
+  order: 700
+) {
+  many_optional: [rerun.testing.datatypes.AffixFuzzer1] (order: 100);
+  single_float_optional: float (order: 101);
+  single_string_required: string (order: 102, required);
+  single_string_optional: string (order: 103);
+  many_floats_optional: [float] (order: 104);
+  many_strings_required: [string] (order: 105, required);
+  many_strings_optional: [string] (order: 106);
+  // TODO(cmc): the ugly bug we need to take care of at some point
+  // many_transparent_optionals: rerun.testing.datatypes.AffixFuzzer2 (order: 107);
+}

crates/re_types/definitions/rerun/testing/datatypes/fuzzy.fbs

@@ -0,0 +1,28 @@
+include "arrow/attributes.fbs";
+include "fbs/attributes.fbs";
+include "rust/attributes.fbs";
+
+namespace rerun.testing.datatypes;
+
+// ---
+
+table AffixFuzzer1 (
+  "attr.rust.derive": "Debug, Clone, PartialEq",
+  order: 100
+) {
+  single_float_optional: float (order: 101);
+  single_string_required: string (order: 102, required);
+  single_string_optional: string (order: 103);
+  many_floats_optional: [float] (order: 104);
+  many_strings_required: [string] (order: 105, required);
+  many_strings_optional: [string] (order: 106);
+}
+
+table AffixFuzzer2 (
+  "attr.arrow.transparent",
+  "attr.rust.derive": "Debug, Clone, PartialEq",
+  "attr.rust.tuple_struct",
+  order: 200
+) {
+  single_float_optional: float (order: 101);
+}

crates/re_types/source_hash.txt

@@ -1,4 +1,4 @@
 # This is a sha256 hash for all direct and indirect dependencies of this crate's build script.
 # It can be safely removed at anytime to force the build script to run again.
 # Check out build.rs to see how it's computed.
-ffa869371b5fc08da6c80ce53a84db674682b38152c458d958649decc8949f35
+32b9eccc275777c29b252374ea6dcd69a3f1a48c42ae79aa5138ec2556526b52