[Proposal] clay_interactions.h

[monodop]

There's been a bunch of discussions in the discord server, and in the github issues (#230, #224, #198, #197, #154, #67) about moving mouse/scroll interactions out of clay.h and leave it for the user space. While I do personally think that clay should come out of the box with support for mouse/keyboard/scroll inputs (as well as accessibility features like tab indexes, keyboard navigation, etc), I wanted to start a thread about how an input system could be developed on top of clay.h. I'll be referring to a theoretical header called clay_interactions.h, which could be bundled in the clay repo. But even if not, I think this is still a useful exercise, as anyone trying to implement this will still have similar requirements.

Note: to be clear, this is mostly a brain dump containing most of my thoughts on this over the last couple weeks, and I'm really interested to hear others' opinions.

Things that I'm intentionally leaving out of scope here:

• Text inputs / text layout queries / etc. While this is definitely related to user io functionality, I think we should probably have a solution for rich text and/or masonry layout first.

State Management

One of the first problems that needs to be solved, is how could such a library store its state? For example:

• For clicks, which elements has the click started in
• For hover, which elements is the mouse entering / leaving
• How long has it been since a click? (e.g. double click)

The library can of course have its own memory storage, like having its own arena. However, this means that the consumer would need to allocate multiple blocks of memory and track them separately, and take extra caution not to mix & match contexts with different interaction memory.

Proposal for clay.h

• Allow the ability to somehow extend Clay_Context to include additional third party data, preferably in a generic/extensible way. This would allow third party libraries to keep their data co-located with the clay context, and could be re-allocated at the same time as the rest of the memory, e.g. if max element count needs to be increased.
• (maybe?) Allow the ability to extend the element hash map entry (different from custom element configs), allowing libraries to store additional element state and have it get cleaned up automatically.

Querying Layout Information

There's a lot of tasks that an interactions library might need to perform based on the final layout data from the previous frame:

• Get an ordered list of elements at an arbitrary point or set of points. Example: click/hover; drag & drop; local multiplayer game with multiple cursors.
• Find elements that are adjacent to a given element. Example: use arrow keys to navigate an inventory; find the next text element to continue text selection into
• Get the bounding box of an element. Example: determine height of scroll bar for a partially obscured element; get target information for an animation
• Determine if an element is currently visible on the current viewport (not obscured or clipped due to a scroll view). Example: load more content dynamically as the user scrolls through a list; pause animations or video that is scrolled out of view

Proposal for clay.h

clay.h doesn't need to directly solve these problems for us, but it should provide some tools that could be used to implement solutions for them:

• Generic way to query under a point that can be called multiple times in a row for different points. [Core] Clay_GetElementIdsAtPoint #241
• Ability to get an element's parent, children, and siblings
• Get the bounding box for an element [Core] Add API to query element bounding boxes #199
• Get the visible bounding box for an element - if we could find a performant way to compute this, then you could have elements self-test their own visibility without needing to be aware of parent scroll views

Scroll Views

Scroll views are already implemented in clay.h, but if we want to move them to user space, we'll need a more generic way to render them.

Proposal for clay.h

• Add an equivalent to CSS overflow: hidden for containers, to allow children to exceed the parent bounding box and clip contents
• Add an offset/translation for content. could be negative margins, relative positioning, or some other concept that can be used to shift the content. Ideally, this should update the resulting bounding boxes as well.

[nicbarker]

Thanks for opening this, for external readers this is definitely the direction that we're going to go long term.
I will take some time to write up my thoughts properly.

[dekdevy]

I really like Clay for its current simplicity and limited scope. This seems in line with the approach of choosing C in the first place. For what its worth, i'd definitely prefer having a layout-only clay core with a rich set of examples to provide further ideas on handling input (the raylib examples work well).

A clay_interactions.h file may quickly run into issues of not being exhaustive enough to cover every possible concern people may ask for; theres some value in explicitly not being super opinionated there.

Do see value in one thing: A query to get all elements at a specific X/Y seems a bit more general-purpose over the current in-house solutions around detecting hovered elements - sorry if i'm missing something there that already exists

[nicbarker]

@dekdevy thanks for the input, just re: your API suggestion, @monodop has done some initial work here already:

#241

[monodop]

@dekdevy

A clay_interactions.h file may quickly run into issues of not being exhaustive enough to cover every possible concern people may ask for; theres some value in explicitly not being super opinionated there.

Yeah, that's basically the inspiration for this proposal in the first place. clay.h will have an incredibly difficult time keeping up with demands in this area. If we have the capability to make something like clay_interactions.h, then theoretically it should be possible for someone to make their own or fork it to meet whatever their requirements are. At the moment, this is not possible, because the necessary APIs are not available.

All that said, I think clay should probably have a recommended solution that covers the 80-90% of use cases to make it easier to get started.