Pen/Stylus support #3810

daxpedda · 2024-07-22T10:13:03Z

Waiting for #3833.

This implements Pen/Stylus support inspired by the Web Pointer API.

Instead of adding new events I decided to add them to CursorMoved and MouseInput (now renamed to CursorInput.
This does preclude any larger changes in the spirit of Richer Touch events / unified pointer event model #336, but I'm happy to include some changes in this PR.
Despite the Web API not giving erasers their own device type, but simply a button, I decided to add a new device type instead. See Stylus eraser: should it be a new pointerType instead of a button state? w3c/pointerevents#134 for why this was considered a mistake to begin with.
X11 was emitting pen input in DeviceEvent::MouseWheel, DeviceEvent::MouseMotion, WindowEvent::CursorMoved, WindowEvent::MouseInput, which I disabled, because users would think all these inputs are mouse. This is also in line with all the other backends, X11 and iOS were the only ones emitting any pen events.
iOS was emitting Force::Calibrated::altitude_angle when a pen is used. I removed this and disabled pen events as well, as this information is now part of the ToolState, which can be used with Force::normalized() to get the old behavior back.

Open questions:

Rename "Tool" to "Stylus": Pen/Stylus support #3810 (comment).
Rename "Cursor" to "Pointer": Pen/Stylus support #3810 (comment).
Change DeviceEvent::MouseMotion to DeviceEvent::CursorMotion?
Implement X11 before merging? (I'm happy to do it)

Follow-up:

More compatibility issues with browser: Browser support tracking #2875.
- Linux Firefox has no pen support (Bugzilla).
Richer Touch events / unified pointer event model #336
Gesture API? This might be able to include things like feat(#3759): Implements Apple Pencil double tap functionality #3768 and all the Apple specific gestures.
Implement more backends.
X11 improvements/platform consistency (maybe after Browser support tracking #2875 and a touch overhaul)
- Don't move mouse with touch (?): Pen/Stylus support #3810 (comment).
- Don't move mouse on focus (?): Pen/Stylus support #3810 (comment).
- Better device detection: Pen/Stylus support #3810 (comment).
- Use correct device ID (?): Pen/Stylus support #3810 (comment).

Public API changes

pub enum WindowEvent {
    CursorMoved {
        device_id: DeviceId,
        position: PhysicalPosition<f64>,
        // new field
        r#type: CursorType,
    },
    // renamed from `MouseInput`
    CursorInput {
        device_id: DeviceId,
        state: ElementState,
        // changed type from `MouseButton`
        button: CursorButton
    },
    ...
}

pub enum Force {
    Calibrated {
        force: f64,
        max_possible_force: f64,
        // removed field
        // altitude_angle: Option<f64>,
    },
    Normalized(f64),
}

impl Force {
    // takes new `angle` parameter
    // now always emits normalized force, only emits perpendicular force if `angle` is passed
    pub fn normalized(&self, angle: Option<ToolAngle>) -> f64 { ... }
}

// new types from here on

pub enum CursorType {
    Mouse,
    Pen(ToolState),
    Eraser(ToolState),
}

pub struct ToolState {
    pub force: Force,
    pub tangential_force: Option<f32>,
    pub twist: Option<u16>,
    pub tilt: Option<ToolTilt>,
    pub angle: Option<ToolAngle>,
}

impl ToolState {
    pub fn tilt(self) -> Option<ToolTilt> { ... }
    pub fn angle(self) -> Option<ToolAngle> { ... }
}

pub struct ToolTilt { pub x: i8, pub y: i8 }

impl ToolTilt {
    pub fn angle(self) -> ToolAngle { ... }
}

pub struct ToolAngle { pub altitude: f64, pub azimuth: f64 }

impl Default for ToolAngle { ... }

impl ToolAngle {
    pub fn tilt(self) -> ToolTilt { ... }
}

pub enum CursorButton {
    Mouse(MouseButton),
    Pen(ToolButton),
    Eraser(ToolButton),
}

impl From<MouseButton> for CursorButton { ... }

pub enum ToolButton {
    Contact,
    Barrel,
    Other(u16),
}

impl From<ToolButton> for MouseButton { ... }

Fixes #99.

daxpedda · 2024-07-22T10:16:47Z

src/event.rs

+
+#[derive(Clone, Copy, Debug, PartialEq)]
+#[cfg_attr(feature = "serde", derive(Deserialize, Serialize))]
+pub struct ToolState {


I would be glad for an alternative to the term "Tool" or "State".
Originally got the idea from Linux, because we need a term that applies to all sorts of different tools.
But honestly I think "Pen" would have been fine as well?

Maybe "Stylus" would fit all the possible tools:

Pen

Eraser

Brush

Pencil

Airbrush

src/event.rs

This aligns with the future Std type name and allows us to introduce `LazyCell`.

daxpedda · 2024-07-22T11:12:11Z

src/platform_impl/linux/x11/event_processor.rs

                    event: WindowEvent::CursorMoved {
                        device_id: mkdid(util::VIRTUAL_CORE_POINTER),
                        position: location.cast(),
+                        r#type: CursorType::Mouse,
                    },
                };
                callback(&self.target, event);


Mouse cursor position changes when touch events are received.
Only the first concurrently active touch ID moves the mouse cursor.

This is a bit confusing to me, especially because I believe it doesn't map to other backends. Why do we tell the user that the mouse moves if the cursor moves with touch input?
Mouse input is cursor movement, but not all cursor movement is mouse input.

I'm really not sure what the purpose of this is.

daxpedda · 2024-07-22T12:26:10Z

src/event.rs

@@ -262,12 +266,12 @@ pub enum WindowEvent {
    /// [`transform`]: https://developer.mozilla.org/en-US/docs/Web/CSS/transform
    CursorLeft { device_id: DeviceId },

+    /// A cursor button press has been received.
+    CursorInput { device_id: DeviceId, state: ElementState, button: CursorButton },


At this point we should probably rename "Cursor" to "Pointer".
"Cursor" commonly refers to the visual representation of an input device on the screen.
But "Pointer" is a broader category that can include any device and is not specific to its visual representation on the screen.

I couldn't really find anything concrete on this online.
WDYT?

daxpedda · 2024-07-22T12:33:58Z

src/platform_impl/linux/x11/event_processor.rs

@@ -1121,7 +1125,7 @@ impl EventProcessor {

            let event = Event::WindowEvent {
                window_id,
-                event: WindowEvent::CursorMoved { device_id, position },
+                event: WindowEvent::CursorMoved { device_id, position, r#type: CursorType::Mouse },


I don't know how to ask X11 what kind of device this is.
Right now it reports my pen as a mouse.
Same in CursorEntered and CursorLeft.

Same issue in DeviceEvents.
Interestingly MouseWheel contains the pen angles, but MouseMotion should probably be excluded for pens as well, unless we want to rename it to CursorMotion (or PointerMotion).

Okay I figured this one out as well.
After digging around in Chromium for a while I found how they determined if this was a stylus or not.

After some digging, I also found that Glazier was doing something quite similar, in addition to detecting erasers (in a hacky way, but it works). There is a lot to learn from them when actually implementing the backends for Pens/Stylus's.

Additionally, after some more digging into X11, I found that there is an alternative to figuring out the type: XListInputDevices. That's right, using the old XInput version and no way to query individual devices. This exposes what they call "Device Types", which unfortunately don't include pens, but they let you recognize the drawing tablet, which works as well.

I found that Chromium was doing something similar for touchpads, but not for pens (and a comment in a much older revision explaining this as well). So maybe this will be useful for us in the future.

daxpedda · 2024-07-22T16:24:00Z

src/platform_impl/linux/x11/event_processor.rs

        let event = Event::WindowEvent {
            window_id,
-            event: WindowEvent::CursorMoved { device_id: mkdid(pointer_id as _), position },
+            event: WindowEvent::CursorMoved {
+                device_id: mkdid(pointer_id as _),
+                position,
+                r#type: CursorType::Mouse,
+            },
        };


This is also a bit confusing to me.
AFAIU focusing the window is not a mouse event and the mouse didn't move.

daxpedda · 2024-07-22T17:01:26Z

src/platform_impl/linux/x11/event_processor.rs

+        let Some(DeviceType::Mouse) = self
+            .devices
+            .borrow()
+            .get(&DeviceId(event.sourceid as xinput::DeviceId))


This came as a surprise to me, but I had to spend quite some time to figure out why this wasn't working for me.
deviceId here uses the cursor, but sourceId actually uses the device that moved the cursor. Shouldn't we use that to send DeviceId to the user?

src/event.rs

- This adds pen/stylus support. - Move `Force::Calibrated::altitude_angle` to `ToolAngle::altitude`. - `Force::normalized()` now takes a `Option<ToolAngle>` to calculate the perpendicular force. - Just introducing types, no implementation yet!

Add a new `CursorButton` and `ToolButton` type.

Fixed: device events are emitted regardless of cursor type.

daxpedda · 2024-07-27T23:02:17Z

After some discussion and thinking, I created #3833, which would basically result in a pre-cursor to this PR.
The implementation of #3833 will be partly based on the code here.

I will resume this PR when we are done with #3833.

daxpedda added the S - enhancement Wouldn't this be the coolest? label Jul 22, 2024

daxpedda commented Jul 22, 2024

View reviewed changes

src/event.rs Outdated Show resolved Hide resolved

Internal: rename Lazy to LazyLock

cb0874a

This aligns with the future Std type name and allows us to introduce `LazyCell`.

daxpedda force-pushed the pen branch 2 times, most recently from aac4f31 to c3b84f8 Compare July 22, 2024 10:31

daxpedda commented Jul 22, 2024

View reviewed changes

daxpedda force-pushed the pen branch from c3b84f8 to f0b9bdd Compare July 22, 2024 12:25

daxpedda commented Jul 22, 2024

View reviewed changes

daxpedda force-pushed the pen branch from f0b9bdd to 74bc843 Compare July 22, 2024 16:47

daxpedda commented Jul 22, 2024

View reviewed changes

src/event.rs Show resolved Hide resolved

daxpedda added 3 commits July 23, 2024 00:16

Rename WindowEvent::MouseButton to CursorButton

584a073

Add a new `CursorButton` and `ToolButton` type.

Web: Pen/Stylus implementation

67e2fec

Fixed: device events are emitted regardless of cursor type.

daxpedda force-pushed the pen branch from 74bc843 to 67e2fec Compare July 22, 2024 22:19

daxpedda marked this pull request as ready for review July 22, 2024 22:52

daxpedda requested review from jackpot51, notgull, kchibisov and madsmtm as code owners July 22, 2024 22:52

This was referenced Jul 23, 2024

Web: don't block pen input #3813

Merged

Pointer Event Overhaul #3833

Open

daxpedda marked this pull request as draft July 27, 2024 23:02

daxpedda added the DS - web label Jul 27, 2024

This was referenced Aug 16, 2024

Remove non-platform-consistent mouse events from X11 #3874

Open

Implement the pointer event overhaul #3876

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pen/Stylus support #3810

Pen/Stylus support #3810

daxpedda commented Jul 22, 2024 •

edited

Loading

daxpedda Jul 22, 2024

daxpedda Jul 22, 2024

daxpedda Jul 22, 2024

daxpedda Jul 22, 2024 •

edited

Loading

daxpedda Jul 22, 2024 •

edited

Loading

daxpedda Jul 22, 2024 •

edited

Loading

daxpedda Jul 22, 2024 •

edited

Loading

daxpedda commented Jul 27, 2024

Pen/Stylus support #3810

Are you sure you want to change the base?

Pen/Stylus support #3810

Conversation

daxpedda commented Jul 22, 2024 • edited Loading

daxpedda Jul 22, 2024

Choose a reason for hiding this comment

daxpedda Jul 22, 2024

Choose a reason for hiding this comment

daxpedda Jul 22, 2024

Choose a reason for hiding this comment

daxpedda Jul 22, 2024 • edited Loading

Choose a reason for hiding this comment

daxpedda Jul 22, 2024 • edited Loading

Choose a reason for hiding this comment

daxpedda Jul 22, 2024 • edited Loading

Choose a reason for hiding this comment

daxpedda Jul 22, 2024 • edited Loading

Choose a reason for hiding this comment

daxpedda commented Jul 27, 2024

daxpedda commented Jul 22, 2024 •

edited

Loading

daxpedda Jul 22, 2024 •

edited

Loading

daxpedda Jul 22, 2024 •

edited

Loading

daxpedda Jul 22, 2024 •

edited

Loading

daxpedda Jul 22, 2024 •

edited

Loading