Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pen/Tablet Input Support #99

Open
ghost opened this issue Dec 11, 2016 · 38 comments · May be fixed by #3810
Open

Pen/Tablet Input Support #99

ghost opened this issue Dec 11, 2016 · 38 comments · May be fixed by #3810
Labels
C - needs discussion Direction must be ironed out D - hard Likely harder than most tasks here H - help wanted Someone please save us P - low Nice to have S - api Design and usability S - enhancement Wouldn't this be the coolest?

Comments

@ghost
Copy link

ghost commented Dec 11, 2016

I propose we support pen tablet input, such as on devices like the Surface Pro/Studio, Galaxy Note, and eventually the Apple Pencil. These devices support things like pressure sensitivity and rotation/gyro. I don't think this is necessarily relevant to games (although it definitely can be), but it's definitely relevant to the larger digital 3D community.

@tomaka tomaka added S - api Design and usability S - enhancement Wouldn't this be the coolest? labels Dec 11, 2016
jrmuizel pushed a commit to jrmuizel/winit that referenced this issue Mar 29, 2017
…ouseInput, r=paulroget

fix return coordinates in MouseInput

It is a fix of rust-windowing#97
When working on [servo/#11794](servo/servo#11794). I found the previous PR rust-windowing#97 is wrong. This PR is a fix of that and is right when testing [servo/#11794](servo/servo#11794).
Actually, we should return the mouse movement position instead of actual coordination. So the previous PR's return value is wrong.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/glutin/99)
<!-- Reviewable:end -->
@francesca64 francesca64 added H - help wanted Someone please save us C - needs discussion Direction must be ironed out D - hard Likely harder than most tasks here P - normal Great to have P - low Nice to have and removed P - normal Great to have labels May 6, 2018
@tangmi
Copy link
Contributor

tangmi commented Mar 22, 2019

There is an issue open on imgui (ocornut/imgui#2372) that has a little documentation and discussion on how to handle pen input across platforms that might be relevant when work on this starts.

@DorianRudolph
Copy link

I made a preliminary implementation for pen support on X11 here https://github.com/DorianRudolph/winit/tree/stylus

Pen pressure can already be accessed on X11 via the raw axis events, but there you get no information about the range of values.

@lehmanju
Copy link

@DorianRudolph What is the state of your implementation? Any help needed? I could look into stylus support for Wayland but first we should probably make sure the API exposed in Winit is correct.

@kchibisov
Copy link
Member

kchibisov commented Aug 14, 2020

@lehmanju if you're interested in contributing Wayland bits, you may add helper for table input to https://github.com/smithay/client-toolkit , since it's something that will be required anyway.

@DorianRudolph
Copy link

@lehmanju I'm not working on this right now. I would consider it a proof of concept. The implementation works, but I did not give much thought to the API. It might make sense to implement #336 prior to tablet support.

@kirawi
Copy link

kirawi commented Nov 18, 2020

A bit confused, but how would you consider it to be not implemented? It looks like it effectively is through Touch.

@tangmi
Copy link
Contributor

tangmi commented Nov 18, 2020

Pen input devices can have additional values, such as pen pressure, tilt, and azimuth that are not currently supported in winit. Additionally (and anecdotally), applications sometimes want to be able to distinguish between touch and pen input, even if most interactions end up being the same.

@kirawi
Copy link

kirawi commented Nov 18, 2020

Touch has the force field which seems to define pressure sensitivity and tilt. I could see the case for the latter and that it only works on IOS and Windows 8+ right now, but wouldn't you still consider that stylus input is mostly supported?

@tangmi
Copy link
Contributor

tangmi commented Nov 18, 2020

That's interesting... Looks like it's only used for the Apple Pencil:

let altitude_angle: Option<f64> = if touch_type == UITouchType::Pencil {

I think we'd still need to hook up azimuth to do that cool tilt shading effect Apple showed off when they released the Pencil...

Although even more broadly than that, I still think it's probably worth it to separate it out into a more generalized Pen event and implement it for all the platforms.

Edit: this may be a breaking change--at least on Windows, if one explicitly asks for pen input, the OS will stop sending them as touch inputs.

@kirawi
Copy link

kirawi commented Nov 18, 2020

It looks like Windows does have pressure sensitivity support:

winuser::PT_PEN => {
let mut pen_info = mem::MaybeUninit::uninit();
GET_POINTER_PEN_INFO.and_then(|GetPointerPenInfo| {
match GetPointerPenInfo(
pointer_info.pointerId,
pen_info.as_mut_ptr(),
) {
0 => None,
_ => {
normalize_pointer_pressure(pen_info.assume_init().pressure)
}
}
})
}
_ => None,

Rotation/tilt should be trivial to add as well.

I'm confused why normalize_pointer_pressure() expects to only get a maximum of 1024, though this issue seems to imply that the Win32 Pointer API is limited to that number compared to WinRT. It might be worth investigating that in the future?

Edit:
Yes, Win32 is limited to 1024.

@tangmi
Copy link
Contributor

tangmi commented Nov 18, 2020

I'll admit the last time I seriously used winit was pre-0.20, and pen support wasn't sufficient for any non-trivial pen-based apps (for reference on what's missing, here's a proof concept supporting pen on Windows and web here 2e11615...tangmi:winit-legacy-2). I don't believe that any significant support for pen/tablet input was added since then, so I think that it's meaningful to keep this issue open.

@kirawi
Copy link

kirawi commented Nov 18, 2020

Agreed, it doesn't seem very mature yet.

@tasgon
Copy link

tasgon commented Mar 3, 2021

Looking through @DorianRudolph's PR, it looks like pressure is normalized, so could we use the Touch event in a similar way to how it's done on Windows (instead of adding another event struct)?

@ActuallyHappening
Copy link

Being able to handle pen double-taps on my iPad would be amazing!

@madsmtm madsmtm added the C - nominated Nominated for discussion in the next meeting label Jun 25, 2024
@madsmtm
Copy link
Member

madsmtm commented Jun 25, 2024

Thanks for the interest in working on this in #3759 @ActuallyHappening! I think the hardest part about this is not so much the implementation, but more figuring out what the generated WindowEvents should look like - maybe you (or someone else) could start by suggesting an API here, and then we can work from there?

@ActuallyHappening
Copy link

ActuallyHappening commented Jun 25, 2024

I work primarily with the bevy wrappers around winit. From that point of view, the only event I would really care about looks like this (allowing for forwards compatibility):

#[non_exhastive]
#[derive(Debug)[
pub enum PenEvent {
  DoubleTab,
  // Squeeze // todo
}

Thinking about this more, I believe only apple pens have the specific 'Double Tap' feature. Of course there would be documentation to show this. Keeping the enum descriptive (rather than pub struct PenEvent { is_tap: bool }) seems a better design, but my only requirement (for my specific use case) is the ability to know when (an event) I double tap my Apple Pencil. It would also be good to wrap some specific information about the requested action is, see preferedTabAction https://developer.apple.com/documentation/uikit/uipencilinteraction/3039593-preferredtapaction

ActuallyHappening added a commit to ActuallyHappening/winit that referenced this issue Jun 28, 2024
@ActuallyHappening ActuallyHappening mentioned this issue Jun 28, 2024
5 tasks
@madsmtm madsmtm removed the C - nominated Nominated for discussion in the next meeting label Jun 28, 2024
@madsmtm madsmtm added this to the Version 0.31.0 milestone Jun 28, 2024
@madsmtm
Copy link
Member

madsmtm commented Jun 28, 2024

Resolution from the meeting today: We want to start with something matching the web API, specifically pointerType, twist and tilt[X|Y]. @daxpedda wants to try to do this implementation.

On top of that, we could probably add some extra methods to know if an event was e.g. a double tap or a special button press, but we want to start with the above.

@ActuallyHappening
Copy link

ActuallyHappening commented Jun 28, 2024

OK interesting, though in the link to the web API you mentioned there is no functionality specific to Apple Pencils / Galaxy SPens buttons. I.e., after implementing the entire web API, I would still not be able to handle Apple Pencil double tap (or squeeze) events like I want to (see bevy issue and #3759).

Would this API be mutually compatible with the decided web API?

/// Pen events not necessarily tied to a normal touch, move, or drag e.t.c.
pub enum PenSpecialEvent {
  DoubleTap { preferred_action: Option<PenPreferredTapAction> },
  // Squeeze // todo
}

I guess I should explain that the Apple Pencil double tap feature often occurs when the pen is not touching the screen, so information like screen position doesn't make sense for it.

@madsmtm
Copy link
Member

madsmtm commented Jun 28, 2024

Maybe it would help to know what kind of events the pen generates when used in a web page?

ActuallyHappening added a commit to ActuallyHappening/winit that referenced this issue Jun 28, 2024
@ActuallyHappening
Copy link

ActuallyHappening commented Jun 29, 2024

Ahh I see, trying to mimic the Web API for native.
I initially tried to gain pen events on web, but special Apple pen events (i.e. DoubleTap) don't trigger anything on Web, which was one of the initial factors pushing me to use rust for native in the first place, so that I could use the features of my apple pencil. I haven't checked that this is still the case recently however.

Looking over the web api we could merge it with the features I want.
From the pointerType property:

The event's pointer type. The supported values are the following strings:
"mouse"
The event was generated by a mouse device.
"pen"
The event was generated by a pen or stylus device.
"touch"
The event was generated by a touch, such as a finger.
If the device type cannot be detected by the browser, the value can be an empty string (""). If the browser supports pointer device types other than those listed above, the value should be vendor-prefixed to avoid conflicting names for different types of devices.

I understand more what you mean by extending the official spec now:

On top of that, we could probably add some extra methods to know if an event was e.g. a double tap or a special button press, but we want to start with the above.

This does seem to be a more consistent and unified API.

The only problem I see with trying to add Apple Pencil double tap event support is that the event itself is not associated with any of the touch event types in the web API spec, here. And, it is required that each pointer event from the spec has a specific .clientX, .clientY e.t.c. from the MouseEvent interface however apple pencil double tap events don't have an associated position on the screen, because you are often holding the pencil off the screen when you double tap

@ActuallyHappening
Copy link

I'm still eager to implement this, see #3768

@daxpedda
Copy link
Member

The only problem I see with trying to add Apple Pencil double tap event support is that the event itself is not associated with any of the touch event types in the web API spec, here.

This is to be expected, there will be a bunch of special events that just don't fit anywhere and maybe are platform-specific with no overlap.

I think to start the design work of post-Web-API-pencil-input we want a proper overview.

  1. A list of relevant events for each backend.
  2. Find overlap between the backend.
  3. Consider which ones to implement or if we want/can let external crates to implement those.

Documentation:

@ActuallyHappening
Copy link

I'm thinking about my future considerations, I'm wanting to custom build a lot of stuff relevant only to iOS.
For that I should really only need a reference to the WinitUIView and WinitUIViewController and the ability to implement protocols on it, but I believe objc2 can only implement objective C protocols in the defining crate.

I'm out of my depth here, but is it possible for winit to expose the ability to add custom views and view controllers to the base / default without messing up any of winits abstractions?
That seems the cleanest and best option to me, since I can then maintain my own crates which implement specific features for iOS only rather than peppering winit with PRs for every new iOS specific change.

I think to start the design work of post-Web-API-pencil-input we want a proper overview.

  • A list of relevant events for each backend.
  • Find overlap between the backend.
  • Consider which ones to implement or if we want/can let external crates to implement those.

I can only speak confidently about iOS since it's the only hardware I have. I'm happy to make a list of iOS features I would eventually want:

@ActuallyHappening
Copy link

I don't think PencilKit can be easily integrated into winit since it relies on a bunch of properties specific to apple's view hierarchy, e.g. inheriting scrolling. To be honest I discounted it immediately when I though that you couldn't get at the underlying data, but that doesn't seem the case

@daxpedda
Copy link
Member

I'm happy to make a list of iOS features I would eventually want ...

The goal is to make a more or less complete list. If you include the links to the apple documentation directly, I'm happy to open an issue about platform-specific functionality so we can start collecting and designing an API for them.

@Doublonmousse
Copy link

As the windows side of the implementation was mentioned, and the proposed implementation is relying on the Win32 API, I would like to say that the Win32 API is limited in some ways that create unwanted behavior (as in, different from any other implementation, with my main gripe being the absence of information on the button presses when hovering. The result is that buttons are usually not working correctly on windows for any application using gtk3/gtk4 or qt).

I've listed my concerns (and a fix using additional WinRT API calls) on an existing PR here: #2396 (comment)

Other Windows-specific features related to pens are only available using WinRT APIS (haptic feedback, pressure levels with more than 1024 levels)

@Doublonmousse
Copy link

Another thing to note on windows is that pen APIs are split between Windows Ink (what we get from the Win32 or WinRT APIS) and Wintab, and these two are still used today.

Windows Ink is used by all modern tablets with a pen digitizer on the screen, whilst Wintab is still widely used by external graphical tablets (wacom, huion). And of course, if your graphical tablet uses Wintab as the API and you're getting events from the Windows Ink API, these events are seen as mouse events, not pen events ...

Usually you have to choose one of the two APIs to catch pen events and restart the app if you want to switch APIs.

https://crates.io/crates/wintab_lite This seems like the only rust crate that deals with wintab right now, although, as I've only looked at the Windows ink part, I'm not familiar with the wintab part. (I know both qt and gtk have a wintab option, so there's other C/C++ code that implement the Wintab API).

@thehappycheese
Copy link

https://crates.io/crates/wintab_lite This seems like the only rust crate that deals with wintab right now, although, as I've only looked at the Windows ink part, I'm not familiar with the wintab part. (I know both qt and gtk have a wintab option, so there's other C/C++ code that implement the Wintab API).

Oh hi there :) I am the author of that wintab_lite crate.

FYI i gave up trying to integrate with winit because winit blocks access to the window event loop by making it private.

wintab does provide an alternative polling-based api to obtain events, I think that method sucks. It is prone to overflowing buffers on the driver side which causes inexplicable bad behaviour. For this reason I think winit's choice to hide the event loop is not compatible with wintab... unless i am somehow mistaken.

This is unfortunate, because I find Windows Ink to be highly unreliable and flakey. Wintab has been working solid as a rock for decades.

@Doublonmousse
Copy link

Doublonmousse commented Aug 15, 2024

Hey. I think part of the reason windows Ink is flaky is the use of the win32 API over the WinRT one (hence why I tried to use WinRT calls in addition to the win32 ones to get the missing info or more info on events, and that gives correct info for pure windows ink devices like a surface pen). That means that for most windows software supporting windows ink that aren't WinUI note taking applications the support is limited and has been for a while now.

But also, as far as external graphics tablets are concerned, it's clear to me the optional windows ink mode is usually less polished and featured, although the fault may be split between drivers manufacturers who already have a good wintab drivers not needing an Ink one, and Windows Ink being less featured than wintab (sometimes using the open tablet driver + windows ink can give you some level of functionality that the official driver lacks in windows ink mode).

But enough ranting, do you think integrating your crate inside of the event loop (not doing this as an external crate, but injecting the necessary code inside of the winit crate directly, similarly to me doing win32 and WinRT calls to get pen info here 0ba247b) could be done instead ?
(also if someone know a better way to get pen info from WinRT let me know)

One other point you might have an answer to : on most toolkits that have wintab and windows ink support, you can usually only choose one and need to restart to use the other one (that's the case for both gtk and qt). Is there a fundamental limitation that makes having both activated at the same time impossible ?
Cause if you have a device that has both a windows ink device (like a pen digitiser) + an external graphics tablet that uses wintab (and you don't want/can't use the windows ink option), there's always one device that won't work correctly.

Anyway, once the API is decided in #3833 and proper platform-dependent support can be added to winit, I'd like to make sure it doesn't have the usual limitations other windows implementation have on the windows ink side.
And if we can do better, why not try to have support for both at the same time (windows ink and wintab) without a need for a restart ?

@daxpedda
Copy link
Member

daxpedda commented Aug 15, 2024

FYI i gave up trying to integrate with winit because winit blocks access to the window event loop by making it private.

wintab does provide an alternative polling-based api to obtain events, I think that method sucks. It is prone to overflowing buffers on the driver side which causes inexplicable bad behaviour. For this reason I think winit's choice to hide the event loop is not compatible with wintab... unless i am somehow mistaken.

We plan to address this with #2120 and the move to a trait-based API, #3432, with the goal that external crates can integrate with Winit exactly like wintab_lite could do.

I was a bit shocked with the terms "private" and "hide the event loop", the event loop is a function and integrating with it requires more API then just "making it public". But maybe I'm missing something, I'm not very familiar with the Windows backend.

But enough ranting, do you think integrating your crate inside of the event loop (not doing this as an external crate, but injecting the necessary code inside of the winit crate directly, similarly to me doing win32 and WinRT calls to get pen info here 0ba247b) could be done instead ?

On first inspection I don't think its viable to add Wintab support directly into Winit. We want to limit the scope to cross-platform functionality and only implement and expose whats absolutely necessary.

But as said above, I think its incredibly important that Winit allows external crates to integrate into the event loop exactly for reasons like this. On the other hand, I would love to see Wintab and Windows Ink support in winit-extra (doesn't exist yet).

@thehappycheese
Copy link

Do you think integrating your crate inside of the event loop (not doing this as an external crate, but injecting the necessary code inside of the winit crate directly, similarly to me doing win32 and WinRT calls to get pen info here 0ba247b) could be done instead?

I am sure a built in solution would work great :) My plate is a little full at the moment to try help, but feel free to copy-pasta my code all you like. I spent a lot of time adding nice rust documentaiton to it.

Is there a fundamental limitation that makes having both activated at the same time impossible

Not as far as I know... I have a Huion device, the configuration software has a toggle to turn windows ink on and off... as far as I can tell the toggle is broken and does nothing. It still works in windows ink apps. Maybe windows has some fallback and translates wintab into windows ink... i didnt look into it.

I can tell you this though; for about a month windows ink was not working for me. The fix: disable logitech RGB LED driver that I didnt even know existed. Go figure? It did not interfere with wintab though.

We plan to address this with #2120 and the move to a trait-based API, #3432, with the goal that external crates can integrate with Winit exactly like wintabl_lite could do.

I was a bit shocked with the terms "private" and "hide the event loop", the event loop is a function and integrating with it requires more API then just "making it public". But maybe I'm missing something, I'm not very familiar with the Windows backend.

Ah thats exciting to hear! By private I mean that at the time I looked, the values windows passed to the message / wparam / lparam parameters of the callback registered with CreateWindowExA() do not appear to be avaliable through the winit API. I dont remember the exact details... I think winit maybe sets up the window / event loop / polling differently to what I understand.

On first inspection I don't think its viable to add Wintab support directly into Winit. We want to limit the scope to cross-platform functionality and only implement and expose whats absolutely necessary.

Possibly licencing is also an issue for serious projects? Wacom's licence / coppyright notice on thier repo seem contradictory to my primitive understanding of these things.

Thank you all for your hard work on winit though, it is otherwise absolutly fantastic and it sounds like its going to be even better in the future! <3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C - needs discussion Direction must be ironed out D - hard Likely harder than most tasks here H - help wanted Someone please save us P - low Nice to have S - api Design and usability S - enhancement Wouldn't this be the coolest?
Development

Successfully merging a pull request may close this issue.