Add touchscreen detection (and apply correct "TD" mod)

[TsavyPrince]

Describe the new feature:
Whenever I play osu!, I use a touch screen. However, even though the osu! the website shows a touchscreen mod icon on leaderboards, osu!lazer doesn't whenever I beat a map using a touchscreen. This also applies to multiplayer tag coop plays.
Proposal designs of the feature:
Add in mod icons for tag coop and touchscreen to display like any other mod on leaderboards.

[peppy]

I'm not sure if the touchscreen mod will be a thing in lazer. It was implemented as a mod on stable because it was the easiest way of making it happen, but really it should probably not be a mod.

This will become a focus once we are closer to having online leaderboards in lazer.

[frenzibyte]

@peppy I've added this issue thread to the "osu!(lazer) ranked play" project as it's crucial to prevent unfair advantage from touchscreen players in the leaderboards.

[Susko3]

I think ppy/osu-framework#5309 is relevant for this issue as the client-side touch detection will probably rely on the event type. And currently some pen tablets are seen as touch events in some cases.

[bdach]

Would be we relying solely on the incoming input event types here, or should stable's detection algorithm be somehow ported? (And if it's going to be the latter, the main question is whether that should be public source...)

[peppy]

If we can base on input detection, that is best IMHO. If we also want the algorithm method as a fallback, it can be made public (as long as the code isn't too bad).

The edge case I can think of is something like a wacom touch tablet where you can use touch or the pen. I'm not sure if we can discern between the two (I think OTD does, but have not confirmed).

[bdach]

I had a very rough go at an extremely basic implementation of this. How's this look UX-wise?

2023-10-25.20-28-31.mp4

• Uses input event type for now.
• Yes I know the "touch device" mod should start showing in the upper right of the screen too. The issue with that is that the entire player has thus far (and probably rightly so) assumed that mods are immutable. So for now I'm only appending the mod to Player.Score, but it doesn't show everywhere yet unless / until I upend half of Player to be able to respond to mod changes.
• The criteria to append the mod / show the toast are as follows:

        private void ensureTouchModActive()
        {
            if (gameplayState.HasPassed || gameplayState.HasFailed || gameplayState.HasQuit)
                return;

            if (player.Score.ScoreInfo.Mods.OfType<OsuModTouchDevice>().Any())
                return;

            if (player.IsBreakTime.Value)
                return;

            if (gameplayState.ScoreProcessor.HitEvents.Any())
                onScreenDisplay?.Display(new TouchDeviceDetectedToast());

            // TODO: this is kinda crude. `Player` (probably rightly so) assumes immutability of mods.
            // this probably should be shown immediately on screen in the HUD,
            // which means that immutability will probably need to be revisited.
            player.Score.ScoreInfo.Mods = player.Score.ScoreInfo.Mods.Append(new OsuModTouchDevice()).ToArray();
        }

i.e. I don't want a mobile user to get spammed with the toast every play, so if the user starts with touch, they don't get the toast. Similarly, touches are ignored for completed scores, and during pause, too.

Rest of source available for perusal at https://github.com/ppy/osu/compare/master...bdach:osu:basic-touchscreen-detection?expand=1 if so inclined.

I haven't tested if this submits properly yet either since that's finicky, but will do so if the UX gets a blessing (plus minus the "upper right mod display doesn't update" part).

[Susko3]

The edge case I can think of is something like a wacom touch tablet where you can use touch or the pen. I'm not sure if we can discern between the two (I think OTD does, but have not confirmed).

OTD does have ITouchReport, but the framework is not handling that specific device report. So touches from a tablet should be ignored.

Edit: tablet events are also handled via IAbsolutePointer, which handles IAbsolutePositionReports (including ITabletReport and IMouseReport).

[Susko3]

A player playing with a touchscreen would probably also navigate the main menu / song select with one. I think applying the mod earlier makes more sense. (The gameplay detection should still be there, to catch anyone trying to cheat or not using touch in song select).

Since TD affects star rating, having it applied at song select makes that more transparent to players. No issue with "I played a 6* map but didn't get the 6* medal."

Also solves the problem with Player not responding to mod changes.

[bdach]

I think applying the mod earlier makes more sense

Maybe farfetched but what if the user navigates the menus via touch but then switches to tablet or mouse only for gameplay?

Also either way the mod will show on the results screen, with the updated star rating I'd hope.

[peppy]

Interesting idea to have it showing earlier. I think this could work?

Maybe farfetched but what if the user navigates the menus via touch but then switches to tablet or mouse only for gameplay?

I think as long as it's very clear it was applied, and... unapplied when switching to mouse? maybe it could work. ie. it would only be in the correct state if they used the same device at song select, but at least it would give an accurate representation based on their current input device usage.

Of course, during gameplay it would only work in one direction (disabled -> enabled).

Maybe this all adds too much complexity though.

Yes I know the "touch device" mod should start showing in the upper right of the screen too.

Also this is arguable. Stable doesn't let the user know until they finish playing. If we want a MVP, we can copy that.

[bdach]

Showing it at song select in an acceptable way would entail

• adding the notion of "touch device" mod to ruleset API
• having song select use that to apply/unapply the mod (or not bother)

If you can accept this extra complexity then I guess it can be tried.

[peppy]

That doesn't sound like too much complexity, but I'd proof-of-concept it first to see how it works in practice. I'm not sure how much things like song select / mod select like a mod being toggled on arbitrarily like that (in theory we should have this covered these days but who knows).

[bdach]

I've written a version of this that would probably work from song select on (it does in tests, but I'm yet to test on an actual device): https://github.com/ppy/osu/compare/master...bdach:osu:less-basic-touchscreen-detection?expand=1

Not sure I like it too much. Aside from the two bullets above, there was also the rather large elephant in the room of being able to detect touch anywhere and based on that toggle the mod in song select. Which I ended up doing by putting a silent component that intercepts all input with priority at game level, i.e. adding extra overhead everywhere (if minuscule). So I'm not sure I like the extra added complexity for this, but maybe it's just me?

Adding to that I also had to add two components that just do a bit of bindable subscribing in song select and in player, just to keep the touch device code well away out of the 30 responsibilities that SongSelect and Player already have each.

[Susko3]

I have a touchscreen monitor so it's quick and easy for me to test.

Just tested your branch at 5a1f094 in a full game scenario and it crashes when changing mouse ⇆ touch when loading, during gameplay and in the results screen (list not exhaustive). Is this expected?
Works fine in the tests and in song select.

2023-10-30 17:45:43 [error]: System.InvalidOperationException: Can not set value to "osu.Game.Rulesets.Mods.Mod[]" as bindable is disabled.
2023-10-30 17:45:43 [error]: at osu.Framework.Bindables.Bindable`1.set_Value(T value)
2023-10-30 17:45:43 [error]: at osu.Game.Screens.Select.SongSelectTouchHandler.updateState() in C:\...\osu\osu.Game\Screens\Select\SongSelectTouchHandler.cs:line 53
2023-10-30 17:45:43 [error]: at osu.Game.Screens.Select.SongSelectTouchHandler.<LoadComplete>b__10_2(ValueChangedEvent`1 _) in C:\...\osu\osu.Game\Screens\Select\SongSelectTouchHandler.cs:line 37
2023-10-30 17:45:43 [error]: at osu.Framework.Bindables.Bindable`1.TriggerValueChange(T previousValue, Bindable`1 source, Boolean propagateToBindings, Boolean bypassChecks)
2023-10-30 17:45:43 [error]: at osu.Framework.Bindables.Bindable`1.SetValue(T previousValue, T value, Boolean bypassChecks, Bindable`1 source)
2023-10-30 17:45:43 [error]: at osu.Framework.Bindables.Bindable`1.TriggerValueChange(T previousValue, Bindable`1 source, Boolean propagateToBindings, Boolean bypassChecks)
2023-10-30 17:45:43 [error]: at osu.Framework.Bindables.Bindable`1.SetValue(T previousValue, T value, Boolean bypassChecks, Bindable`1 source)
2023-10-30 17:45:43 [error]: at osu.Framework.Bindables.Bindable`1.set_Value(T value)
2023-10-30 17:45:43 [error]: at osu.Framework.Configuration.ConfigManager`1.SetValue[TValue](TLookup lookup, TValue value)
2023-10-30 17:45:43 [error]: at osu.Game.Input.TouchInterceptor.Handle(UIEvent e) in C:\...\osu\osu.Game\Input\TouchInterceptor.cs:line 26
2023-10-30 17:45:43 [error]: at osu.Framework.Graphics.Drawable.OnMouseMove(MouseMoveEvent e)
2023-10-30 17:45:43 [error]: at osu.Framework.Graphics.Drawable.TriggerEvent(UIEvent e)
2023-10-30 17:45:43 [error]: at osu.Framework.Input.InputManager.PropagateBlockableEvent(SlimReadOnlyListWrapper`1 drawables, UIEvent e)
2023-10-30 17:45:43 [error]: at osu.Framework.Input.InputManager.handleMouseMove(InputState state, Vector2 lastPosition)
2023-10-30 17:45:43 [error]: at osu.Framework.Input.InputManager.HandleMousePositionChange(MousePositionChangeEvent e)
2023-10-30 17:45:43 [error]: at osu.Framework.Input.InputManager.HandleInputStateChange(InputStateChangeEvent inputStateChange)
2023-10-30 17:45:43 [error]: at osu.Framework.Input.UserInputManager.HandleInputStateChange(InputStateChangeEvent inputStateChange)
2023-10-30 17:45:43 [error]: at osu.Framework.Input.StateChanges.MousePositionAbsoluteInput.Apply(InputState state, IInputStateChangeHandler handler)
2023-10-30 17:45:43 [error]: at osu.Framework.Input.InputManager.Update()
2023-10-30 17:45:43 [error]: at osu.Framework.Input.PassThroughInputManager.Update()
2023-10-30 17:45:43 [error]: at osu.Framework.Graphics.Drawable.UpdateSubTree()
2023-10-30 17:45:43 [error]: at osu.Framework.Graphics.Containers.CompositeDrawable.UpdateSubTree()
2023-10-30 17:45:43 [error]: at osu.Framework.Platform.GameHost.UpdateFrame()
2023-10-30 17:45:43 [error]: at osu.Framework.Threading.GameThread.processFrame()

[bdach]

Is this expected?

Do I have to answer that? I did say it was untested...

I forgot about the decoupled mod bindable monstrosity in song select. That will piss in my cereal some more because now there's a possibility that a touch was registered while song select isn't current and the song select handler should do nothing while suspended but then actually apply the touch device mod on resume so it's not an immediately obvious fix that I could try and write without testing. This scenario wasn't exercised in the tests because there's no full-stack test exercising that yet I guess.

Scheduling the value changed callbacks in SongSelectTouchHandler could possibly maybe fix it I guess. But I'll test that theory later.

[bdach]

I've pushed fixes to the aforementioned branch, and solo seems to work alright in basic testing, but multiplayer / playlists are going to be the big remaining issue.

Basically, in both multiplayer and playlists:

• touch device must never be a required mod
• touch device must never be an explicitly-specified free mod
• ...but touch device must always be accepted on submission if it's there

This is a bit of a new notion and will likely require osu-web changes so that touch device can essentially be attached to a submitted multiplayer score at will completely bypassing the required/allowed mod systems.

[peppy]

One could argue that playlists should be able to require touch mod to create touch-only rooms (or touch-forbidden rooms)

Probably something to consider in the future, not now.

[Joehuu]

Resolved in #25348.