drivers/touch_dev_gestures: add gesture recognition for touch devices #19884

gschorcht · 2023-08-14T16:37:42Z

Contribution description

This PR adds simple gesture recognition for touch devices accessed via the generic Touch Device API. It can be used in conjunction with device drivers that use either interrupts or polling mode. It supports up to two touches and the following gestures:

Single and double tap at given position
Long press and release given position
Moving while pressed with current position
Swipe left, right, up and down
Zoom in (spread) and out (pinch)

Gesture recognition has been tested with:

stm32f746g-disco (works out of the box)
stm32f723e-disco (works out of the box)
stm32f429i-disc1 (works on top of PR drivers/stmpe811: changes for interrupt-driven touch handling and gesture recognition #19885)
stm32l496g-disco (works with my local LCD display changes waiting for PR drivers: rename st7735 to more generic st77xx #19825, not yet provided)
esp32s3-wt32-sc01-plus (new board, not yet provided)

Testing procedure

Flash tests/drivers/touch_dev_gestures to a board with touch pane, for example:

BOARD=stm32f746g-disco make -j8 -C tests/drivers/touch_dev_gestures/ flash

PR #19885 is required for the stm32f429i-disc1 board.

The output should look like this:

main(): This is RIOT! (Version: 2023.10-devel-121-g81c5c-drivers/touch_dev_gestures)
Single Tap X: 255, Y:154
Single Tap X: 253, Y:153
Double Tap X: 253, Y:149
Swipe right
Swipe down
Swipe left
Swipe up
Pressed    X: 257, Y:155
Moving     X: 257, Y:155
Moving     X: 257, Y:155
Moving     X: 259, Y:156
Moving     X: 262, Y:157
Moving     X: 266, Y:158
Moving     X: 269, Y:160
Moving     X: 273, Y:162
Moving     X: 276, Y:165
Moving     X: 278, Y:167
Moving     X: 278, Y:169
Moving     X: 278, Y:169
Released   X: 279, Y:172

Issues/PRs references

riot-ci · 2023-08-14T16:56:35Z

Murdock results

✔️ PASSED

2d3409b tests/drivers/touch_dev_gestures: add gesture recognition test

Success	Failures	Total	Runtime
7955	0	7955	13m:11s

Artifacts

Documentation preview

aabadie · 2023-08-24T12:37:50Z

cst816s touch device is not adapted by this PR and the driver is already proposing some gesture recognition (swipe, single/double/long press). Do you think it is possible to also adapt it to the touch_dev_gesture module ?

gschorcht · 2023-08-25T15:23:03Z

cst816s touch device is not adapted by this PR and the driver is already proposing some gesture recognition (swipe, single/double/long press).

Yeah, I already saw that. I just wanted to start with the low hanging fruits 😉

Do you think it is possible to also adapt it to the touch_dev_gesture module ?

It should be possible but might become a bit tricky. Usually, the app calls the touch_dev_gesture module either event-driven or on regular base to get recognized gestures. The touch_dev_device simply calls the touch device to get the current position and uses the current as well as previous positions to detect the gestures.

If the touch device already detects some gestures, either we don't use the gesture recognition of the device at all or we use some gestures from the touch and realize the missing gestures in the touch_dev_device module. This would require some additional configuration what gestures are supported by the touch device and what gestures the touch_dev_device module has to implement.

We need the current position anyway to detect the gestures the touch device doesn't support.

19885: drivers/stmpe811: changes for interrupt-driven touch handling and gesture recognition r=aabadie a=gschorcht ### Contribution description This PR contains changes needed for the purely interrupt-driven handling of the touch position as `touch_dev` device, which is a prerequisite for the realization of gesture recognition. (PR #19884). The interrupt-driven approach of `touch_dev` devices (PR #19882) and the gesture recognition (PR #19884) need continuous reporting of the touch position as long as there is at least one touch. Since the driver so far only uses the _Touch Detect_ interrupt, only the position at the beginning of a touch is available. All further positions must be polled. Therefore, the changes in this PR additionally enable the _FIFO Threshold_ interrupt when the `touch_dev` module is enabled. However, since the _Touch Detect_ interrupt does not work reliably when the _FIFO Threshold_ interrupt is enabled and the FIFO Threshold is 1, the FIFO Threshold is set to 2 by default when the `touch_dev` module is enabled. Furthermore, the FIFO queue has to be reset after reading one touch position. Otherwise new touch positions are processed with a delay if the rate of calling the function to read the FIFO is slower than the rate at which the FIFO is filled. The reason for this is that with each call of this function only the oldest touch position is read value by value from the FIFO. Gestures can't be implemented with such a behavior. ### Testing procedure 1. `tests/drivers/stmpe811` should work as before (only a single position is shown): ``` BOARD=stm32f429i-disco make -j8 -C tests/drivers/stmpe811 flash ``` ``` main(): This is RIOT! (Version: 2023.10-devel-120-g848d1- drivers/stmpe811_touch_dev_gestures) STMPE811 test application +------------Initializing------------+ Initialization successful Pressed! X: 135, Y:131 Released! ``` 2. `tests/drivers/stmpe811` should work on top of PR #19882 with continuous outputs of touch positions: ``` BOARD=stm32f429i-disco make -j8 -C tests/drivers/touch_dev flash ``` ``` main(): This is RIOT! (Version: 2023.10-devel-121-g38d3db-drivers/stmpe811_touch_dev_gestures) Event: pressed! X: 131, Y:145 X: 133, Y:141 X: 135, Y:138 X: 138, Y:133 X: 141, Y:128 X: 146, Y:122 X: 151, Y:117 Event: released! ``` 3. `tests/driver/touch_dev_gestures` of PR #19884 should work on top of this PR: ``` BOARD=stm32f429i-disco make -j8 -C tests/drivers/touch_dev_gestures flash ``` ``` main(): This is RIOT! (Version: 2023.10-devel-128-g05f690-drivers/touch_dev_gestures_work) Swipe right Swipe left Swipe down Swipe up Single Tap X: 246, Y:148 Single Tap X: 256, Y:139 Double Tap X: 247, Y:136 Pressed X: 246, Y:131 Moving X: 248, Y:132 Moving X: 250, Y:133 Moving X: 258, Y:135 Moving X: 270, Y:136 Moving X: 285, Y:132 Moving X: 300, Y:126 Moving X: 309, Y:122 Moving X: 310, Y:119 Released X: 310, Y:119 ``` ### Issues/PRs references Co-authored-by: Gunar Schorcht <gunar@schorcht.net>

drivers/include/touch_dev_gestures.h

tests/drivers/touch_dev_gestures/main.c

aabadie

looks good, please squash!

aabadie · 2023-09-01T05:23:52Z

bors merge

19884: drivers/touch_dev_gestures: add gesture recognition for touch devices r=aabadie a=gschorcht ### Contribution description This PR adds simple gesture recognition for touch devices accessed via the generic Touch Device API. It can be used in conjunction with device drivers that use either interrupts or polling mode. It supports up to two touches and the following gestures: - Single and double tap at given position - Long press and release given position - Moving while pressed with current position - Swipe left, right, up and down - Zoom in (spread) and out (pinch) Gesture recognition has been tested with: - [x] `stm32f746g-disco` (works out of the box) - [x] `stm32f723e-disco` (works out of the box) - [x] `stm32f429i-disc1` (works on top of PR #19885) - [x] `stm32l496g-disco` (works with my local LCD display changes waiting for PR #19825, not yet provided) - [x] `esp32s3-wt32-sc01-plus` (new board, not yet provided) ### Testing procedure Flash `tests/drivers/touch_dev_gestures` to a board with touch pane, for example: ``` BOARD=stm32f746g-disco make -j8 -C tests/drivers/touch_dev_gestures/ flash ``` PR #19885 is required for the `stm32f429i-disc1` board. The output should look like this: ``` main(): This is RIOT! (Version: 2023.10-devel-121-g81c5c-drivers/touch_dev_gestures) Single Tap X: 255, Y:154 Single Tap X: 253, Y:153 Double Tap X: 253, Y:149 Swipe right Swipe down Swipe left Swipe up Pressed X: 257, Y:155 Moving X: 257, Y:155 Moving X: 257, Y:155 Moving X: 259, Y:156 Moving X: 262, Y:157 Moving X: 266, Y:158 Moving X: 269, Y:160 Moving X: 273, Y:162 Moving X: 276, Y:165 Moving X: 278, Y:167 Moving X: 278, Y:169 Moving X: 278, Y:169 Released X: 279, Y:172 ``` ### Issues/PRs references Co-authored-by: Gunar Schorcht <gunar@schorcht.net>

bors · 2023-09-01T08:58:30Z

Build failed:

Murdock

aabadie · 2023-09-01T09:01:37Z

bors merge

19884: drivers/touch_dev_gestures: add gesture recognition for touch devices r=aabadie a=gschorcht ### Contribution description This PR adds simple gesture recognition for touch devices accessed via the generic Touch Device API. It can be used in conjunction with device drivers that use either interrupts or polling mode. It supports up to two touches and the following gestures: - Single and double tap at given position - Long press and release given position - Moving while pressed with current position - Swipe left, right, up and down - Zoom in (spread) and out (pinch) Gesture recognition has been tested with: - [x] `stm32f746g-disco` (works out of the box) - [x] `stm32f723e-disco` (works out of the box) - [x] `stm32f429i-disc1` (works on top of PR #19885) - [x] `stm32l496g-disco` (works with my local LCD display changes waiting for PR #19825, not yet provided) - [x] `esp32s3-wt32-sc01-plus` (new board, not yet provided) ### Testing procedure Flash `tests/drivers/touch_dev_gestures` to a board with touch pane, for example: ``` BOARD=stm32f746g-disco make -j8 -C tests/drivers/touch_dev_gestures/ flash ``` PR #19885 is required for the `stm32f429i-disc1` board. The output should look like this: ``` main(): This is RIOT! (Version: 2023.10-devel-121-g81c5c-drivers/touch_dev_gestures) Single Tap X: 255, Y:154 Single Tap X: 253, Y:153 Double Tap X: 253, Y:149 Swipe right Swipe down Swipe left Swipe up Pressed X: 257, Y:155 Moving X: 257, Y:155 Moving X: 257, Y:155 Moving X: 259, Y:156 Moving X: 262, Y:157 Moving X: 266, Y:158 Moving X: 269, Y:160 Moving X: 273, Y:162 Moving X: 276, Y:165 Moving X: 278, Y:167 Moving X: 278, Y:169 Moving X: 278, Y:169 Released X: 279, Y:172 ``` ### Issues/PRs references Co-authored-by: Gunar Schorcht <gunar@schorcht.net>

bors · 2023-09-01T12:36:33Z

Build failed:

Murdock

aabadie · 2023-09-01T12:56:56Z

bors merge

bors · 2023-09-01T16:29:59Z

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

gschorcht · 2023-09-02T11:47:57Z

@aabadie Thanks for reviewing and merging.

gschorcht added 2 commits August 14, 2023 18:26

drivers/touch_dev: define invalid touch position

09f0726

drivers/touch_dev: add a function to get the maximum number of touches

a5a08f0

gschorcht requested review from leandrolanzieri, jia200x, MrKevinWeiss, smlng, aabadie, MichelRottleuthner and fjmolinas as code owners August 14, 2023 16:37

github-actions bot added Area: doc Area: Documentation Area: tests Area: tests and testing framework Area: drivers Area: Device drivers Area: Kconfig Area: Kconfig integration labels Aug 14, 2023

gschorcht added Type: new feature The issue requests / The PR implemements a new feature for RIOT CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Aug 14, 2023

gschorcht force-pushed the drivers/touch_dev_gestures branch from 81c5c21 to c88d67f Compare August 14, 2023 16:42

gschorcht mentioned this pull request Aug 14, 2023

drivers/stmpe811: changes for interrupt-driven touch handling and gesture recognition #19885

Merged

gschorcht changed the title ~~drivers: add gesture recognition for touch devices~~ drivers/touch_dev_gestures: add gesture recognition for touch devices Aug 15, 2023

aabadie reviewed Aug 31, 2023

View reviewed changes

drivers/include/touch_dev_gestures.h Outdated Show resolved Hide resolved

tests/drivers/touch_dev_gestures/main.c Outdated Show resolved Hide resolved

tests/drivers/touch_dev_gestures/main.c Outdated Show resolved Hide resolved

aabadie approved these changes Aug 31, 2023

View reviewed changes

gschorcht added 2 commits August 31, 2023 20:12

drivers: add gesture recognition for touch devices

54ec84d

tests/drivers/touch_dev_gestures: add gesture recognition test

2d3409b

gschorcht force-pushed the drivers/touch_dev_gestures branch from ab5218f to 2d3409b Compare August 31, 2023 18:18

bors bot merged commit 930811d into RIOT-OS:master Sep 1, 2023

gschorcht deleted the drivers/touch_dev_gestures branch September 2, 2023 14:59

MrKevinWeiss added this to the Release 2023.10 milestone Nov 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

drivers/touch_dev_gestures: add gesture recognition for touch devices #19884

drivers/touch_dev_gestures: add gesture recognition for touch devices #19884

gschorcht commented Aug 14, 2023 •

edited

Loading

riot-ci commented Aug 14, 2023 •

edited

Loading

aabadie commented Aug 24, 2023

gschorcht commented Aug 25, 2023

aabadie left a comment

aabadie commented Sep 1, 2023

bors bot commented Sep 1, 2023

aabadie commented Sep 1, 2023

bors bot commented Sep 1, 2023

aabadie commented Sep 1, 2023

bors bot commented Sep 1, 2023

gschorcht commented Sep 2, 2023

drivers/touch_dev_gestures: add gesture recognition for touch devices #19884

drivers/touch_dev_gestures: add gesture recognition for touch devices #19884

Conversation

gschorcht commented Aug 14, 2023 • edited Loading

Contribution description

Testing procedure

Issues/PRs references

riot-ci commented Aug 14, 2023 • edited Loading

Murdock results

Artifacts

aabadie commented Aug 24, 2023

gschorcht commented Aug 25, 2023

aabadie left a comment

Choose a reason for hiding this comment

aabadie commented Sep 1, 2023

bors bot commented Sep 1, 2023

aabadie commented Sep 1, 2023

bors bot commented Sep 1, 2023

aabadie commented Sep 1, 2023

bors bot commented Sep 1, 2023

gschorcht commented Sep 2, 2023

gschorcht commented Aug 14, 2023 •

edited

Loading

riot-ci commented Aug 14, 2023 •

edited

Loading