Add virtual display feature #5370
Conversation
|
Hi @rom1v, binary file cannot be downloaded |
|
@huynhtanloc2612 Thanks, fixed. |
|
@rom1v |
|
tested this on mac(m1pro) - works! Steps I followed -
|
|
Thank you for your contribution. Test platform: Mac(M2)/Meizu 21 Test way: git fetch origin pull/5370/head:pr-5370
git checkout pr-5370
meson setup x --buildtype=release --strip -Db_lto=true
ninja -Cx # DO NOT RUN AS ROOT
./run x --new-display=1920x1080I mainly tested the following scenarios:
Due to this flag VIRTUAL_DISPLAY_FLAG_SHOULD_SHOW_SYSTEM_DECORATIONSOn the Meizu device I tested, the Launcher will start on this virtual display. This is not an issue, but rather an unexpected benefit. Here is a screenshot. 
By the way, I have ported scrcpy to Android. Once I complete the remaining work, I invite you to experience this software. Thank you very much. |
|
A small issue: IME will not appear on the virtual display unless the following setting is applied: This is likely because the Additionally, on some devices, the launcher displays incorrectly on non-default screens. |
Launcher I tested that works: Lawnchair (playstore, github). Another instance of the launcher can be opened on secondary displays without additional flags. Good enough for quickly launching apps for now scrcpy-vd-lawnchair.mp4 |
|
@rom1v thank you for working on this feature! I'm getting permission issue on android 12. @yume-chan's proof-of-concept works on this device |
|
We probably need to remove this flag for Android < 13 (according to c7b04a6): diff --git a/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java b/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
index 0622998ed..9440199ce 100644
--- a/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
+++ b/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
@@ -85,7 +85,6 @@ public class NewDisplayCapture extends SurfaceCapture {
| VIRTUAL_DISPLAY_FLAG_ROTATES_WITH_CONTENT
| VIRTUAL_DISPLAY_FLAG_DESTROY_CONTENT_ON_REMOVAL
| VIRTUAL_DISPLAY_FLAG_SHOULD_SHOW_SYSTEM_DECORATIONS
- | VIRTUAL_DISPLAY_FLAG_TRUSTED
| VIRTUAL_DISPLAY_FLAG_OWN_DISPLAY_GROUP
| VIRTUAL_DISPLAY_FLAG_ALWAYS_UNLOCKED
| VIRTUAL_DISPLAY_FLAG_TOUCH_FEEDBACK_DISABLED |
|
|
@eiyooooo The flag exists, but apparently in Android 12, shell does not have permission to create a virtual display with this flag enabled. |
Yes, at least Android 13 is required to use the |
If you don't pass it, what happens? |
Based on my tests, the flags |
On the S20+ (Android 13), after multiple attempts to create virtual displays using |
|
I added features to list and start apps. I updated the main post, and also published a binary for this new version. Please test and give feedback 😉 |
rom1v
force-pushed
the
virtual_display
branch
from
51fff66 to
11eca93
Compare
After digging deeper, I found that Android 12L still encounters the following issue: References:
Therefore, I think this part of the code: should be updated to: if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) { |
rom1v
force-pushed
the
virtual_display
branch
from
11eca93 to
e828667
Compare
|
@eiyooooo OK.
Indeed, it has been merged in Android 13: # in aosp/framework_base
git tag --contains 990e3429636382175ca8e7bf04df054f48fbd130I first added readable Android version constants (already merged into Then I set the TRUSTED and other flags only since Android 13: e05be0b |
@rom1v By the way, Android 15 has been released, so you'll need to add a new Android version constant. 😄 public static final int API_35_ANDROID_15 = Build.VERSION_CODES.VANILLA_ICE_CREAM; |
rom1v
force-pushed
the
virtual_display
branch
from
e828667 to
b771db3
Compare
|
Given that loading the app names may take time (several seconds), by default, I reworked so that For convenience, however, it is still possible to explicitly select an app by its name (but with some delay) by adding a Like in the previous version, a scrcpy --new-display --start-app=+org.mozilla.firefox # by package name
scrcpy --new-display --start-app=+?firefox # by name |
FYI, Actually there is per-display audio routing for companion service: https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/media/java/android/media/PlayerBase.java;l=582;drc=0a293aabaef57434b7cde9f431bacf6dc3ee7fe2 If we create the VirtualDisplay using companion service: https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/services/companion/java/com/android/server/companion/virtual/VirtualDeviceManagerService.java;l=494;drc=0a293aabaef57434b7cde9f431bacf6dc3ee7fe2 It also modifies audio routing like my previous comment, just automatically based on the display the app is on. Companion service is a hidden API for projecting displays, inputs, audio and even cameras to companion devices (like smart watches). Normally it needs a companion device to work, but shell apps can use the testing API to create a fake one: https://cs.android.com/android/platform/superproject/main/+/main:cts/tests/tests/companion/common/src/android/companion/cts/common/AppHelper.kt;l=29;drc=831c3ced50d5574cd5645afc0277bc6c59e83278 I was particularly interested in the virtual camera injection feature, but it's hide behind a flag that's disabled by default. |
Allow capturing virtual displays at a lower resolution using -m/--max-size. In the original implementation in #5370, the virtual display size was necessarily the same as the capture size. The --max-size value was only allowed to determine the virtual display size when no explicit size was provided. Since the dpi was scaled down accordingly, it is often better to create a virtual display at the target capture size directly. However, not everything is rendered according to the virtual display DPI. For example, a page in Firefox is rendered too big on small virtual displays. Thus, it makes sense to be able create a virtual display at a given size, and capture it at a lower resolution with --max-size. This is now possible using OpenGL filters. Therefore, change the behavior of --max-size for virtual displays: - it does not impact --new-display without size argument anymore (the virtual display size is the main display size); - it is used to limit the capture size (whether an explicit size is provided or not). This new behavior is consistent with main display capture. Refs #5370 comment <#5370 (comment)> Refs <#5370>
|
I changed the behavior of |
Allow capturing virtual displays at a lower resolution using -m/--max-size. In the original implementation in #5370, the virtual display size was necessarily the same as the capture size. The --max-size value was only allowed to determine the virtual display size when no explicit size was provided. Since the dpi was scaled down accordingly, it is often better to create a virtual display at the target capture size directly. However, not everything is rendered according to the virtual display DPI. For example, a page in Firefox is rendered too big on small virtual displays. Thus, it makes sense to be able create a virtual display at a given size, and capture it at a lower resolution with --max-size. This is now possible using OpenGL filters. Therefore, change the behavior of --max-size for virtual displays: - --max-size does not impact --new-display without size argument anymore (the virtual display size is the main display size); - it is used to limit the capture size (whether an explicit size is provided or not). This new behavior is consistent with main display capture. Refs #5370 comment <#5370 (comment)> Refs #5370 <#5370> PR #5506 <#5506>
|
I wonder why the virtual display automatic resize feature was dropped Controller.java#217 would be cool to have apps streaming with resizeable windows |
|
@Xtr126 The work I did to implement virtual display i based on previous work, as mentioned in the post:
In particular, the prototype by @anirudhb attempted to support resizing: https://github.com/anirudhb/scrcpy/commits/virtual-display/ But there is more work need it to make it work properly (and this brings new problems to solve), and virtual displays was already a lot of work, so I preferred implementing a non-resizable version first. Making it resizable could be done in the future. |
When mirroring a secondary display, touch and scroll events must be sent to the mirroring virtual display id (with coordinates relative to the virtual display size), rather than to the original display (with coordinates relative to the original display size). This behavior, introduced by d193967, was also applied for the main display for consistency. However, this mechanism has been found to cause some UI elements to become unclickable. To minimize inconveniences, restore the previous behavior when mirroring the main display: send all events to the original display id (0) with coordinates relative to the original display size. Fixes #5545 <#5545> Fixes #5605 <#5605> Refs #4598 <#4598> Refs #5137 <#5137> Refs #5370 <#5370>
When mirroring a secondary display, touch and scroll events must be sent to the mirroring virtual display id (with coordinates relative to the virtual display size), rather than to the original display (with coordinates relative to the original display size). This behavior, introduced by d193967, was also applied for the main display for consistency. However, it has been found to cause some UI elements to become unclickable. To minimize inconveniences, restore the previous behavior when mirroring the main display: send all events to the original display id (0) with coordinates relative to the original display size. Fixes #5545 <#5545> Fixes #5605 <#5605> Refs #4598 <#4598> Refs #5137 <#5137> Refs #5370 <#5370>
When mirroring a secondary display, touch and scroll events must be sent to the mirroring virtual display id (with coordinates relative to the virtual display size), rather than to the original display (with coordinates relative to the original display size). This behavior, introduced by d193967, was also applied for the main display for consistency. However, it causes some UI elements to become unclickable. To minimize inconveniences, restore the previous behavior when mirroring the main display: send all events to the original display id (0) with coordinates relative to the original display size. Fixes #5545 <#5545> Fixes #5605 <#5605> Fixes #5616 <#5616> Refs #4598 <#4598> Refs #5137 <#5137> Refs #5370 <#5370> PR #5614 <#5614>
|
I'm getting a issue, when the phone goes to lock screen i lose control(keyboard and left&right buttons, if a home button is pressed it is work, so just input actions related to lock screen works.) of virtual display. |
|
@mauriciofelippe On Samsung phones, clicks stop working on virtual display when the device goes to sleep apparently: #5557 (comment) |
This PR adds the possibility to start a new virtual display instead of mirroring the screen.
Implementation
There are 3 steps:
Refactors
The (7) firsts commits refactor to move code from
DevicetoControllerandScreenCapture, to prepare the next commits.Events handling
The commit
Inject display-related events to virtual displayaims to fix #4598 and #5137 (based on the work by @mengyanshou in #5137 and #5214), also necessary for making virtual display events work.Basically, for event injection, there are two display ids:
--display-id=…, default to 0 for the main display)(In case the ScreenCapture uses the "SurfaceControl API", then both ids are equals, but this is an implementation detail.)
In order to make events work correctly in all cases:
Virtual display
Then, based on the discussions in #1887 and the work by @yume-chan and @anirudhb:
The commit
Add virtual display featureimplements a first version of virtual display support:The size is fixed (not resizable).
On a Pixel 8, the virtual display contains a menu, so it is possible to open an app easily.
On other devices, it may be empty. In that case, you can start your app manually. Find the new display id is scrcpy logs (let's say it's 42), and start an app manually:
To start an app manually:
TODO
Start app
The feature really needs a new command to start an app, for example:
# not implemented, just an idea scrcpy --new-display --start-app=firefoxI might need to rework how arguments are passed to the server (to start an app "My gallery" for example), because currently it does not support spaces or other special characters (see bec3321).
I don't know if the expected behavior is to just have a virtual display and open any app (like on Pixel 8, where there is a "launcher"), or expose something like a "single app" feature.
It probably depends on the device (I don't know if it's possible to limit a single app on Pixel 8 for example, or to navigate to other apps on device without a launcher). What do you think? (also ref discussion #1887 (comment) @4nric)
Detect failure
If I run
scrcpy --new-displayon a Nexus 5 with Android 6, it does not fail, it just waits indefinitely for a frame. How to detect that "it works"? Should we only use checks based on the Android version?Draft v2
Start app
Here is a draft v2, with new options to list and start apps (edit: changed by #5370 (comment)):
There is a performance issue on some devices to retrieve the app names (but not package names), because
loadLabel()must be called for each app). As a consequence,--list-appsmay take several seconds, and selecting an application by name may also take time. However, starting via package name is quick. Any solution welcome.