Multiple bugs with Android immersive mode, especially after pause/resume

[hackalog]

p4a 0.5.3, kivy 1.10.0)
Reproducer code: https://gist.github.com/hackalog/c02703f78f09cf606cf5c1f32c074f23

How to reproduce:

• build the apk and install it to the device:

p4a apk --name=immersive --private . --version=1 --requirements=hostpython2,kivy,android \
  --orientation=landscape --package=bug.immersive

• Start the app. Tap "refresh"
• BUG 1 Tap "fullscreen". Screen will shift so the top is offscreen; i.e. pos (0,0) is not shifted down to where the android soft buttons used to be. updating the canvas (by tapping any button, or pressing the "update canvas" button, which explicity does an ask_update() will fix this.
• Leave the app by swiping up from the bottom and hitting the circle button.
• relaunch the app
• Note, android soft menu bar has reappeared. App is positioned under the menu bar (BUG 2)
  *Press "fullscreen". Soft menu will disappear, but note app window is not full-sized (black space at top, resolution wrong) BUG 3
• Worse, notice that your touch events on the buttons are offset by the size of the black bar (pos is out of sync with displayed graphics)

This was reproduced on a Google Pixel C tablet with android 6.0.1 (Build MXC89L)

[hackalog]

Possibly related: kivy/kivy#2856

[inclement]

This is also probably related to a problem with rotation - as I remember, if you start in portrait mode, enable rotation, rotate to landscape, pause and reopen the app, the app remains landscape but the kivy 'window' is portrait, and half off the screen. There are probably other similar variant problems as well.

[LarsDu]

Has anyone found a solution to this problem? I'm getting the same issue at the moment. My app isn't going to fullscreen on_resume and there's a big black bar at the top.

[LarsDu]

Here's my hack solution: Manually set the android window size to the kivy window size:
activity.getWindow().setLayout(Window.width,Window.height)

You may need to increase the height a bit.

[IgorCode]

@LarsDu
I had to set different height every time so your hack would work, when I say work I mean draw on entire screen without any black bars. But I had some other issues like battery and signal being drawn over my app.

[LarsDu]

I noticed that the hack I listed would mostly not work unless you changed the command to something like

activity.getWindow().setLayout(Window.width,Window.height-32)

or something so that the new window height isn't exactly the same as the kivy window height (or maybe not the same as a standard android cell phone window???)

This leaves small black bar gaps on the top and bottom of the screen, and it also doesn't work 100% of the time when coming back from waking up on suspend mode.

If anyone else has run into this problem, I would be so grateful for a more permanent/substantial solution.

[hackalog]

I have a hunch (and nothing to back this up), that size/pos changes in the root window aren't making it to kivy. Something probably guesses - wrongly - that the root window size/pos doesn't change on a mobile device. But clearly, it does when you rotate, or enable/disable immersive mode.

This is on my long list of issues to track down. I wonder if it can be reproduced with just SDL code?

[LarsDu]

I believe doing something like setLayout(1920,960) explicitly doesn't fix the black bar problem, so it's definitely not something to do with Kivy Window not getting the right height, but I'll try this in my app if that does not work.

[hackalog]

So perhaps kivy isn't expecting the position of the root window (with respect to the openGL coordinates) to change?

[RobertFlatt]

@hackalog its seems you may have some interest in this, here is a simple example and some opinion.

After on_start() the black bar is at the bottom; after on_resume() the black bar is at the top. The existence of the black bar is the issue, it's location is an artifact that will disappear when it gets the size is right.

Moving self.set_background_color() to the end of on_start() does not change the issue. So Window.size is unchanged after changing the app window size. I suggest this is the issue.

Looked at this way, Window.size is not static. And Kivy may not know Window.size till the end of on_start()/on_resume(). And I'd guess Kivy gets app Window.size from Android as in general Kivy could not know the size of the Navigation bar on some device.

Perhaps the issue can be addressed by re-making that Android call returning Window.size (explicitly or implicitly) at the end of on_start()/on_resume(). The BIG assumption here is that Kivy evaluation order does not assume Window.size is static - and I have no clue on that!

Note: on the latest p4a this requires the workaround to Issue #1504 else p4a crashes.

Edit: I also tried putting set_android_immersive_sticky() in build() and removing on_start() and on_resume(). The behavior was the same as with on_start() and on_resume(). To me this suggests p4a gets the device size before build().

from kivy.app import App
from kivy.uix.label import Label
from kivy.graphics import Color, Rectangle
from kivy.core.window import Window
try:
    from jnius import autoclass
    from android.runnable import run_on_ui_thread
except:
    def run_on_ui_thread(f):
        def f2(*args, **kwargs):
            pass
        return f2

class Demo(App):

    @run_on_ui_thread
    def set_android_immersive_sticky(self):
        AndroidView = autoclass('android.view.View')
        AndroidPythonActivity = autoclass('org.kivy.android.PythonActivity')
        view = AndroidPythonActivity.mActivity.getWindow().getDecorView()
        view.setSystemUiVisibility(
            AndroidView.SYSTEM_UI_FLAG_LAYOUT_STABLE |
            AndroidView.SYSTEM_UI_FLAG_LAYOUT_HIDE_NAVIGATION |
            AndroidView.SYSTEM_UI_FLAG_LAYOUT_FULLSCREEN |
            AndroidView.SYSTEM_UI_FLAG_HIDE_NAVIGATION |
            AndroidView.SYSTEM_UI_FLAG_FULLSCREEN |
            AndroidView.SYSTEM_UI_FLAG_IMMERSIVE_STICKY)

    
    # Kivy default background color is same color as Android Navigation Bar
    # Change Label background so it is easier to see what is happening
    def set_background_color(self):
        with self.label.canvas.before:
            Color(0.5, 0.0, 0.5)
            Rectangle(size=Window.size)  # BUT THIS IS NOT THE size WE GET :(

    def on_start(self):
        self.set_android_immersive_sticky()
        
    def on_resume(self):
        self.set_android_immersive_sticky()

    def build(self):
        self.label = Label(text = 'Greetings Earthlings')
        self.set_background_color()
        return self.label

if __name__ == '__main__':
    Demo().run()

.p4a

--dist_name=ex
--private .
--package=com.example.ex
--name Ex
--requirements=python3,kivy,android
--arch=armeabi-v7a
--sdk_dir /home/me/androidtools/sdk
--ndk_dir /home/me/androidtools/android-ndk-r17c
--ndk_version 17c
--android_api 28
--ndk-api 21
--version 0.0.1

[ghost]

This sounds like it could possibly be resolved by a simple update to latest SDL2 and SDLActivity. If I had to guess, that's where I would assume the bug is (in SDLActivity/the java wrapper, specifically). I'm still busy with a few other pull requests but plan to tackle this soon (in case nobody else has picked it up yet once I get to it), since this will also solve a lot of other issues like missing key events for certain physical keys, CTRL not working on physical keyboards and other problems that appear to be resolved in later SDL2 versions for Android

[hackalog]

Heh, if I remember the last time we tried to update to latest SDL2, the word "simple" might be a misnomer. But yes, an update to newer SDL would be fantastic for a number of reasons (including this one: kivy/kivy-ios#274).

@inclement The rotation-related bugs are a definite issue, too (both in p4a and in kivy-ios). I'd like to see if updating SDL helps there. If not, I'll another issue. We have an easy reproducer, at least on the IOS side)

[RobertFlatt]

I drilled down into this and there are two separate issues: one at first 'UI mode' call, and one at on_resume()

1. At first 'UI mode' call a black bar appears at the bottom of the screen.

This is due to a missing resize handler in the Kivy code. From Kivy's point of view a 'UI mode' change is a screen resize. We get a black bar because the app is not written to respond to UI change.

If the app works on a desktop 'window resize' or on a portable 'sensor rotate', these are cases where there is already a resize handler and this problem should (!) not appear.

The app need something of the form:

def _resize_handler(self,obj,size):
        Window.canvas.ask_update()  # Or whatever

def build():
       Window.bind(size=self._resize_handler)
       # and so on...

2. At on_resume()

If an immersive 'UI mode' is called in build() or on_start(), then SDL2 generates a 'windowresize' event after on_start() and another after on_resume(). These events then set the Kivy Window.size.

The on_start() 'windowresize' contains the immersive window size values. But the on_resume() 'windowresize' contains the NON-immersive window size values. After on_resume Kivy then paints a smaller window, and Android keeps the Navigation Bar hidden. Giving rise to the black bar.

To me this looks like an SDL2 issue. The on_resume event is redundant as Android does not have a state change. Kivy does not have a state change till it receives this misleading event. The resulting Kivy state change is bogus.

No idea how to workaround at this time.

[ghost]

I'm currently looking into an SDL2 version bump: #1528 pull request will not work/build yet as-is, but I'll let you know when it does (should in 1-2 days, got it working locally but I'm still waiting on other merge requests). You could then try it out and see if it helps with anything. I'm not familiar enough with immersive mode myself to be able to tell for sure, I can't see anything obviously wrong in my test app but I might have misunderstood what the issue is

[RobertFlatt]

When you are ready I'd be happy to try the latest... Here is my current thinking.....

It seems SDL2 always (?) calls setSystemUiVisibility() [see SDLActivity.java] at start time and at on_resume()

A second setSystemUiVisibility() call from Kivy gives additional resize events, these are filtered for a recursion case (I think) so these event dont get back to Kivy hence the black bars. Tried lots of workarounds - no cigar.

Not clear this is an error, but an explicit setSystemUiVisibility() call seems to require explicit disabling of the implicit setSystemUiVisibility() call. Which would be an enhancement request, and might be an SDL2 thing, I can't tell yet.

Trying a different approach. What I want is Android Immersive_stickey mode. In SDLAactavity.java the call setSystemUiVisibility('IMMERSIVE_STICKEY') exists, and is enabled by the 'fullscreen' option. So try 'fullscreen' as shown in the example below.

The 'fullscreen' option works on Windows but not on Android.

From Kivy's point of view Android 'immersive' is fullscreen, so its seems like is 'should' work; and it looks like the code is in SDL2.

In kivy/core/window/window_sdl2.py I can trace fullscreen into _window_sdl2.so but can't find the source code which presumably is in kivy.deps.sdl2 wherever that is located.

Any suggestions on how to proceed?

from kivy.app import App
from kivy.uix.label import Label
from kivy.graphics import Color, Rectangle
from kivy.core.window import Window
from kivy.config import Config

class Demo(App):
    def set_background_color(self):
        with self.label.canvas.before:
            Color(0.5, 0.0, 0.5)
            Rectangle(size=Window.size) 
       
    def on_start(self):
        # Both of these work on Windows but not on Android
        # seems they should, see SDLActivity.java COMMAND_CHANGE_WINDOW_STYLE
        Window.fullscreen = True    
        #Config.set('graphics', 'fullscreen', 'True')

    def _resize_handler(self,obj,size):
        self.set_background_color()

    def build(self):
        Window.bind(size=self._resize_handler)
        self.label = Label(text = 'Greetings Earthlings')
        self.set_background_color()
        return self.label

if __name__ == '__main__':
    Demo().run()

[inclement]

I wonder if this is still happening since the SDL2.0.9 upgrade? That definitely had implications for some fullscreen behaviour.

[hackalog]

I should be able to try it... I'll try building this eve

[RobertFlatt]

I planned to try sdl2==2.0.9 but found this is now the default with python-for-android
master installed 2/2/2019

Two issues (pretty basic so I wonder about my sanity):

1. HelloWorld defaults to fullscreen
   Unable to change to traditional Kivy style with either of these:

   Window.fullscreen = False
   Config.set('graphics', 'fullscreen', 'False')

2. --window correctly enables the status bar
   but incorrectly enables the navigation bar

3. --orientation sensor is ignored, defaults to portrait
   --orientation landscape is ignored, defaults to portrait

This message was in the log file:

WARNING: A problem occurred while running pkg-config --libs --cflags sdl2 SDL2_ttf SDL2_image SDL2_mixer (code 1)

b"Package sdl2 was not found in the pkg-config search path.\nPerhaps you should add the directory containing `sdl2.pc'\nto the PKG_CONFIG_PATH environment variable\nNo package 'sdl2' found\nPackage SDL2_ttf was not found in the pkg-config search path.\nPerhaps you should add the directory containing `SDL2_ttf.pc'\nto the PKG_CONFIG_PATH environment variable\nNo package 'SDL2_ttf' found\nPackage SDL2_image was not found in the pkg-config search path.\nPerhaps you should add the directory containing `SDL2_image.pc'\nto the PKG_CONFIG_PATH environment variable\nNo package 'SDL2_image' found\nPackage SDL2_mixer was not found in the pkg-config search path.\nPerhaps you should add the directory containing `SDL2_mixer.pc'\nto the PKG_CONFIG_PATH environment variable\nNo package 'SDL2_mixer' found\n"

[ghost]

@Ham-Merhead can you try without the version pin? It should be possible to pin it like that but maybe something's broken about it and the recipe isn't recognized. Can you see in the output whether it's built as a recipe module or pure python one?

@ two issues: things like that are expected if you run kivy master on pre-2.0.9 p4a (with old SDL2), or you run kivy stable on 2.0.9 master p4a. pretty sure they only work correctly if you upgrade both (or none)

[ghost]

Also, shouldn't --window enable both status & navigation bar? (while omitting it should disable both) Or did I miss something

[RobertFlatt]

@Jonast ..."can you try without the version pin?"...
I'm using python-for-android installed 2/2/2019 Without any versioning I get 2.0.9
Based in your comment I think I want master for both p4a and kivy.
Three cases (tell me if I'm not answering your questions):

Case 1) If I specify --requirements=python3,kivy
I get
[INFO]: Downloading sdl2 from https://www.libsdl.org/release/SDL2-2.0.9.tar.gz--requirements=python3,kivy

Case 2) if I change to --requirements=python3,kivy==master
[INFO]: Recipe kivy: version "master" requested

The behavior is the same in both the first two cases:
A) default fullscreen which is different from before, which would be OK if I could get to the legacy behavior.

B) --orientation is disabled.
if I set --orientation sensor and start the app with the phone physically oriented landscape, the "Loading..." message starts in landscape and flips to portrait without moving the phone. That looks like an implicit setting of portrait, which does not match the build option.

C) In the docs https://python-for-android.readthedocs.io/en/latest/buildoptions/
"--window: If the argument is included, the application will not cover the Android status bar."
Which does not tell us anything about the navigation bar.
For me the issue is there is no way to get to the the legacy 'navigation bar only' case which was the default.

Case 3)
p4a clean_all then p4a apk with --requirements=python3,kivy==master
I get a load time crash:

02-05 14:51:01.264 16675 16700 I python  : 
02-05 14:51:01.265 16675 16700 I python  :  Traceback (most recent call last):
02-05 14:51:01.265 16675 16700 I python  :    File "/mnt/c/users/bobf/documents/pm/tst/main.py", line 31, in <module>
02-05 14:51:01.266 16675 16700 I python  :    File "/home/bobf/.local/share/python-for-android/build/python-installs/ex/kivy/app.py", line 829, in run
02-05 14:51:01.266 16675 16700 I python  :    File "/mnt/c/users/bobf/documents/pm/tst/main.py", line 25, in build
02-05 14:51:01.266 16675 16700 I python  :  AttributeError: 'NoneType' object has no attribute 'bind'
02-05 14:51:01.266 16675 16700 I python  : Python for android ended.

the snippet of main.py is (size is NoneType) !

    def _resize_handler(self,obj,size):
        self.set_background_color()

    def build(self):
        Window.bind(size=self._resize_handler)     ####<<<< Line 25
        self.label = Label(text = 'Greetings Earthlings')     
        self.set_background_color()
        return self.label

[ghost]

p4a clean_all then p4a apk with --requirements=python3,kivy==master

That is the approach I'd recommend, yes. (if you're using p4a master) If that crashes, can you file a bug here?

For me the issue is there is no way to get to the the legacy 'navigation bar only' case which was the default.

Independently of the crash, I honestly think this is intended. Fullscreen is with status bar and navigation hidden (VARIANT 2 BELOW), windowed is with both visible (VARIANT 1 BELOW). The "mixed" variant I'm pretty sure was simply a bug, and I don't think we ever had a proper option for this...?

Maybe it could be added as a feature, but given the settings right now there is no good way to specify it

[RobertFlatt]

(if you're using p4a master)

yes

If that crashes, can you file a bug

yes .
But 'heads up' Window.size is set from SDL2 ...
Perhaps there is some other version dependency that kicks in after the clean_all

The "mixed" variant I'm pretty sure was simply a bug,

perhaps it is a legacy from when the navigation bar was physical buttons?

but given the settings right now there is no good way to specify it

yes I see

What about --orientation , that seems to be a SDL2 thing..?

Btw Google is quite good on the current documentation of display modes:
https://developer.android.com/training/system-ui/immersive

[ghost]

@Ham-Merhead I can assure you p4a master returns window size just fine, so this is 99% not an SDL2 issue (I use an alternative to kivy where it works fine)

@ google: yes I'm aware of the various modes, p4a just simply only supports windowed (ALL bars visible) and sticky immersive (everything gone as much as possible) historically. I know there are more, but p4a has so far never offered any options to access the others 🤷‍♂️ if this is a common request we could possibly add it, it's not impossible or challenging per se it's just that nobody has added options for that so far

[RobertFlatt]

IMHO Those other modes would be really cool. The same as Java, that would keep the marketing folks happy. Oh, wait, we don't .....

--orientation sensor
While retesting the Kivy issue before posting an issue (its Window not Window.size that is not defined)
I noticed the following in the log file.
Unrelated to the Kivy thing, it at least explains how I get stuck in portrait if not why:

02-06 13:02:06.770 26560 26584 I python  : [WARNING] Could not satisfy Android orientation "LandscapeLeft LandscapeRight", only {portrait,landscape} are currently supported. Defaulting to portrait
02-06 13:02:06.774 26560 26584 E libEGL  : validate_display:92 error 3008 (EGL_BAD_DISPLAY)
02-06 13:02:06.774 26560 26584 V SDL     : setOrientation() orientation=1 width=1080 height=2160 resizable=true hint=Portrait
02-06 13:02:06.831 26560 26560 V SDL     : surfaceChanged()
02-06 13:02:06.831 26560 26560 V SDL     : pixel format RGB_565
02-06 13:02:06.832 26560 26560 V SDL     : Window size: 1080x2160
02-06 13:02:06.832 26560 26560 V SDL     : Device size: 1080x2160
02-06 13:02:06.834 26560 26584 I python  : [INFO   ] [GL          ] Using the "OpenGL ES 2" graphics system

I thought LandscapeLeft and LandscapeRight were an iOS thing. But maybe not only iOS.
I'll try to look at it a little more, and maybe post an issue.

[RobertFlatt]

Happy, happy, joy, joy Immersive rotates just fine.

I know there are more, but p4a has so far never offered any options to access the others ..... it's not impossible or challenging per se

Not challenging for whom? ;)

I can call Android/Java .setSystemUiVisibility() and change the UI options. This works.

But the UI is immediately reset to the default after a single touch event.
I can imagine why this would be; digging around it seems SDL2 does an internal restart in a lot of cases,
and this calls setSystemUiVisibility() . Wiping out my call.

I could write a listener in Java, but presumably that would keep firing. A not a good idea.

Any suggestions?

[ghost]

Oh right, I forgot SDL2 doesn't support these modes. Nevertheless it should be doable, you need to add things to our SDLActivity.java.patch to change this particular function/line:

https://github.com/kivy/python-for-android/blob/master/pythonforandroid/bootstraps/sdl2/build/src/main/java/org/libsdl/app/SDLActivity.java#L1344

(That is where SDL2 triggers this UI fiddling that you are seeing, in an attempt to ensure Android doesn't slide back in the bars in sticky immersive which it likes to do)

[RobertFlatt]

Was looking for something a little more user friendly. Continuing that quest:

A more basic question, how to switch between the two modes we now have programmatically?
I can read and set Window.fullscreen andConfig('graphics', 'fullscreen') [both default to 'auto']
But neither changes either of the states defined by --window

my assumption had been that --window == not fullscreen

Clearly --window works, but I can't find the 'window' option in toolchain.py and p4a apk --help doesn't mention it; so I'm having trouble following the path.

?

[ghost]

I can read and set Window.fullscreen andConfig('graphics', 'fullscreen') [both default to 'auto']

The default is 'auto' is because you are supposed to use --window. Is there a specific reason why you would need to be able to control this programmatically?

Clearly --window works

Yes, which is why I recommend you to use that 🙂 and leave 'auto'

(As a side note, per design programmatically cannot really be done right now at least without a very visible delayed switch, because the loading screen comes up before your program runs and doesn't know what your program will decide upon, so it is impossible to launch it programmatically from the start in the right mode)

I can't find the 'window' option in toolchain.py

The --window option is currently defined here:

python-for-android/pythonforandroid/bootstraps/common/build/build.py

Line 615 in 339907a

ap.add_argument('--window', dest='window', action='store_true',

[ghost]

Another side-note: programmatically changing the mode from my tests appears to work, through SDL_SetWindowFullscreen - but doing this would be entirely up to kivy, so if there is no obvious way how to do such a later switch to a different mode, this likely would need to be addressed in kivy (and I recommend you file a bug there if for some reason you really need this behavior of programmatically switching away from whatever was the launch mode)

[RobertFlatt]

Getting wiser slowly:

--window is used to set P4A_IS_WINDOWED here:

python-for-android/pythonforandroid/bootstraps/common/build/build.py

Line 300 in 339907a

f.write("P4A_IS_WINDOWED=" + str(args.window) + "\n")

P4A_IS_WINDOWED is used in Kivy here
https://github.com/kivy/kivy/blob/master/kivy/core/window/_window_sdl2.pyx#L81

Where it says:

# Android is handled separately because it is important to create the window with
# the same fullscreen setting as AndroidManifest.xml.

Which means:

1. No point is asking the Kivy folks because of AndroidManifest
2. User UI configuration would have to be done in a fullscreen context via .setSystemUiVisibility() [or?] . Which means convincing SDL2 not to keep resetting to immersive_sticky.

Back to square one. But not giving up.....

[ghost]

I'm not sure that code comment is correct for SDL_SetWindowFullscreen used afterwards. I would need to test more to be sure but toggling appeared to work fine when I tested it (in a non-kivy app)

Regarding "convincing SDL2", for that you need to modify the line of SDLActivity.java that I quoted

(similarly to p4a and kivy, SDL2 only knows fullscreen and windowed and not the in-between you want as far as I'm aware, but it adapts to surface size changes properly. So if you prevent it from enforcing the fullscreen by hacking around in the java code it will probably work fine if you set whatever different style)

[RobertFlatt]

FYI I tried this in _window_sdl2.pyx (yes, propagated to .c and .so)

              #if environ.get('P4A_IS_WINDOWED', 'True') == 'False':
               if fullscreen is True:

I tried --window present and absent, and Window.fullscreen=True/False. I tried Window.fullscreen in build() and on_start()

I always get a windowed screen. Which in retrospect I can rationalize.