CommandBufferGL::drawElements Crash #1211
Comments
At risk of stating the obvious, have you actually attempted to use a debugger or other methods of tracking down why it crashes? A breakpoint in the right place may be the only thing you need to find out the issue. You've provided no source code, no context to the crash, whether it's on a real device or emulator, or what the app was doing at the time. It's something in your own code causing this issue, and unlikely given to be something in the game engine, and given that you ported it from what I can only assume is a working version of the app, then something like this must be trivial for you to fix. |
|
Hey @rh101 The issue is not reproducible in any of our test devices. Crash logs are from Google Play live users. 3.6% of our GPLAY users have the issue. |
|
Another similar crash: [split_config.armeabi_v7a.apk!libMatchAnimal.so] CommandBufferGL.cpp - ax::backend::CommandBufferGL::drawArrays(ax::backend::PrimitiveType, unsigned int, unsigned int, bool) date_range Version: 10205 (1.2.5)Occurred: 3 hours ago pid: 0, tid: 10195 >>> games.spearmint.matchanimal <<< backtrace: |
Is it limited to specific devices or Android versions, and if so, what devices and Android versions are having this issue? I notice the Also, please put code tags around the on the crash dumps so they're formatted and easier to read. Just add 3 backtick ` symbols before and after the crash dumps. |
|
@rh101 Issue also in arm64_v8 devices. [split_config.arm64_v8a.apk!libMatchAnimal.so] CommandBufferGL.cpp - ax::backend::CommandBufferGL::drawElements(ax::backend::PrimitiveType, ax::backend::IndexFormat, unsigned long, unsigned long, bool) |
|
@kiranb47 Crashes that affect a small subset of users may have very specific triggers, and since you are the one with access to the source code, you're in the best position to narrow down the possible causes. Do you collect any form of diagnostic information from your application? Can you tell what a user is doing in the app just prior to the crash? |
|
According to provided logs, all your issues are related to gl buffers. The falt occurred inside gldrawelements, gldrawarrays and while buffer creating. For example, axmol supports int as base type for index array as opposed to cocos. I would check how your buffers creating in your code at first, since the issues looks logical. You created them manually? |
|
Hey @rh101 Firebase is not reporting this issue, so not able to identify pre crash events logs. Hey @solan-solan I have no technical knowledge about OpenGL. We have not applied any changes to axmol source code for this game. |
|
@rh101 @solan-solan Issue is related to UI ScrollView |
|
It's great that you have managed to narrow down the issue. Can you show the snippet of code related to how you use the |
|
FYI. I migrated my app from cocos2d-x 3.17 to latest axmol (4e664e6) and see in Google Play similar crashes in drawElements and drawArrays and In my game I use ui::ScrollView as well. I used many (100-1000) Sprite and Scale9Sprite. App with cocos2d-x 3.17 has ~0 crashes after upgrading I see significant increase. I can not reproduce issue on local devices ... Please check below screenshot how much crashes increased in Google Play after update: |
|
is your project open source? I can debug and potentially fix this for you if so. otherwise why not compile a small project that showcases the issue like @rh101 said? |
|
@DelinWorks Thank you for an offer. The problem is that I can not reproduce problem locally ... I see crashes only in Google Play. I am so sorry my project is not open source. That could mean that _indexBuffer is incorrect or parameters to glDrawElements are incorrect. Unfortunately I am not familiar with GL. Do you think that it would be possible having above callstack analyze existing code and narrow problem through code inspection? |
|
You were able to produce it on your device or is that a google analysis ghost crash |
|
if google analysis could show you which population of devices it crashes on it would be helpful |
|
@DelinWorks I can not reproduce crash on my device. I see crash only in Google Play console. FYI. This game is was updated 3 days ago and only ~2K users updated version. |
|
unfortunately you'll need to get your hands on one of those devices to test it, or try to roughly match the specs, os version, and sdk of the devices to hopefully regenerate the bug. |
|
Is there any chance you could create a debug version of your applications, and upload them to either the "Internal testing" track in your Google Play Console? I can provide you with a gmail address to add to the internal tester list, and I suggest others here provide an email address too if they can help (best to generate a new gmail address for this purpose, and not use your personal address). Once you upload the APK to that internal testing track, you can share the link to it so that the users on your testers list may download it. This issue has me a little concerned, and I'm going to try to get my hands on at least one of the specific devices that seem to be having this issue. Also, describe exactly what it is that needs to be done in the application to cause it to crash (like which screen to go to, what exactly to click on etc). Also, @crazyhappygame which of the devices that you listed have the most crashes? Another thing, we really need a sample of the |
May be this is some type of memory leack? Do you sure that gl buffers and internal arrays of your sprites properly free? And is there opportunity to analysis how many time user spent in game before crashing? |
|
@solan-solan I used just Sprite and Scale9Sprite with image. No custom tricks, shaders etc. Not sure how to get information how much time user spend in the app before crash.... What I have seen in stacktrace that it crashes in memcpy. For me that means that there is no problem with allocation but with access to unavailable memory. @rh101 @DelinWorks @kiranb47 do you know how to build entire (all 3rdparty libs, axmol, gsme) axmol with address sanitizer? https://developer.android.com/ndk/guides/asan#cmake |
|
@rh101 could you send me instructions how to build debug version of app and Gmail address? (Gradle assembleDebug)? |
If you're using Android Studio, then just use the menus to configure it: and this window pops up: There may be a way to do it from the command line too, but I can't recall it at the moment, so perhaps someone else knows how to do it. EDIT: email removed. @crazyhappygame if you need it again, I'll post it up. |
Are the instructions on that page not working for you? |
|
Below have to be set for all targets including all 3rdparty libs I do not know how to rebuild all android deps |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
@battulasaivinesh Does your application have any full-screen video ads (using an ad SDK etc.) or similar that would cause the app to be put into the background then back to the foreground on completion? Also, what percentage of users/devices are experiencing this specific crash? |
|
Yes, we do have rewarded video ads at multiple places in-game. We use Google Ads SDK for the same. Currently, we just rolled out axmol engine change to 100% yesterday. Out of the total crashes happening with this release (Around 7k in the last 24 hrs), drawElements crashes take almost 25% (2k). In terms of total users though, this crash is happening for less the 0.5% of users. |
maybe somebody can change to test |
|
Another potential issue is that axmol/core/renderer/backend/opengl/CommandBufferGL.cpp Lines 327 to 351 in c071395 |
The reason I asked is because this issue has existed since Cocos2d-x v4 came out, as that's when the OpenGL renderer changed significantly from Cocos2d-x v3.17. This issue was posted about here on the Cocos forums: https://discuss.cocos2d-x.org/t/commandbuffergl-related-crash/50293/14 Check if any info there regarding full-screen ads helps at all, because it was the cause of that specific developer's problem. |
That is a problem that needs to be fixed, whether or not it is the cause of the crash described in this thread. |
|
@rh101 Thanks for the link. All the callbacks from ads sdk are wrapped around performFunctionInCocosThread, so not the cause of the crash. We have a release tomorrow, should I try adding back glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, 0);? |
Fair enough.
That's entirely up to you. |
|
As suggested by @rarepixels here - #1211 (comment), simply setting "Preserve EGL context" to false (this.mGLSurfaceView.setPreserveEGLContextOnPause(false)) and commenting out //super.onPause(); completely eliminated all crashes related to drawing elements. I'm not entirely certain if this is the correct approach, but Cocos also implemented the same solution - cocos2d/cocos2d-x#19996. |
Calling One thing that would need to be looked into is any side-effect of commenting out |
|
@battulasaivinesh Looking more into the The other issue is that ``AxmolGLSurfaceView.onResume() The render was never actually paused, so that loop would still be running, and now we end up in another while loop every time the app is put into the background and then brought back to the foreground. I feel that the logic should be improved in the main activity, being
So, perhaps the code in AxmolActivity should be updated to this: So, essentially what happens is that if the app no longer has focus, Now, if the user has navigated away from the app, or closed it somehow, then If the app is brought back to the foreground, then I tested a few non-Axmol apps to see what happens when "switch apps" button is clicked (the square button). All the apps simply reduced in size, and lost focus, but they never stopped animating/running, so their render loops were in fact still operating ( I'm not sure if this would actually fix any issue, but the primary purpose of this change is to stop the renderer only when required, which is when the current app activity is replaced by another activity, and there is no point destroying the EGL context if the app never actually goes into the background. |
|
For starters it would be good to check and fix this issue: #1211 (comment). If no one checks it, I'll probably have some time to look into it this or next week. |
One additional thing regarding this apparent fix. It's not actually fixing the issue, but rather avoiding the issue. If Since Why it's no longer crashing is because the EGL context is never released, since the |
|
Just discovered a problem that may affect anyone who changes the enabled state of EventDispatcher in The event for That is assuming that Axmol has not been modified, meaning I'm curious to know if the developers experiencing the crash in their apps are actually disabling the event handler at any point when the application is paused. @battulasaivinesh @rarepixels @kiranb47 @crazyhappygame Can you please check if you have code that disables the event handler on an application pause, such as in You can easily check this by doing the following: Make sure In Put a breakpoint on Check if it is in fact correctly dispatching the event. Alternatively, put a breakpoint on any handler function for Now, navigate away from your application to another app so it goes into the background, then go back to the same application again. If it does not enter any handlers for |
|
It starts in the disabled state too, which might be already problematic: axmol/core/base/EventDispatcher.cpp Line 195 in eb0e6ef
Looks like these events should ignore disabled dispatcher:
I think |
That's actually interesting, because given the low percentage of affected users, it's most likely a very specific set of conditions that is triggering it. This scenario is most likely to cause a crash: 1 - event dispatcher is disabled (such as on entering a scene transition) I can reproduce this issue 100% of the time with a test case, by setting preserve EGL context to false with
Something does need to be done about this. |
With the suggested changes in my previous post above, the code would no longer be required, since |
|
Why not to enable event dispatcher forcibly inside Java_org_axmol_lib_AxmolRenderer_nativeInit before EVENT_RENDERER_RECREATED? |
That would require code to save the current state of the event dispatcher, enable it, then reset it to the previous state, and this would also be required on each and every event that must be sent out, such as Aside from create a better solution for these types of events, a quick fix would be to just add an optional parameter to all Checking through the rest of the code in the dispatcher, nothing checks for the This change should not impact existing usage of the dispatch methods. Does it seem like a reasonable change? |
lgtm, PR? |
Done, #1940 With this change, having It would be great if anyone experiencing the crash issue in their released application could test this out to verify if it in fact fixes the problem. All required events are now sent out correctly even if the event dispatcher is disabled. |
|
Hi all, I got all the new messages about this issue just now. I was away from my computer last weeks. I can tell that I set //super.onPause(); since months, and yes, as @rh101 pointed, this do not fix, but, instead avoid the problem. but for my two live apps, I can tell that I had no secondary effects at all, at least, no new crashes or ANR. I hope I will finish a new project soon, and then I will check how things are going with all the new changes on production. ( I still never saw this crash locally ) |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
It turns out there is an issue with this specific change, which is now fixed in #2162 Along with that fix, there have been many other fixes related to crashes due to the EGL context loss issue. The current dev branch (future 2.2 release) should have, hopefully, addressed the crash issues completely. |
Steps to Reproduce:
We have migrated our Match Animal game to Axmol from Cocos2dx V3 latest yesterday. Getting this crash now. How to fix this issue?
backtrace:
#00 pc 0x00000000005ffa58 /data/app/games.spearmint.matchanimal-ayOwq5oTYpeKYIRIPMWVUQ==/lib/arm64/libMatchAnimal.so (ax::Renderer::TriangleCommandBufferManager::createBuffer()+1681)
#1 pc 0x00000000005fdbc0 /data/app/games.spearmint.matchanimal-ayOwq5oTYpeKYIRIPMWVUQ==/lib/arm64/libMatchAnimal.so (ax::Renderer::init()+964)
#2 pc 0x000000000063b060 /data/app/games.spearmint.matchanimal-ayOwq5oTYpeKYIRIPMWVUQ==/lib/arm64/libMatchAnimal.so (ax::Director::setOpenGLView(ax::GLView*)+394)
#3 pc 0x00000000005640cc /data/app/games.spearmint.matchanimal-ayOwq5oTYpeKYIRIPMWVUQ==/lib/arm64/libMatchAnimal.so (Java_org_axmol_lib_AxmolRenderer_nativeInit+99)
#4 pc 0x00000000001c5ff0 /data/app/games.spearmint.matchanimal-ayOwq5oTYpeKYIRIPMWVUQ==/oat/arm64/base.odex
The text was updated successfully, but these errors were encountered: