FirebaseCrashlytics run vs upload-symbols -- missing optional dsyms

[dloic]

[REQUIRED] Step 1: Describe your environment

• Xcode version: 11.2.1
• Firebase SDK version: 6.17.0, 6.24.0
• Firebase Component: Crashlytics
• Component version: 6.17.0, 6.24.0
• Installation method: Carthage

[REQUIRED] Step 2: Describe the problem

We're seeing missing optional dsym for our apps since a month now and we can't explain why it's happening. As said in the Firebase console, it's only affecting result and grouping, but still we would like to understand why it's like this and what did broke our integration. We've seen this behavior on ad-hoc build without bitcode (we generate dsym) and also on store build (bitcode enabled, dsym pulled from ASC).

Steps to reproduce:

To reproduce the issue, I've run a test outside of our CI, to ensure the problem is not there.
Initial steps to ensure app is compiled and dsym are uploaded:

• Xcode clean
• Xcode build for device arch
• Xcode archive

As seen in screenshot below, it seems everything is fine during symbols upload:

In the app, we have a way from our debug menu to trigger a crash, to easily test our Crashlytics integration is working from release to release. After the build is installed on a device (ad-hoc distribution), I simply open the debug menu and crash the app. Result in Firebase console:

This is telling us we're missing the AA71654B-XXX dsym, with turns to be the one generated for Eureka third-party. Having a look within the Xcode archive and using dwarfdump, I can confirm this dsym has been properly generated:

I then manually upload-symbols. End result is now the console is able to properly symbolicate new crashes (not older ones):

Does it means that during build, upload was somehow not successful? Or we're missing something in our integration?
As for our full update path, we were using Fabric Crashlytics for a while, and we did migrate to FirebaseCrashlytics with Firebase 6.17.0 in Feb. I'm currently updating Firebase to latest (6.24.0), but it seems the issue is still there.

Thanks for your help!

[dloic]

Hi @TKBurner , any news/feedback regarding my issue?

[eric-chamberlain-ck]

We've encountered the same issue. It appears that upload-symbols with the —build-phase flag only uploads the dSYM for the build target, not all the dSYMs in DWARF_DSYM_FOLDER_PATH.

The workaround is to call it manually with something like ${PROJECT_DIR}/../scripts/FirebaseCrashlytics/upload-symbols -d -p ios -gsp ${BUILT_PRODUCTS_DIR}/${UNLOCALIZED_RESOURCES_FOLDER_PATH}/GoogleService-Info.plist ${DWARF_DSYM_FOLDER_PATH}

[dloic]

@eric-chamberlain-ck Thanks for the info. As said in my initial comment, it's also working for us if upload manually.

Seems there is an issue in upload-symbols binary, because as you said, looking at https://github.com/firebase/firebase-ios-sdk/blob/master/Crashlytics/run the only diff is --build-phase option which seems to be some kind of shortcut to avoid giving platform and path to service info plist. But maybe it's doing more than this?

Can someone from the Firebase team confirms this?

[TKBurner]

@samedson is going to take a look at this.

[samedson]

@dloic you're right: the --build-phase flag uses environment variables that Xcode defines during the build to determine the app's platform and path to the plist. But in addition, it is also able to get all the dSYM locations from those environment variables.

When using upload-symbols directly, it should take either a directory or a list of dSYM paths, so you'll want to pass all your dSYMs paths to upload-symbols separated by spaces.

[nickprkins]

@samedson so why is the build phase script not uploading these optional DSYMs? I have a ton of optional DSYMs that need to be uploaded. I tried downloading them from AppStoreConnect and from our CS and uploading them and it finds none of these optional DSYMs that Firebase says are missing. Something is not working properly with the way FirebaseCrashlytics wants these setup to upload. Losing confidence in this product.

[samedson]

Ah so let me clarify a few things about dSYMs in general.

• Usually your app has a main dSYM that's put in the Products directory. Crashlytics is usually able to gather this dSYM because it's clearly defined in Xcode's Build Phase environment variables, that upload-symbols reads from. This is the "Required" dSYM in Crashlytics.

  • When you download a dSYM from iTunesConnect, it will be this "Required" dSYM.

• When there are "Optional" dSYMs in Crashlytics, this usually indicates there are separate SDKs included in your app. There are a few possibilities on where to find these:

  • In some cases the dSYM gets generated for the SDK into the Products directory as well, in which case upload-symbols should find it.
  • Sometimes if the .framework folder for the SDK has the dSYM inside of it, it will need to be uploaded manually via upload-symbols separately. upload-symbols can have trouble finding these dSYMs because they are not usually referenced by Xcode's build environment variables.
  • Sometimes you have an SDK with stripped symbols and no .dSYM. In this situation there's now way to get symbolicated crashes for those lines of code because the SDK developer purposefully hid all symbol information.

The easiest way to find a missing dSYM file is to run:

mdfind "com_apple_xcode_dsym_uuids == UUID"

In general, Xcode determines if dSYMs are generated, and where they're put on the filesystem. Crashlytics has no control over this and won't modify Xcode projects.

[dloic]

Thanks for the info @samedson. As mentioned by others, we're definitely seeing a new behavior. Do you confirm?

Could it be related to the new Crashlytics SDK?

As mentioned by @nickprkins we're seeing the same issue for AppStore build, none of the dsym downloaded manually from ASC are matching the missing optional dsyms.

Could this be also related to Carthage? @eric-chamberlain-ck @nickprkins are you using Carthage?

[eric-chamberlain-ck]

We are using Carthage and Fastlane. All our dSYMs are in the same folder at the end of the build, using the —build-phase flag only uploads the app dSYM and not all the dSYMs in the folder.

We ended up moving the upload out of our build phases, manually specifying the flags for upload-symbols, and trigger the upload later in our CI flow.

[dloic]

@samedson I got no feedback from you. Should we do a hack and call upload-symbols, as described above? And thus no longer follow official documentation?

[samedson]

@dloic yeah for now I would manually call upload-symbols by passing all the dSYMs to it. I haven't yet had time to figure out what is going wrong.

[dloic]

Ok, thanks for your reply!

I guess we'll keep the issue open until you can have a look at it, so it can be helpful for others.

[nickprkins]

Thanks for the info @samedson. As mentioned by others, we're definitely seeing a new behavior. Do you confirm?

Could it be related to the new Crashlytics SDK?

As mentioned by @nickprkins we're seeing the same issue for AppStore build, none of the dsym downloaded manually from ASC are matching the missing optional dsyms.

Could this be also related to Carthage? @eric-chamberlain-ck @nickprkins are you using Carthage?

Using Cocoapods and Bitrise

[Elshad]

I use manual upload via upload-symbols, but there are stil huge amount of missing dSYM warnings. Even some times required dSYM warnings. I upload release dSYMs, debug simular and device dSYMs, but still same warnings.
But is there such a thing that the warnings disappears after a few days or hours or minutes after successful dSYM upload?

Edit: Uploaded all available dSYMs from DerviedData, Carthage and other folders, after few time (maybe hours) all warning disappeared -> Edit 2 -> Wrong alarm -> #5569 (comment)

[dloic]

@samedson It appears all missing dsyms are gone since Friday for our production app. We did nothing on our side. Have you find a bug on your end and manage to solve it?

[samedson]

@dloic we actually just found a backend bug that was preventing missing dSYMs from showing up. They should start showing up again today or tomorrow. Apologies for the issue!

[nickprkins]

@samedson YES!!!! Thank you. Will be looking!

[dloic]

@samedson thanks, confirming they did appear again, even if it still make no sense based on discussion we had in this issue :/

We're still seeing many many crashes with missing dsyms, despite seeing nothing on AppStoreConnect, nor getting any user feedback.

[samedson]

@dloic one thing to note is optional dSYMs will not hide crashes, only required dSYMs will hide crashes. If you have a crash with a missing optional dSYM, it will just show "" in the line for that library in the stacktrace.

Are you seeing crashes in Crashlytics?

[dloic]

No I don't think. Here are our numbers:
(please note that version 1.9.0 of our app is live since May 25, 2020, to give you an idea)

That's why we started to have a look at all those missing dsyms files, leading to the fact we found that some reports were not properly symbolized (the essence of this bug report).

Does it make sense?

[anoop4real]

We are also facing the issue we recently moved to Firebase crashlytics SDK and implemented /FirebaseCrashlytics/run in build phase and added necessary input files but we are seeing errors saying that dsyms are missing.... any more setting to be done?

[marcoruggine]

We too are running in the same issue, hopefully there will be a way to automatically upload optional dSYMs otherwise it becomes a real pain.

[M0rtyMerr]

I can reproduce it too

[NicolaiBure]

Same issue for me. No one of the versions are uploading the DYSM automatically, I have uploaded the files using upload-symbols. The console tells me that it have been uploaded successfully but I still have the Missing files on the Crashlytics console. I have been using it on Fabric for years and I hoped that it will work nice when I change to Firebase but it doesn't. Do u have any update?

[chickendiver]

Same issue here.

In our production (App Store) target, we have two "Run Script" build phases:

In the first, we call "${PODS_ROOT}/FirebaseCrashlytics/run", with an input file: ${DWARF_DSYM_FOLDER_PATH}/${DWARF_DSYM_FILE_NAME}/Contents/Resources/DWARF/${TARGET_NAME}

In the second "Run Script" phase, we execute the following:

find "${DWARF_DSYM_FOLDER_PATH}/${DWARF_DSYM_FILE_NAME}" -name "*.dSYM" | xargs -I \{\} $PODS_ROOT/FirebaseCrashlytics/upload-symbols -gsp ${PROJECT_DIR}/.../GoogleService-Info.plist -p ios \{\}

We're finding that no Optional OR Required DSYMs are appearing, even though they were a few weeks ago. This is a new issue for us as of the last < 30 days. We were running Fabric Crashlytics, and transitioned to Firebase Crashlytics earlier this year.

Current CocoaPods FirebaseCrashlytics version in production: 4.1.0. Updating to 4.1.1 and testing on development today.

Update 1: Just did a test to look at the build logs when running this, and discovered that only one DSYM upload was done, when there should have been many. I then changed my second "Run Script" phase script to: $PODS_ROOT/FirebaseCrashlytics/upload-symbols -gsp ${PROJECT_DIR}/.../GoogleService-Info.plist -p ios ${DWARF_DSYM_FOLDER_PATH} and found that approximately 40 DSYMs were uploaded during the build process, which was perhaps part of my problem in finding the "Required" DSYMs on the Crashlytics console. I won't know if this solved my problem or not until pushing to production.

Update 2: We manually downloaded the DSYMs from App Store Connect for each version displaying missing DSYMs on the Crashlytics console, and then manually uploaded them with upload-symbols in the terminal. This got rid of most of our missing DSYMs, but there are still missing optional DSYMs from our last version that was released approximately at the time this issue was opened. This was done before Update 1, so we're still waiting for Apple to approve our latest version with the build phase change to see if it impacted what we're seeing with missing symbols.

Update 3 (July 15, 2020): Still no required DSYMs showing up on the console. At a bit of a loss, but will try @joshuapoq's suggestion below and report back

[Elshad]

@chickendiver But it seems second run script will take more time to upload and build process will wait for it, isn't it?

[chickendiver]

@chickendiver But it seems second run script will take more time to upload and build process will wait for it, isn't it?

For us, the difference is almost negligible: 240.6 s to build with the first version of the second run script, and 305.4 s to build with the second version. One minute extra is something we can live with if the end result is that our crash reports keep rolling in. Might vary depending on the scope of your project, though.

[joshuapoq]

@chickendiver Update 1: Just did a test to look at the build logs when running this, and discovered that only one DSYM upload was done, when there should have been many. I then changed my second "Run Script" phase script to: $PODS_ROOT/FirebaseCrashlytics/upload-symbols -gsp ${PROJECT_DIR}/.../GoogleService-Info.plist -p ios ${DWARF_DSYM_FOLDER_PATH} and found that approximately 40 DSYMs were uploaded during the build process, which was perhaps part of my problem in finding the "Required" DSYMs on the Crashlytics console. I won't know if this solved my problem or not until pushing to production.

We have a lot of clients with many targets each so I have modified the run script to this which looks like it works and 'finds' the Google Service Plist based on it being named exactly that and no weird copying into the app.

UPLOAD_ARGUMENTS="-gsp $BUILT_PRODUCTS_DIR/$EXECUTABLE_FOLDER_PATH/GoogleService-Info.plist -p ios $DWARF_DSYM_FOLDER_PATH $ARGUMENTS"
eval $COMMAND_PATH$UPLOAD_ARGUMENTS

This adds some time to the build phase so you can add > /dev/null 2>&1 & to the eval to move it to the background but I prefer it to be part of Xcode.

[chickendiver]

I did some testing on a local production version of our app in the simulator, detached from XCode.

I'm still seeing missing optional symbols, as reported in this issue. However, it looks like the required symbols are being uploaded fine.

The only change I made was rearranging the order of my Run Script phases, so that my "${PODS_ROOT}/FirebaseCrashlytics/run" script is called after my upload-symbols run script.

It appears as though my problem above may be related to a different issue involving Required DSYMs, so I will stop commenting here unless I have any progress with the Optional DSYMs issue I'm experiencing.

[anoop4real]

Ok finally got some way to get some crashes. For me both manual upload from computer as well as the script in build phase didnt work . I feel the culprit was bitcode.... after reading this .... i downloaded the Dsyms from Appstore connect and uploaded manually via upload-symbols..... at least I am getting Required Dsyms (where most of the core crashes are).... still optionals are missing.

[klmitchell2]

@anoop4real Where did you find the dSYMs to download in Appstore Connect? I've been unable to locate a download link.

[anoop4real]

@klmitchell2 Its AppstoreConnect-->MyApps-->Select your app-> Activity (Select from the top bar menu options, the one next to TestFlight) --> In activity select the version --> Select the build --> In General Information you will be able to see this

[klmitchell2]

Interesting, this is what I see:

Being able to download the dSYM would fix my issue, since I cannot locate the missing dSYMs that are required.

[anoop4real]

@klmitchell2 hmm strange... is bitcodeenabled in your project? It seems the download link is available for those projects as per this ...and sometimes it takes a bit of time for the link to appear it seems

[3a4oT]

We use Carthage and implemented /FirebaseCrashlytics/run in the build phase but it doesn't work properly! Please fix it.

[OJDee]

Is there a reason to use the run script AND call upload-symbols separately when the run script calls upload-symbols at least twice itself?

[glennposadas]

Wow this is such a hassle from the Firebase team. What's the reason behind all these? :/ Automation? It's so complicated.

[NordicBat]

So, no final solution yet?
Looks like some managed with some hacks and workarounds, but the same hack doesn’t work for others

[simdani]

What to do if missing uuids in firebase console do not match the local ones?

[samidaik00]

I'm having the exact same issues.
Also tried the upload_symbols_to_crashlytics function from fastlane which was showing as sucess in Logs , but still noting in Firebase.

My workaround for now is same as what @anoop4real described: download from AppStore Connect and locally run the script. At least can see my crashes now.

[alexargo]

We're experience this issue as well. We have most of our code separated into Frameworks and it seems like none of the dsyms for those are are uploaded and are listed as "Optional". Using the upload-symbols tool on the dsyms in Xcode's archive that is generated when you build and upload seems to get us the dsyms needed to symbolicate our crash reports luckily. We don't use bitcode, so that at least eliminates that as an issue for us and helps isolate the issue. This makes sense based on @samedson 's explanation earlier in the bug thread. We aren't doing anything special or custom as far as I can tell about telling the dsyms where to go. I wonder if there's a setting that we can set that change where our debug symbols show up. Alternatively, would it make sense for Firebase run to check a few more 'default' places for debug symbols? I think I'm going to try to go down the path of manually running upload-symbols as part of our build process in the meantime.

[rogerluan]

It would already help A TON if upload-symbols was a bit more verbose and spit out useful info such as: the path of the dsym it is uploading and status of the upload (whether it started, succeeded, failed, etc)....

EDIT: There is such debugging tool, you gotta pass --debug or -d to upload-symbols 👍

Turns out this is what happens to my dSYMS when I enable that option:

// This is using `| grep "error: "`, there're also dsym uploads that do succeed.
error: Could not fetch upload-symbols settings: An unknown error occurred fetching settings.
error: Could not fetch upload-symbols settings: An unknown error occurred fetching settings.
error: Could not fetch upload-symbols settings: An unknown error occurred fetching settings.
error: Could not fetch upload-symbols settings: An unknown error occurred fetching settings.
error: Could not fetch upload-symbols settings: An unknown error occurred fetching settings.
error: Could not fetch upload-symbols settings: An unknown error occurred fetching settings.
error: Could not fetch upload-symbols settings: An unknown error occurred fetching settings.
error: Could not fetch upload-symbols settings: An unknown error occurred fetching settings.
error: Could not fetch upload-symbols settings: An unknown error occurred fetching settings.
error: Could not fetch upload-symbols settings: The request timed out.
error: Could not fetch upload-symbols settings: The request timed out.
error: Could not fetch upload-symbols settings: The request timed out.
error: Could not fetch upload-symbols settings: The request timed out.
error: Could not fetch upload-symbols settings: The request timed out.
error: Could not fetch upload-symbols settings: The request timed out.
error: Could not fetch upload-symbols settings: The request timed out.
error: Could not fetch upload-symbols settings: The request timed out.
error: Could not fetch upload-symbols settings: The request timed out.
error: Could not fetch upload-symbols settings: The request timed out.
error: Could not fetch upload-symbols settings: An unknown error occurred fetching settings.
error: Could not fetch upload-symbols settings: An unknown error occurred fetching settings.
error: Could not fetch upload-symbols settings: An unknown error occurred fetching settings.
error: Could not fetch upload-symbols settings: The request timed out.
error: Could not fetch upload-symbols settings: The request timed out.

Note that my internet is perfectly fine, high speed, not behind any firewall, sketchy VPNs and such 😞

---

Also, based on @samedson 's comment here:

Sometimes if the .framework folder for the SDK has the dSYM inside of it, it will need to be uploaded manually via upload-symbols separately. upload-symbols can have trouble finding these dSYMs because they are not usually referenced by Xcode's build environment variables.

Isn't there anything that can be done to recursively find dsyms for a given build (e.g. inside .framework folders?), and upload all of them?

[fdstevex]

Seeing this as well, but with a required dSYM. Crashlytics says "Upload 1 missing dSYM required to process 1 crash". Says ID "EDAC2425-5C1D-3E4E-B6FD-2C7653A9EBDD" for version 1.9.26 (202011261509) is missing.

But when I run the script with -d it prints:

Inspecting next path: /Users/stevex/Downloads/Snap Stage/Snap Stage.app.dSYM
Found dSYM at path: /Users/stevex/Downloads/Snap Stage/Snap Stage.app.dSYM
Begin processing dSYM at /Users/stevex/Downloads/Snap Stage/Snap Stage.app.dSYM
Zipping csym for uuid edac24255c1d3e4eb6fd2c7653a9ebdd, architecture: arm64
Zipping csym succeeded for uuid edac24255c1d3e4eb6fd2c7653a9ebdd, architecture arm64... getting compressed data
Deleting file at path: /Users/stevex/Downloads/Snap Stage/arm64-edac24255c1d3e4eb6fd2c7653a9ebdd.csym
Deleted file at path /Users/stevex/Downloads/Snap Stage/arm64-edac24255c1d3e4eb6fd2c7653a9ebdd.csym
Deleting file at path: /var/folders/0w/fz8czngd4l523lwwymgrjsrm0000gr/T/ED62FE8D-B12D-4FB6-9693-A3918139A40E-19077-000311F87F91940C.zip
Deleted file at path /var/folders/0w/fz8czngd4l523lwwymgrjsrm0000gr/T/ED62FE8D-B12D-4FB6-9693-A3918139A40E-19077-000311F87F91940C.zip
Uploading cSYM to https://firebasecrashlyticssymbols.googleapis.com/v1/project/-/app/1:762461417480:ios:9489589838e15781/upload/native/edac24255c1d3e4eb6fd2c7653a9ebdd
Uploading cSYM for uuid edac24255c1d3e4eb6fd2c7653a9ebdd, architecture arm64
Successfully submitted symbols for architecture arm64 with UUID edac24255c1d3e4eb6fd2c7653a9ebdd in dSYM: /Users/stevex/Downloads/Snap Stage/Snap Stage.app.dSYM

[sethfri]

Is there a reason to use the run script AND call upload-symbols separately when the run script calls upload-symbols at least twice itself?

@OJDee I was wondering this too when I first found the thread. After reading through, my take is yes, the reason being that run doesn't upload any optional dSYMs. You seem to have to upload those manually using upload-symbols

[mmdock]

Using Fastlane with the following lanes on install build archives.

  lane :dsyms do
    version = get_version_number(target: "MyApp")
    upload_dsyms(version: version)
  end

  desc "Download dSYMs from ASC for version and upload dSYMs to Crashlytics"
  lane :upload_dsyms do |options|
    download_dsyms(version: options[:version])
    upload_symbols_to_crashlytics
    clean_build_artifacts
  end

This results in a call to:
${PODS_ROOT}/FirebaseCrashlytics/upload-symbols -gsp ${PROJECT_DIR}/.../GoogleService-Info.plist -p ios ${PROJECT_DIR}.{BUNDLE_ID}-{VERSION}-{BUILD_NUM}.dSYM.zip

This seems to get me a mix of the required and the optional dSYMs uploaded, but I am frequently left with a number of required and dSYMs still needed with each version, for example:

I become unable to discover these remaining dSYMs. Does this have something to do with the need to run FIrebaseCrashlytics/run? or are there locally generated dSYMS on builds that do not get included in the Testflight Archive upload?

I thought I would be able to find the local dSYM path via
xcodebuild -project MyProject.xcodeproj -target MyTarget -showBuildSettings

but there doesn't seem to be anything of relevance showing in this output.

[chickendiver]

Also, based on @samedson 's comment here:

Sometimes if the .framework folder for the SDK has the dSYM inside of it, it will need to be uploaded manually via upload-symbols separately. upload-symbols can have trouble finding these dSYMs because they are not usually referenced by Xcode's build environment variables.

Isn't there anything that can be done to recursively find dsyms for a given build (e.g. inside .framework folders?), and upload all of them?

This inspired me to build a tool that will simply crawl your archive directory and upload all dSYMs that it finds. The tool is an open source Python script, and it even caches the results of its last run so that it doesn't re-upload previously uploaded scripts!

https://github.com/chickendiver/firebase-iOS-dSYM-uploader

Edit: Realizing this was posted exactly one year after the original issue was opened. Happy birthday, everyone! 🎉

[pepasibble]

Here is my take on a dsym uploader: https://gist.github.com/cellularmitosis/37782a684ec83e1ce623ff4cc6335b7b

[samedson]

Hey folks, we're going through and closing old issues. I'm going to close this for now, but feel free to reopen if you are still running into issues.