Countly Documentation

Countly Resources

Here you'll find comprehensive guides to help you start working with Countly as quickly as possible.

Crash symbolication

Introduction to symbolication

When releasing your mobile app, you might have come to the conclusion that you want to either obfuscate your application or reduce the size of it by optimizing the bytecode. A benefit that would come out of these actions is that this would make your application also harder to reverse engineer.

Whatever reason you might have to do that, the result is that now you receive error stack traces that have obfuscated information in them that make it impossible to track down the source of your crash.

Depending on the platform, your stack trace would be full of memory addreses or transformed function names. Symbolication is the process of converting those into human readable class/method name, file name and line number.

Below you can view a tutorial about how symbolication works:

Enterprise feature

Crash symbolication is available for Enterprise Edition users.

For the obfuscation process to be reversible, a symbol or mapping file is also produced while building your application. It is imperative for you to keep and archive those files as symbolication process is only possible with them. You also need to archive symbol files for every version you want to symbolicate.

Currently Countly supports symbolication for dexguard,proguard in Android and Apple provided tools in iOS mobile applications.

Android symbolication sample:

java.lang.IllegalStateException: Could not execute method for android:onClick
at android.view.View$DeclaredOnClickListener.onClick(View.java:4725)
at android.view.View.performClick(View.java:5637)
at android.view.View$PerformClick.run(View.java:22429)
at android.os.Handler.handleCallback(Handler.java:751)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:154)
at android.app.ActivityThread.main(ActivityThread.java:6121)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)
Caused by: java.lang.reflect.InvocationTargetException
at java.lang.reflect.Method.invoke(Native Method)
at android.view.View$DeclaredOnClickListener.onClick(View.java:4720)
... 9 more
Caused by: java.lang.Exception: Exception at the end of the call
at ly.count.android.demo.a.b(SourceFile:29)
at ly.count.android.demo.a.a(SourceFile:21)
at ly.count.android.demo.ActivityExampleCrashReporting.c(SourceFile:98)
at ly.count.android.demo.ActivityExampleCrashReporting.b(SourceFile:94)
at ly.count.android.demo.ActivityExampleCrashReporting.a(SourceFile:90)
at ly.count.android.demo.ActivityExampleCrashReporting.onClickCrashReporting10(SourceFile:82)
... 11 more
java.lang.IllegalStateException: Could not execute method for android:onClick
at android.view.View$DeclaredOnClickListener.onClick(View.java:4725)
at android.view.View.performClick(View.java:5637)
at android.view.View$PerformClick.run(View.java:22429)
at android.os.Handler.handleCallback(Handler.java:751)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:154)
at android.app.ActivityThread.main(ActivityThread.java:6121)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)
Caused by: java.lang.reflect.InvocationTargetException
at java.lang.reflect.Method.invoke(Native Method)
at android.view.View$DeclaredOnClickListener.onClick(View.java:4720)
... 9 more
Caused by: java.lang.Exception: Exception at the end of the call
at ly.count.android.demo.Utility.void DeepCall_b()(SourceFile:29)
at ly.count.android.demo.Utility.void DeepCall_a()(SourceFile:21)
at ly.count.android.demo.ActivityExampleCrashReporting.void deepFunctionCall_3()(SourceFile:98)
at ly.count.android.demo.ActivityExampleCrashReporting.void deepFunctionCall_2()(SourceFile:94)
at ly.count.android.demo.ActivityExampleCrashReporting.void deepFunctionCall_1()(SourceFile:90)
at ly.count.android.demo.ActivityExampleCrashReporting.void onClickCrashReporting10(android.view.View)(SourceFile:82)
... 11 more

iOS symbolication sample:

libsystem_platform.dylib            0x00000001826c130c _sigtramp + 36
mahya                               0x000000010006e174 mahya + 156020
mahya                               0x000000010006d060 mahya + 151648
mahya                               0x000000010006ad34 mahya + 142644
UIKit                               0x0000000189925d74  + 184
UIKit                               0x00000001898eedb8  + 128
UIKit                               0x00000001898ff2b0  + 88
UIKit                               0x000000018a1c0ff8  + 272
UIKit                               0x0000000189b5e138  + 280
UIKit                               0x0000000189b5d96c  + 1064
UIKit                               0x0000000189b5d4c4  + 60
UIKit                               0x000000018a1c0ff8  + 272
UIKit                               0x0000000189b5d400  + 436
UIKit                               0x00000001898eea40  + 672
UIKit                               0x00000001898ee364  + 1780
UIKit                               0x00000001898ec954  + 228
UIKit                               0x00000001898eab04  + 3152
UIKit                               0x0000000189b74750  + 228
UIKit                               0x000000018975ea50  + 384
UIKit                               0x0000000189b74400  + 344
UIKit                               0x0000000189768fd4  + 2480
UIKit                               0x000000018976436c  + 3192
UIKit                               0x0000000189734f80  + 340
UIKit                               0x0000000189f2ea20  + 2400
UIKit                               0x0000000189f2917c  + 4268
UIKit                               0x0000000189f295a8  + 148
CoreFoundation                      0x00000001835b142c  + 24
CoreFoundation                      0x00000001835b0d9c  + 540
CoreFoundation                      0x00000001835ae9a8  + 744
CoreFoundation                      0x00000001834deda4 CFRunLoopRunSpecific + 424
GraphicsServices                    0x0000000184f49074 GSEventRunModal + 100
UIKit                               0x0000000189799c9c UIApplicationMain + 208
mahya                               0x000000010005fe14 mahya + 97812
libdyld.dylib                       0x00000001824ed59c  + 4
libsystem_platform.dylib            0x00000001826c130c _sigtramp + 36
-[MHViewController countlyProductionTest] (in mahya) (MHViewController.m:620)
-[MHViewController transitionToMahya] (in mahya) (MHViewController.m:443)
-[MHViewController textFieldShouldReturn:] (in mahya) (MHViewController.m:210)
UIKit                               0x0000000189925d74  + 184
UIKit                               0x00000001898eedb8  + 128
UIKit                               0x00000001898ff2b0  + 88
UIKit                               0x000000018a1c0ff8  + 272
UIKit                               0x0000000189b5e138  + 280
UIKit                               0x0000000189b5d96c  + 1064
UIKit                               0x0000000189b5d4c4  + 60
UIKit                               0x000000018a1c0ff8  + 272
UIKit                               0x0000000189b5d400  + 436
UIKit                               0x00000001898eea40  + 672
UIKit                               0x00000001898ee364  + 1780
UIKit                               0x00000001898ec954  + 228
UIKit                               0x00000001898eab04  + 3152
UIKit                               0x0000000189b74750  + 228
UIKit                               0x000000018975ea50  + 384
UIKit                               0x0000000189b74400  + 344
UIKit                               0x0000000189768fd4  + 2480
UIKit                               0x000000018976436c  + 3192
UIKit                               0x0000000189734f80  + 340
UIKit                               0x0000000189f2ea20  + 2400
UIKit                               0x0000000189f2917c  + 4268
UIKit                               0x0000000189f295a8  + 148
CoreFoundation                      0x00000001835b142c  + 24
CoreFoundation                      0x00000001835b0d9c  + 540
CoreFoundation                      0x00000001835ae9a8  + 744
CoreFoundation                      0x00000001834deda4 CFRunLoopRunSpecific + 424
GraphicsServices                    0x0000000184f49074 GSEventRunModal + 100
UIKit                               0x0000000189799c9c UIApplicationMain + 208
main (in mahya) (main.m:16)
libdyld.dylib                       0x00000001824ed59c  + 4

Configuring server for symbolication

First step to using symbolication is installing and enabling the symbolication plugin, which is called "Crash symbolication".

The next step to using symbolication is creating the connection to the Countly symbolication server. For that you need a symbolication API key provided by Countly. Countly symbolication server address is https://symbolication.count.ly - do not change this part unless you have your own symbolication server. When you get your API key, you need to set them in the server configuration screen under the "crashes" section.

After you have made sure, that those fields contain valid information, press the "Test Connection" button to make sure that everything is working fine. If your provided server url or API key is wrong or your countly server can't reach the symbolication server, you will diagnose it with this button.

If everything is working fine, you should see these alert messages.

With this, you should be done with setting up your Countly instance to use crash symbolication.

Preparing your app for symbolication

This part will guide you through Android & iOS symbolication processes.

Android

Android's official tool for code shrinking and obfuscation is called ProGuard. Detailed descriptions of its usage can be found here. There is also a paid tool with additional features, called DexGuard. Both Proguard or Dexguard can be used for Countly crash symbolication. Currently we don't support any other Android obfuscation libraries.

If you are using Android Studio for development, mapping files will not be produced when you make instant runs. For them to appear you need either to generate a signed APK or choose the "Build APK" option.

After the build is complete, the symbol file called mapping.txt can be found under <module-name>/build/outputs/mapping/release/ or <module-name>/build/outputs/mapping/debug/, depending on how you initiate the build process.

Proguard rules

It's possible to add some rules to proguard/dexguard and modify how it's running. They should be added in the "proguard-rules.pro" file.

If you are deciding if to use proguard, you have to keep in mind that it will rename function and class names. Therefore it will break code that is using reflection if you don't take steps to safeguard from that. More on that in the proguard manual

At the very least you should include these lines in your proguard rule file:

-keep class org.openudid.** { *; }

-renamesourcefileattribute SourceFile
-keepattributes SourceFile,LineNumberTable

The first one is needed to prevent openudid from breaking, it is used by countly for generating user id's. The third and fourth are needed to add source file and line number information to your android stack traces.

The proguard rule file should be named "proguard-rules.pro", usually it is found at the root of your module, more on that here.

iOS

For iOS, the symbol file is dSYM file.

  • A dSYM file is Apple's standard Mach-o file which contains debug symbols for a given build.
  • It includes debug symbols for all architectures used for the build (e.g. armv7, arm64).
  • Its size can vary depending on original source code and libraries used in the project.
  • It is actually a file-like folder structure and actual dSYM data is in the binary file AppName.app.dSYM/Contents/Resources/DWARF/AppName.

dSYM Location:

First of all, for each executable target (main app, extensions and Cocoa Touch Frameworks) in the project, a separate dSYM file is generated when the project is built. And dSYM location depends on build settings. By default it is defined by $DWARF_DSYM_FOLDER_PATH Xcode Environment Variable.

An example location is:
~/Library/Developer/Xcode/DerivedData/AppName-abcdef0123456789/Build/Products/Release-iphoneos/AppName.app.dSYM

In addition to this, if you use Product > Archive option in Xcode to create .xcarchive of your app, you can find dSYM inside created .xcarchive. By default its location is: ~/Library/Developer/Xcode/Archives/YYYY-MM-DD/AppName DD-MM-YYYY, HH.mm.xcarchive/dSYMs

Automatic dSYM Upload

For automatic dSYM upload please Countly iOS SDK documentation:
https://resources.count.ly/docs/countly-sdk-for-ios-and-os-x#section-symbolication

Using symbolication

This section will guide you to uploading a symbol file and then symbolicate your stack trace for Android and iOS.

Uploading the Symbol File

Now that the connection to the symbolication server is established and the necessary symbol files are procured, we can get to the symbolication of specific stack traces.

After enabling the crash symbolication plugin, you might have noticed that there are additional choices added to your crash sub menu items:

"Overview" takes you to the usual crash overview. "Manage Symbols" takes you to a screen where you can manage all the symbol files uploaded for this application. "Symbolication logs" is similar to the event log where a short history of your symbolication requests is kept. You would visit that screen only if there are some problems with symbolication process.

To upload a symbol file, open the "Manage symbols" section. At first it will look empty, as shown below:

There you will have to click on "Add Symbolication File" button. In the side dialog, select the type, add the build id, mapping/symbol file, note and click "Upload file". You would do this for every version of your app you want to symbolicate.

Only one symbol file upload required

You should upload only one symbol file per application version. Multiple symbol file upload is not supported.

In the Type selection, you need choose the platform for the symbol file you are uploading.

For Android, Build ID field is the version name, and for iOS Build UUID field is your app's Build UUIDs.

You can get Build UUIDs using dwarfdump --uuid command on dSYM file. And you can specify more then one Build UUID comma separated, for each architecture.

Example:

erkanmbp:Desktop erkanyildiz$ dwarfdump --uuid CountlyTestApp-iOS.app.dSYM
UUID: AD4F5647-B2A0-3D85-8DF4-78D4DFD83296 (armv7) CountlyTestApp-iOS.app.dSYM/Contents/Resources/DWARF/CountlyTestApp-iOS
UUID: 70B0DE66-D337-3952-914D-A1E542667E20 (arm64) CountlyTestApp-iOS.app.dSYM/Contents/Resources/DWARF/CountlyTestApp-iOS

While choosing Symbolication/Mapping File on upload page for iOS, please do not forget to zip dSYM file before uploading. Otherwise you can not upload as dSYM file is a folder structure in fact.

The "Notes" field is optional

When everything is filled out, just click "Upload file".

Note that Android Build ID and iOS Build UUID specified here have to match your app's Build ID or Build UUID.

You should do this step for every version/build you want to symbolicate.

Symbolicating a Crash

After a while you will see new exceptions in your crash overview pane. If you have uploaded correct symbols for those versions, then you are ready to move forward with symbolication process.

To symbolicate a crash, you first have to go to it's detailed view by clicking on "view".

The detailed crash view can be divided in two parts. The top contains the crash group stack trace, and the bottom contains entries for each specific crash and its circumstances.

When clicking on a specific crash entry, it opens up additional information, which also contains a "Symbolicate" button if it's possible to symbolicate the crash (eg symbol file is uploaded for corresponding version). At the top the crash group section, you also can see a "Symbolicate" button.

When you click on a "Symbolicate" button, you should see the "Symbolication in progress" message. When that process is complete, you should be able to see the symbolicated stack trace.

In the current symbolication implementation there are some technical limitations, and the top crash group will show a "symbolicate" button only after a crash from a newer app version is received. Therefore if you want to symbolicate crashes from a version you had before you enabled crash symbolication, you will have to go down to a specific crash entry at the bottom of the detailed crash view.

Diagnosing a problem

There might be a situation where trying to symbolicate a crash, you see something like this:

That means that there was a problem while trying to symbolicate your crash. To get more information on the problem, either click on the red text or open the "Symbolication Logs" section.

After the reason for the failure has been fixed, you can rerun that symbolication task.