Debugging Mobile Apps In The Wild

Building mobile applications is no walk in the park. Often you are required to integrate multiple third-party cloud services and fail gracefully if (or more likely when) they do not work as expected. Your application will target a wide variety of operating system versions, screen sizes and hardware profiles. Once released, the code will have to fend for itself in a wildly fragmented ecosystem. I’m not trying to scare you, but unless you are Bear Grylls, you should be at least a little worried:

OS versions for active users of an example Adapptor app

Device types for active users of the same app

The Importance of Testing

Hopefully you have a great test team and most issues are discovered early. Inevitably however, end users will find problems that we or our test team could not have predicted or else simply miss. Users who experience a frustrating problem can easily vent with a 1-star app review. In this case, at least you know something is wrong. Many users will simply uninstall the app. It would be so much better if we could have found that bug first.

As developers, we have a suite of tools at our disposal to diagnose issues. We can use the debugger, or trawl through error logs to pinpoint the root cause of bugs. However, once we’ve said goodbye and unleashed our lovingly crafted creations on the world, those tools are less useful as they rely on us being able to reproduce the problem.

So, what can we do?

Thankfully, we don’t have to rely completely on raw intuition. There are a number of excellent tools to help discover and diagnose these problems in the wild. I’ll discuss some of the tools we use at Adapptor and how they help keep our apps running smoothly.

Crash and Error Reporting

The first line of defence against crashes is knowing that they are occurring. Google Play and AppStore Connect include automatic crash reporting, however there are better options. These tools hook into the global error handler and write error details to storage so that on next app launch, useful debugging info such as stack traces and device stats are automatically sent to a cloud service. You can then log in and review the crashes your users are experiencing.

The Tools

Whilst there are many options, at Adapptor we primarily use Crashlytics and Sentry.io. Other tools have similar feature sets.

Crashlytics

Part of Google’s Firebase suite, this tool is well suited to native mobile applications. It’s free and the quality of the crash reports is excellent. Simply add the dependency and some initialisation code and you are away. One of the nicer features is the automatic notifications that occur when previously unseen crashes appear, or when a crash gets a sudden spike in numbers.

Sentry.io

This is our go-to for crash reporting in React Native applications as it has great out-of-the-box support for JavaScript stack traces. This essential feature was lacking in Crashlytics, but has since been added by react-native-firebase. Whilst Sentry has a free tier, it is ultimately a paid service, unless you’re up for the challenge of running your own server. The dashboard features of Sentry are not currently up to the standard of Crashlytics. After filtering the list of events (e.g. to get a picture of how a new release is performing), the total count and mini-graph of crashes over time is unchanged, leaving you wondering just how often the remaining crashes are occurring.

Error Event Reporting

Both Crashlytics and Sentry support logging of arbitrary error events (aka non-fatals). You’ll need to add the code to your error handlers manually, but the benefit is that you get a view into non-fatal errors occurring for your users. Any unexpected error that you’d normally write to a local log could be a good candidate for reporting. Be judicious as having too many expected errors (e.g. network request failures) can cause important errors to be drowned out. They may also impact the performance of your app as the error, stack trace and other details need to be serialised and sent over the network.

Sentry also supports capturing messages, which can be useful when you want to diagnose problems that don’t manifest as errors. Keep in mind the Sentry dashboard will try to group these messages together, so best not to include details such as timestamps. Typically we’ve only added this feature and released a new version of the app when we’ve been unable to track down the source of a problem using existing reporting and debugging.

Bread Crumbs

Imagine your app users are Hansel and Gretel. You can make them leave a trail of steps as they use the app so that when they run into trouble you can retrace their path, giving invaluable insights into the cause of the problem. These tidbits are recorded along with crash or error reports, and presented when viewing the report in the dashboard. Breadcrumbs is the Sentry term; Crashlytics has the more traditionally named custom log messages. A great idea is to use a custom logging class that wraps your normal logger and additionally sends messages of a certain level to the crash reporting tool library.

Analytics and Reporting

The previously mentioned tools are awesome but have one major shortcoming. They only attach the breadcrumbs or logs when an event is reported. You will likely get hard-to-reproduce bug reports without any associated crashes or error events, and to diagnose these, you’ll want to implement another solution to trace the user’s path through the app. The solution described below uses Google’s Firebase Analytics, Big Query and Data Studio to provide a holistic view of errors occurring in your app, whilst also enabling you to drill down and trace a specific user’s error and breadcrumb trail.

Firebase Analytics

Firebase Analytics provides built in device stats and a basic level of screen reporting for native apps. To get the most out of it, you’ll want to add custom events for UI interactions such as button presses as well as events for server requests, responses and errors. If you include the current user identifier as a parameter when logging these events, you will gain a detailed picture of what any particular user is doing in your app. It may feel a little Big Brother, but for complex apps, this data will become invaluable for tracking down difficult issues. Best practice is to protect privacy by avoiding sending any identifying information such as email, username or phone number. Instead, use an internal user id, which can be obfuscated if you need to be extra careful.

Similar to the Breadcrumbs idea, you could add analytics events for log messages above a certain level.

Big Query

By itself, the Firebase Analytics dashboard is not suited to debugging user flows. Enable export to Big Query so you can write custom queries and effectively filter your analytics data. This is a one-off step and the app events will be available within an hour after they occur. Whilst it is a paid service, the cost is minimal and scales with the amount of data stored. The benefit is unlocking the potential of your analytics data.

Big Query’s syntax is close to other SQL databases. One unusual feature is that event parameters are stored in a nested table and require the UNNEST keyword to extract. Here’s an example view that can be used as a starting point to extract parameters from the imported analytics `events_*` table. The star syntax simply means the query applies to all partitions.

SELECTevent_timestamp,event_name,-- Firebase's generated id which changes on each app install user_pseudo_id,-- event parameter for our internal user id(SELECT value.string_valueFROM UNNEST(event_params)WHERE key = "user_id") AS user_id,-- event parameter for categorising errors, eg "GET /user/friends"(SELECT value.string_valueFROM UNNEST(event_params)WHERE key = "error_tag") AS error_tag,-- event parameter for errors or other arbitrary messages(SELECT value.string_valueFROM UNNEST(event_params)WHERE key = "message") AS message,-- useful built in firebase device infoapp_info.version AS version,app_info.id AS app_id,platform,device.mobile_brand_name AS mobile_brand_name,device.mobile_model_name AS mobile_model_name,device.mobile_os_hardware_model AS mobile_os_hardware_model,device.operating_system AS operating_system,device.operating_system_version AS operating_system_versionFROM `app_project.analytics_123.events_*`

Save the above query as a view and next time a user reports an unusual problem without detailed reproduction steps, you can locate it by running a simple query:

SELECT * FROM `app_project.analytics_123.AppUserEvents WHERE user_id = “1231” ORDER BY event_timestamp DESC

Data Studio

Google’s Data Studio is a great tool for quickly creating dashboards. It links easily to Big Query enabling you to create a variety of reports based on your app’s analytics data. Clients love the visibility, but my personal favourite is a visualisation of app errors. Filter your analytics events view by errors (excluding any common “expected” errors) and drop the query into a stacked bar chart.

On the 21st, an API was temporarily returning invalid responses

Incident Management

If your app’s functionality is critical to the business, you may consider a spike in errors as an incident that needs immediate attention. The error stream can be periodically queried and analysed for counts exceeding some threshold. Along with API downtime checks, these spike events can be fed into incident management tools such as Opsgenie and PagerDuty.

Stay in Command

At Adapptor we provide clients with a bespoke tool to help manage and analyse their app performance. We call it Mobile App Command Centre, or MACC.  It bundles a range of performance monitoring services into one powerful web based command centre. MACC can also integrate with third party incident management tools and in-app survey systems to bring other app performance data to the attention of the Product Owner, as it becomes available.

Convinced?

Implementing the tools and processes discussed in this post helps to give developers and business owners peace of mind that their creations are functioning well in the big wide world. When you’re off on your next wilderness trek, that feeling will be invaluable.

Photo by Chris Hayashi on Unsplash