About/Methodology

Variable Definitions

  • crash_rate_main: crashes experienced due to a browser crash (full crash) per user per thousand hours.
    (note: we use crash_submit_success_main)
  • crash_rate_content: crashes experienced due to a content crash (tab crash) per user per thousand hours.
    (note: we subtract ShutDownKill from crashes_detected_content)
  • crash_rate_plugin: crashes experienced due to a plugin per user per thousand hours.
    (note: we sum crashes_detected_plugin and crashes_detected_gmplugin as a single type of plugin crash)
  • weekly_active_users: number of users active in the week ending on the provided date.
  • wau_crashes: out of all weekly active users, how many experienced a product crash that week?
  • wau_2+_crashes: out of all weekly active users, how many experienced more than 2 product crashes that week?
  • multiple_crashes: out of all users that crashed during the week, how many have had a prior product crash in their history?
  • first_time_crashes: out of all users that crashed during the week, for how many was it their first product crash?
  • median_hours_between_crashes: median number of active hours between the two most recent product crashes for users who have had multiple product crashes.
  • geom_mean_hours_between_crashes: geometric mean of active hours between the two most recent product crashes for users who have had multiple product crashes.
  • e10s_enabled/disabled: out of all weekly active users that crashed, how many had e10s enabled/disabled?
  • main/content/plugin_e10s_enabled/disabled: main/content/plugin crash rates as above, for profiles that had e10s enabled or disabled for that week.
  • new_profiles: out of all weekly active users, how many created their profile within the past week?
  • percentage_new_crashes: out of all the new profiles created the past week, how many experienced a product crash this week?
  • percentage_new_2+_crashes: out of all the new profiles created the past week, how many experienced more than two product crashes this week?
  • percentage_new_crashes_bis: out of all the new profiles created three weeks ago, how many experienced a product crash in their first two weeks of activity?

Date Ranges

Calculations are made for periods of seven (7) days ending on the day shown on the graphs (which are Mondays, Wednesdays, and Fridays). Example: if the graph shows "Jan 17", then the metrics were measured during the week of "Jan 10 - Jan 17".

Weekly Active Users

Weekly active users are measured as the total number of distinct profiles that were active at least once in the week from a representative 1% sample of the main summary dataset (schema definition). Activity is measured using submission date instead of subsession start date. This allows for the numbers to stay constant independently of when the script is run. I.e., running the experiment for the week ending on day $X$ on day $Y$ or $Y+k$ will give the same results ($Y > X, k > 0$).

Data Aggregation

Data has been aggregated by user and by day of activity, so for each $(user,\ day)$ tuple, we have the sum of subsession lengths, and the sum of each type of crash. For each user that crashed during the week, we find the most recent date when a product crash occurred. We then evaluate whether one or more product crashes were recorded for that user on that day (so the sum of main and content crashes for that user on that day).

  • If this total is more than 1 (through any combination of the crash types), then that user is said to have had multiple product crashes, and the number of days between the two most recents product crashes is 0. Since the data has been aggregated by day, this is the highest resolution we can get for this metric, so the number of hours between product crashes is also set to 0.
  • If this total is exactly 1 (i.e., the user experienced only one crash of any kind on that day), then we look in their previous available history for another day where a product crash was recorded.
    • If one such day is found, then the user is said to have had multiple product crashes, and we measure the time between them. This way, we can get the number of days between product crashes and the number of hours between product crashes (this is the sum of subsession lengths for days between the two most recent product crash dates).
    • If no such day is found, then the user has no previous recorded product crashes, and is said to be a first-time product crash. For these users, we cannot compute time between product crashes. However, this count could be correlated to the number of new profiles created that week and needs to be investigated further.

Crash Rates

Crash rates are calculated as the average of the total number of crashes (of each type) divided by the total sum of subsession lengths for each user (i.e., $avg(\frac{\sum_{users}{crash\_type}}{\sum_{users}{subsession\_length}})$). These rates are then multiplied by $1,000$ to get crash statistics per $1,000$ hours, per user. This calculation method allows for each user to have the same contribution to the crash rates, independent of how long they were active for.

e10s

e10s refers to Electrolysis. Profiles that have this option enabled gain, among others, the ability to host, render, or execute web related content in background child processes. In other words, tabs can function as independent entities. Therefore, it has a direct impact on content crashes, as tabs can crash independently of the whole browser crashing. On the release channel, e10s was enabled by default starting on Firefox v48, which was released on August 1st 2016 (schedule). Previous to that, e10s could be enabled manually by activating the option in the settings. Rate of adoption can be seen in this graph for active users that crashed per week. Main, content, and plugin crash rates for each group of users (whether they have e10s enabled or disabled) are shown in this graph.

Note: you may notice that the sum of the two groups in the graph depicting rate of e10s adoption is usually slightly above $100\%$. This is due to the fact that some users may enable and/or disable e10s within the week, and so would be counted in both groups independently. This is ok, since their browser experience (and crash rates) would theoretically be different in each status.

New Users

New users are defined as those whose profile creation date lies within the week of interest. For example, if we are looking at the week ending on Jan 17, then new users are such that their profile was created between Jan 11 and Jan 17 (and they were active between Jan 11 and Jan 17). This definition can be tweaked if necessary.

Histograms

The histograms of hours between product crashes are heavily skewed towards lower values. Taking the median of these counts to create time-series plots may not be the most accurate representation of such distributions. Therefore, we also plot the geometric mean using log-transformed values (i.e., $exp(avg(log(x+1)))-1$, where we add 1 to the log since we have to deal with zeroes. The extra 1 is then removed from the calculated mean) to correct for this skewness. This should be a better and more accurate metric for these variables. In general, these two metrics behave similarly.

The Team

Saptarshi Guha
Senior Data/Applied Statistics Scientist
@saptarshiguha
Andre Duarte
Data Analyst
@aguimaraesduarte
Connor Ameres
Data Engineer
@cameres
Mozilla