Performance Regressions Policy

Basic Policy

Performance is a critical goal for Firefox releases, as well as Mozilla’s other products and services. We cannot allow performance regressions to go unnoticed or unresolved during our development cycles. When a regression is identified, a bug will be filed and the patch author will be asked to provide information via the needinfo flag in bugzilla. We expect a response and reasonable dialog within 3 business days. If there is no acknowledgement within that time, and no investigation has started, then we may back out the offending patch(es).

Critical Tests

Some of our performance tests are marked as critical tests. These are the ones where we do not want to regress their metric under any circumstances unless the regression is explicitly approved by appropriate parties. When one of these tests regresses, the regressor patch is backed out immediately by sheriffs after it has been found and confirmed. See the existing list of critical tests here.

Sheriffing Criteria

In order to make efficient use of our performance sheriff resources, the following criteria must be met before results will be monitored for regressions:

• Test jobs run on integration or mozilla-beta.
• Test jobs run as tier 2 or above.
• Tests can be run and debugged locally.
• Tests are documented in the Firefox Source Docs.

Bug Requirements

To be fair, some initial work up front should be done prior to filing a bug and requesting a needinfo from the original author. Here is what to expect:

• On integration branches (higher volume), a performance sheriff will have verified the root cause within 5 business days of the patch landing.
• A patch or set of patches from a bug must be identified as the root cause. This can take place through retriggers on the tree or in the case of many patches landing at once this would take place through a push to try backing out the suspected patch(es).
• Links in the bug to document the regression (and any related regressions/improvements).
• If we are confident this is the root cause and it meets a 2% regression threshold, then the needinfo request will mention that this policy will be enforced.

Acceptable Outcomes

What we expect from a developer when asking them to look at a regression:

• A promise to attempt to fix the bug is agreed upon, the bug is assigned to someone and an estimated timeline is documented in the bug.
• The bug will contain enough details and evidence to support accepting this regression, we will mark it as wontfix.
• It is agreed that this should be backed out and a sheriff will verify the back out removes the regressions.

Other Scenarios

There are many reasons why a regression will not fall into the above workflow. Here a few examples of other scenarios and how we will handle them:

• Regression is unrelated to code (e.g. infrastructure changes) - this should be documented in the bug and closed as wontfix.
• A back out or fix doesn’t fully resolve the regression. We will do our best to understand the difference and fix or document the remainder.