Performance Regressions Policy
Basic Policy: Bug filed
Performance is a critical goal for Mozilla releases and the commercial products that will be based on Mozilla. We cannot allow performance regressions to go unnoticed or unresolved during our development cycles. When a regression is identified a bug will be filed and the patch author will be asked to provide information via the needinfo flag in bugzilla. We expect a response and reasonable dialog within 3 business days. In the unfortunate event that we cannot learn more about the regression, we will backout the offending patch(es) and the patch author can reland when they have more time to investigate.
To be fair, some initial work up front should be done prior to filing a bug and requesting a needinfo from the original author. Here is what to expect:
- On integration branches (higher volume), a talos sheriff will have verified the root cause within 1 week of the patch landing.
- a patch or set of patches from a bug must be identified as the root cause. This can take place through retriggers on the tree or in the case of many patches landing at once this would take place through a push to try backing out the suspected patch(es).
- links in the bug to document the regression (and any related regressions/improvements).
- if we are confident this is the root cause and it meets a 3% regression threshold, then the needinfo request will mention that this policy will be enforced.
What we expect from a developer when asking them to look at a regression:
- A promise to attempt a fix at the bug is agreed upon, the bug is assigned to someone and an estimated timeline is documented in the bug.
- The bug will contain enough details and evidence to support accepting this regression, we will mark it as wontfix.
- It is agreed that this should be backed out and a sheriff will verify the backout removes the regressions.
There are many reasons why a regression will not fall into the above workflow. Here a few examples of other scenarios and how we will handle them:
- A bug related to the alert is not filed within 1 week of the patch landing. This removes the urgency and required action.
- We only caught a regression at uplift time. There is a chance this isn’t easily determined, this will be documented and identified patch authors will use their judgement to fix the bug.
- Regression is unrelated to code (say pgo issue) - this should be documented in the bug and closed as wontfix.
- When we uplift to Aurora or Beta, all regressions filed before the uplift that show up on the upstream branch will have a needinfo flag set and require action to be taken.
- A backout or fix doesn’t fully resolve the regression. We will do our best to understand the difference and fix or document the remainder.
Contact: Joel Maher <email@example.com>