Kai Lindström

In August 2014, a routine rack decommission at Google erased the disks of every satellite machine in the fleet. The Google SRE Workbook uses this incident as a case study in postmortem quality, presenting a deliberately bad example postmortem alongside the real one. The bad example’s action items table has four rows: one has a tracking bug, two use the verbs “improve” and “make better,” and all four are tagged P2. The bad example was published four months after the incident. Three years later, a similar incident drained a number of satellites again. The action items implemented from the original postmortem reduced the blast radius and rate of the second incident. That is the strongest public evidence we have that closing the loop on postmortem action items changes outcomes — and it is also evidence that the loop took three years to close.

Action items decay when they are written as prose intentions rather than tracked, owned, testable artifacts. The decay is measurable without inventing new tooling: count the fraction of items that lack a tracking ID, and count the fraction that have no status change in a rolling window. Both numbers are computable from the postmortem repository and the issue tracker.

What the bad postmortem actually contained

The Google SRE Workbook uses the satellite diskerase incident as a case study in postmortem quality. It presents a deliberately bad postmortem alongside the real one. The bad version’s action items table has four rows:

  • Make automation better. Type: mitigate. Priority: P2. Owner: logantwo@. Tracking bug: (none).
  • Improve paging and alerting. Type: detect. Priority: P2. Owner: sydneythree@. Tracking bug: (none).
  • Needs to learn proper cross-site handoff protocol so nobody has to work on duplicate issues. Type: mitigate. Priority: P2. Owner: (none). Tracking bug: BUG6789.
  • Train humans not to run unsafe commands. Type: prevent. Priority: P2. Owner: (none). Tracking bug: (none).

The workbook’s critique is specific. The action items are mostly mitigative; the one preventative item suggests changing human behavior, which the workbook calls less reliable than changing automated systems and processes. All items share the same priority, so there is no way to determine which to tackle first. The verbs “improve” and “make better” are vague and open to interpretation, making success criteria difficult to measure. Only one item has a tracking bug. The workbook states plainly: “Without a formal tracking process, action items from postmortems are often forgotten, resulting in outages.”

The bad postmortem also listed four owners. The workbook recommends a single point of contact responsible for the postmortem, follow-up, and completion. It notes that action items without clear owners are less likely to be resolved, and that it is better to have a single owner and multiple collaborators.

The rule that makes the gap visible

Ben Treynor Sloss, Google’s VP for 24/7 Operations, is quoted in the workbook: “To our users, a postmortem without subsequent action is indistinguishable from no postmortem. Therefore, all postmortems which follow a user-affecting outage must have at least one P[01] bug associated with them. I personally review exceptions. There are very few exceptions.”

That is a policy, not a metric. But it implies a metric. If every user-affecting postmortem must have at least one P0 or P1 bug, then the fraction of user-affecting postmortems with zero P0/P1 bugs is a direct measure of policy compliance. It is also a measure of decay: an item that never gets a high-priority tracking bug is an item that will not be worked on ahead of feature work.

The SRE Book adds a review criterion: “Is the action plan appropriate and are resulting bug fixes at appropriate priority?” That question is answerable by inspection. It does not require a new dashboard.

What recurrence looks like when you can measure it

The satellite diskerase incident recurred three years later. The workbook says the action items implemented from the original postmortem “dramatically reduced the blast radius and rate of the second incident.” It does not say the second incident was identical in cause. It says a number of satellites were drained, resulting in increased user latency.

This is a natural experiment with a confound: we do not know which action items were implemented, when, or how. We know only that some were, and that the second incident was less severe. That is enough to support a directional claim — closing the loop changes outcomes — but not enough to support a quantitative decay rate. No public dataset measures action-item closure rates across postmortems. Any specific half-life or percentage would be invented.

What we can measure without inventing anything is the shape of the action item itself. The bad postmortem’s defects are detectable at review time:

  • No tracking ID: the item cannot be found in the issue tracker.
  • No owner: the item has no single point of contact.
  • Equal priority: the item cannot be ordered against other work.
  • Ambiguous verb: the item has no verification method.
  • Mitigative only: the item reduces impact but does not prevent recurrence.
  • Delayed publication: the item was written four months after the incident, when details were already fading.

Each of these is a lint rule. Each can be checked before the postmortem is published. None requires a new tool.

The contract model for action items

Treat every action item as a contract with four fields:

  1. Owner. One person, not a team. The workbook says a single owner with multiple collaborators is better than four owners.
  2. Tracking ID. A bug number, a ticket, a Jira key. Something that lives outside the postmortem document and has its own state machine.
  3. Priority. A P0/P1/P2/P3 label that orders the item against other work. The Treynor Sloss rule sets a floor: user-affecting outages need at least one P0/P1.
  4. Verification method. How you will know the item worked. A test, a metric, a game day, a code review checklist. “Improve paging” has no verification method. “Add a page for X condition that fires within Y minutes of Z” does.

If any field is missing, the item is a wish. Wishes decay.

The contract model is not a Google invention. It is a reading of what the workbook’s good postmortem does differently. The good postmortem names owners per action item, assigns priorities, and links to tracking bugs. The workbook’s review criteria include whether the action plan is appropriate and whether resulting bug fixes are at appropriate priority. Those criteria are only checkable if the fields exist.

Measuring decay without inventing numbers

Decay is the rate at which action items stop moving. You can measure it with two counts:

  • Tracking-ID coverage: the fraction of action items in a postmortem that have a tracking ID. A postmortem with zero tracking IDs has zero coverage. The bad postmortem has one tracking bug across four items.
  • Time-to-status-change: for each tracked item, the number of days between the last status change and now. An item that has been “open” for 180 days without a comment, a reassignment, or a priority change is decaying.

Neither count requires a new system. The first is a text search over the postmortem repository. The second is a query against the issue tracker. The SRE Book notes that Google has enhanced its postmortem template with metadata fields to facilitate automated analysis, and that tools to aggregate postmortems help identify common themes across product boundaries. The same metadata that enables trend analysis enables decay measurement.

What you do with the number is a management decision. A team with 40% tracking-ID coverage and a median time-to-status-change of 90 days has a different problem than a team with 90% coverage and a median of 10 days. The first team needs to fix its postmortem review. The second team needs to fix its prioritization.

What I would do differently

I would not add a new tool. I would add a review gate. Before a postmortem is published, the reviewer reads each action item aloud and asks four questions: Who owns this? Where is it tracked? What priority? How will we know it worked? If any answer is missing, the item goes back to the author. The workbook’s review criteria already include whether the action plan is appropriate and whether bug fixes are at appropriate priority. The four questions are a concrete way to apply those criteria.

I would also publish the postmortem within a week, not four months. The workbook notes that the bad postmortem was published four months after the incident, and that if the incident had recurred during that delay, team members likely would have forgotten key details. A timely postmortem captures details that a delayed one loses. The delay is itself a decay signal.

I would not claim that this fixes everything. The satellite diskerase incident recurred three years later despite action items. The action items reduced the blast radius, but they did not prevent recurrence. That is the honest outcome: closing the loop makes the next incident smaller, not impossible.

FAQ

Does every action item need a tracking bug?
The workbook says only one item in the bad postmortem had a tracking bug, and that without a formal tracking process, action items are often forgotten. The Treynor Sloss rule requires at least one P0/P1 bug for user-affecting outages. A reasonable policy is: every action item gets a tracking ID, and every user-affecting postmortem gets at least one P0/P1.

What if the action item is “write a design doc”?
That is still a contract. The owner is the author. The tracking ID is the doc’s ticket or the review request. The priority is the priority of the design work. The verification method is the review approval and the subsequent implementation.

How do I measure decay if my postmortems are in a wiki?
Count the fraction of action items with a tracking ID. That is a text search. If the number is low, the decay is already visible. You do not need a dashboard to start.

Is the Google case study representative?
It is one incident, one recurrence, and one set of action items. It is not a dataset. It supports the claim that closing the loop changes outcomes. It does not support a specific decay rate or half-life.

The heuristic

If you cannot point to a tracking ID and a verification method, you have written a wish, not an action item. Wishes decay. Contracts get closed.