A postmortem should explain how an incident became possible. It should not decide who deserves blame.
When an outage ends, teams are often tired, frustrated, and under pressure to produce an answer quickly. That pressure can turn a review into a search for the person who made the last change. The result may feel decisive, but it rarely makes the service safer.
A useful postmortem studies the conditions around the incident: system design, missing safeguards, unclear ownership, weak signals, incomplete procedures, and decisions made with limited information. It turns those findings into improvements that can be owned and verified.
Blameless does not mean accountability-free
Blamelessness is sometimes misunderstood as avoiding difficult questions. The opposite is true. A strong review examines decisions, assumptions, and actions in detail. It simply avoids treating human error as the final explanation.
Google’s SRE guidance on postmortem culture recommends assuming that the people involved had good intentions and acted using the information available at the time. This allows the review to ask better questions:
- What made the action appear safe?
- Which warning was missing, unclear, or ignored?
- Why could one change create this level of impact?
- Which safeguard failed or did not exist?
- What would help another engineer make a safer decision?
Accountability still matters. Owners must complete corrective work, leaders must fund important fixes, and teams must verify that risks have actually been reduced. Blame focuses on a person. Accountability connects an improvement to an owner and a measurable result.
Define when a postmortem is required
Do not decide whether an incident deserves review based only on emotion or visibility. Establish criteria before the next event.
Useful triggers can include:
- customer-facing downtime or serious degradation;
- data loss, corruption, or security impact;
- manual intervention by an on-call engineer;
- a recovery time above an agreed threshold;
- a monitoring failure that delayed detection;
- a near miss that exposed a serious weakness.
Google’s guidance recommends defining these triggers in advance. A near miss also deserves attention when the only difference between a warning and an outage was luck.
Start with a factual timeline
The timeline is the backbone of the review. Build it from evidence rather than memory alone.
Use deployment records, alerts, traces, logs, status updates, tickets, and the incident record. Capture timestamps in one time zone. Distinguish confirmed facts from hypotheses. Record both technical changes and important coordination decisions.
A useful timeline answers:
- When did customer impact begin?
- When did the team detect the incident?
- What signals were available at that moment?
- Which actions were taken and why?
- When did impact stop?
- How did the team confirm recovery?
The timeline should not become a transcript of every message. Keep the events that explain changes in impact, understanding, decisions, and recovery.
Separate the trigger from contributing conditions
The last change before an outage may be the trigger, but it is rarely the whole explanation. A reliable system should expect people to make mistakes, tools to behave unexpectedly, and dependencies to fail.
Look for contributing conditions across several areas:
- Design: Was the system too tightly coupled or missing isolation?
- Change safety: Were rollout limits, validation, or rollback controls missing?
- Observability: Did the team lack a signal tied to user impact?
- Operations: Was the runbook incomplete, unavailable, or difficult to follow?
- Ownership: Was it unclear who could decide, escalate, or communicate?
- Capacity: Did limited staffing, connectivity, or tool access slow the response?
This deeper view matters for lean teams. If the conclusion is only “be more careful,” the same operating conditions remain in place for the next responder.
Review the response, not only the failure
A postmortem should examine how the team responded. Ask what helped recovery and what created delay.
Review detection, escalation, role assignment, technical mitigation, stakeholder updates, handoffs, and recovery validation. The recent Observability Africa guide on a lean incident command model provides a simple structure for separating coordination, technical response, and communication.
Include what went well. Successful safeguards and good decisions should be understood and preserved. A review that records only failure gives an incomplete picture of the system.
Write action items that can close
Weak action items sound responsible but cannot be verified:
- improve monitoring;
- be more careful;
- review the process;
- add more tests.
A useful action item has an owner, a deadline, a clear outcome, and a way to verify completion. For example: “Add a deployment check that blocks a rollout when the service-level burn rate exceeds the agreed threshold, then test the block in staging.”
Balance prevention and mitigation. Some incidents cannot be eliminated at a reasonable cost. In those cases, improve detection, limit the blast radius, make rollback safer, and reduce recovery time.
Do not create a long list that the team cannot finish. Prioritize the few actions that remove the most risk. Put them into the normal engineering backlog, review their status, and escalate when feature work repeatedly displaces important reliability work.
Protect sensitive information
Postmortems often contain customer impact, internal architecture, screenshots, chat excerpts, or telemetry. Share the learning widely, but do not expose credentials, personal data, customer identifiers, or unnecessary security details.
Create an internal version when detailed evidence is required. Prepare a sanitized version for broader learning or external communication. The goal is transparency with responsible boundaries.
Adapt the process to a lean team
A good postmortem does not need expensive software or a large meeting. A small team can use one shared document and a 45-minute review.
- Assign one facilitator who was not the main technical responder when possible.
- Prepare the timeline before the meeting.
- Agree on impact and contributing conditions.
- Select no more than a small set of high-value actions.
- Assign owners and verification dates.
- Share the lessons with teams that face similar risks.
For distributed teams or environments with unreliable connectivity, keep a plain-text template that works offline. Attach dashboards and transcripts later rather than blocking the first factual record.
Use reviews to find repeated weaknesses
One postmortem improves one incident. A collection of structured postmortems can reveal repeated patterns: missing ownership, unsafe changes, noisy alerts, fragile dependencies, weak capacity planning, or actions that never close.
Use consistent fields for impact, detection method, contributing conditions, affected services, and action status. Review themes quarterly. NIST SP 800-61 Revision 3 places incident response inside ongoing risk management, which supports treating lessons, preparation, response, and recovery as connected work rather than isolated documents.
The review is complete when learning becomes change
A polished document is not the final outcome. The purpose of a postmortem is to improve the system and the organization around it.
Teams build trust when reviews are factual, blameless, and honest. They build reliability when action items are funded, completed, and tested.
Observability Africa welcomes organizations interested in supporting independent reliability education, incident exercises, and community resources while respecting editorial independence. Teams that want a structured review of alerting, incident response, and operational readiness can also request an independent observability audit.