Production Incident Kit

Changelog

• 1.0.0, 22 October 2026: first release

Production Incident Kit

Version 1.0.0

A complete incident process you can adopt in an afternoon: a runbook from first alert to close, severity levels with example response targets, a card for the first 15 minutes, message templates for the status page, internal updates and customer emails, a blameless postmortem template, an on-call checklist and a runbook template for each service. Everything is plain Markdown with fill-in fields, so it works with whatever paging, chat and status tools you already use.

Who it's for

• Engineering teams setting up on-call for the first time
• Teams that have on-call but handle each incident differently depending on who's awake
• Engineering managers who need a written process for customers, auditors or new hires
• Support and communications staff who write status updates during incidents

How to use it in 5 minutes

1. Read severity-levels.md and change the response targets to numbers your team can meet.
2. Print or pin first-15-minutes.md for everyone on the rota.
3. Fill in the "Team decisions" section at the bottom of severity-levels.md: paging tool, channel naming, status page.
4. Save communications/status-page-templates.md as canned responses in your status page tool.
5. Pick your most important service and start its runbook from service-runbook-template.md.

Then walk the team through incident-runbook.md once, ideally with a practice incident.

File map

A note on limits

This kit gives you a process and the words to go with it. It can't replace practice: run a game day or a tabletop exercise so the process is familiar before a real incident. The response targets are examples, not recommendations for your business. For security incidents and anything involving personal data, involve your security and legal contacts, because notification rules vary by country and sector.

Licence

MIT License. See LICENSE.txt.

First 15 minutes

Keep this card open on your second screen or printed by your desk. It's the short version of incident-runbook.md for the moment the page goes off.

---

Minute 0 to 2: acknowledge

• Acknowledge the page so it stops escalating and others know someone is on it.
• Open the alert. Read what fired, when, and the linked dashboard.
• Is it real? Check one independent signal: the site in a private window, a second dashboard, the error tracker.

Minute 2 to 5: declare

If customers are or may be affected, declare an incident now. You can always stand it down.

• Create the incident channel: #inc-[date]-[short-name].
• Post the first message:

Incident declared. [What is broken], seen from [time UTC]. Severity: [SEV level], working guess. I'm [name], acting incident lead until someone takes over. Dashboard: [link].

• Set the severity using severity-levels.md. When unsure, go higher.
• For SEV1 or SEV2, page the incident lead rota and a communications person.

Minute 5 to 10: understand just enough

You're not looking for the root cause yet. You're looking for the fastest safe way to stop the harm.

• What changed? Recent deploys, config changes, feature flags, infrastructure changes, provider status pages, traffic spikes.
• How wide? All users or some? One region, one feature, one customer?
• Since when? Find the start of the graph change. It often lines up with a change.
• Write each finding in the channel with the time. This is the start of your timeline.

Minute 10 to 15: mitigate and communicate

Common mitigations, roughly in order of preference:

• Pick the safest mitigation and say it in the channel before you do it, for example: "Rolling back deploy [ID] to [previous ID]. Expect 3 minutes."
• Post the first status page update (communications/status-page-templates.md, "Investigating"). Short and honest is fine.
• Set a time for the next update and put it in the channel.

Don't

• Don't debug alone and silently. Post what you're seeing, even if it's half a thought.
• Don't make several changes at once. You won't know which one helped.
• Don't hunt for the root cause while customers are still affected and a rollback is available.
• Don't blame. Ask "what happened", not "who did this".

Hand over when the lead arrives

Summary: [what's broken] since [time]. Impact: [who and how]. Tried: [actions and results]. Current theory: [theory]. Next step: [step]. Last status update: [time]. Next due: [time].

Numbers to have ready

Incident log

Copy this for each incident, or paste the headings into the incident channel's pinned message. Use UTC for every time. Short factual lines are best.

Summary

Roles

Impact

• Who is affected: [all users / region / plan / named customers]
• What they experience: [errors, slowness, missing data, cannot sign in]
• How many: [estimate and how it was measured]
• Data affected: [none known / describe]
• Money affected: [failed payments, refunds, credits]

Timeline

Types: Detection, Declaration, Finding, Theory, Action, Decision, Severity, Escalation, Comms, Mitigation, Resolution, Handover.

Changes made during the incident

Every change, so they can be reviewed and made permanent or undone.

Communications sent

Open questions and follow-ups

Close-out

• Stable for the agreed period
• Status page resolved
• Final internal update sent
• Temporary changes listed with owners
• Postmortem owner: [name], due [date]

Incident runbook

This is the full process for a production incident, from the first alert to the postmortem. It's written for a team of three to thirty engineers. Smaller teams can combine roles; larger teams can add more.

An incident is any unplanned event that harms, or is about to harm, the service your customers rely on. If you're asking "is this an incident?", treat it as one. Standing one down costs a few minutes; missing one costs much more.

The principles

1. Mitigate first, root cause later. Your first job is to stop the harm. Rolling back a deploy you don't yet understand is a good outcome.
2. One person leads. The incident lead coordinates. They don't have to be the most senior or the person typing commands.
3. Write everything down, as it happens. A timeline written live is accurate; one reconstructed tomorrow is not.
4. Communicate early and on a schedule. Silence feels like neglect to customers and colleagues.
5. Blameless. People act on the information they had at the time. Ask how the system let it happen.

Phase 1: detect

Incidents start from an alert, a customer report, a colleague's message or a provider notice.

• The person who notices checks one independent signal to confirm it's real.
• If customer impact is possible, they declare (phase 2). They don't wait for certainty.
• If it's clearly not customer-affecting, they open a ticket at SEV4 instead.

Good alerts page on symptoms customers feel (error rate, latency, failed checkouts), not on every cause (CPU at 80%). If your team gets paged for things that don't need action, fix the alert after the incident. See on-call-checklist.md.

Phase 2: declare

1. Create a dedicated incident channel, named #inc-[YYYY-MM-DD]-[short-name]. Keep all incident discussion there.
2. Post the declaration: what's broken, since when, initial severity, who is acting lead, link to the main dashboard.
3. Set the severity with severity-levels.md.
4. Page the right people for that severity.
5. Start the incident log (incident-log-template.md) or use the channel as the log, with times on every entry.
6. Open a video or voice bridge for SEV1 and SEV2. Link it in the channel. Decisions made on the call get written in the channel.

Phase 3: assign roles

For SEV1 and SEV2, fill these three roles as soon as possible. For SEV3, one person can hold all three.

Incident lead

Owns the incident until it's closed or handed over.

• Keeps the big picture: what's the impact, what are we trying, what's next.
• Assigns work to named people with a time to report back: "[Name], check the queue depth and report back in 10 minutes."
• Makes decisions when the group is stuck. A reasonable decision now beats a perfect one in an hour.
• Sets and changes the severity.
• Decides when to escalate, bring in other teams, or call vendors.
• Keeps the team from going down rabbit holes: "Is this helping us mitigate?"
• Doesn't do hands-on debugging on a SEV1 if anyone else can. Leading is a full-time job.

Communications

Owns every message that leaves the incident channel.

• Posts status page updates on schedule (communications/status-page-templates.md).
• Sends internal updates to leadership, support and sales (communications/internal-update-template.md).
• Gives support a holding reply for customers.
• Drafts the customer email after resolution if needed (communications/customer-email-template.md).
• Gets technical facts from the lead, not from interrupting the people debugging.
• Reminds the lead when an update is due.

Operations

The people investigating and making changes. There can be several.

• Announce every change in the channel before making it: what, where and how to undo it.
• Make one change at a time, then watch the effect.
• Report findings as they happen, including dead ends ("Database CPU is normal, not the cause").
• Ask the lead before trying anything risky or irreversible.

Optional roles for large incidents

Phase 4: investigate and mitigate

Work in loops of about 10 to 15 minutes: observe, form a theory, try the safest action, check the result, report.

Questions that find most causes:

1. What changed just before it started? Deploys, config, flags, infrastructure, data, traffic, dependencies, certificates, DNS, scheduled jobs.
2. What's the scope? All users, a region, a customer, a feature, a platform?
3. What do the four golden signals say? Latency, traffic, errors, saturation.
4. Is a dependency unhealthy? Check provider status pages and your own dependency dashboards.

Mitigations, safest first:

Once impact stops, say so: "Mitigated at 14:42 UTC. Error rate back to baseline. Monitoring." Move the status page to "Monitoring". The root cause can now be investigated calmly, often the next working day.

Phase 5: keep the timeline

Record, with UTC times:

• When the problem started (from graphs, not when it was noticed)
• When it was detected, and how
• When it was declared, and at what severity
• Each significant finding, theory, decision and action, with who did it
• Severity changes
• Each external update sent
• When impact was mitigated, and when it was fully resolved

Short lines are fine: "14:31 [Name]: rolled back api to v2.41.3." The postmortem will be built from this.

Phase 6: escalate

Escalate when:

• There's no clear mitigation after 30 minutes (SEV1) or 60 minutes (SEV2).
• The incident needs access or knowledge the current responders don't have.
• Impact is growing.
• There is any sign of a security cause. Bring in your security lead and switch to evidence-preserving actions.
• A vendor is the cause. Open a support case at the highest priority your contract allows, and post the case number in the channel.
• People are tired. Plan handovers every few hours on long incidents.

How to escalate: page the next person on the escalation list, tell them the severity, the impact and what you need from them in one message.

Phase 7: hand over

On long incidents, or when the lead changes:

Handing over incident lead to [name] at [time]. Impact: [summary]. Status: [mitigated / not mitigated]. Current work: [who is doing what]. Next status update due: [time]. Open questions: [list].

The new lead confirms in the channel: "I have the lead."

Phase 8: resolve and close

Resolved means customers are no longer affected and the system is stable. It doesn't mean the cause is fixed for good.

Before closing:

• Metrics have been at normal levels for an agreed period (for example 30 minutes for SEV1).
• Temporary mitigations are listed, with owners to make them permanent or remove them.
• The status page shows "Resolved" with a short summary.
• Internal stakeholders have a final update.
• Support knows what to tell customers who ask.
• A postmortem owner and due date are set (SEV1 and SEV2).
• The incident log is saved where the postmortem author can find it.
• Thank the people who helped. Note who was woken up, and make sure they can take time back.

Post the closing message in the channel:

Resolved at [time UTC]. Duration: [x]. Impact: [summary]. Postmortem owner: [name], draft due [date]. Thanks all.

Phase 9: learn

Write the postmortem with postmortem-template.md, review it as a team and track the action items to completion. An incident without follow-up is likely to happen again.

Quick reference

On-call checklist

On-call works when the person holding the pager has what they need, knows what's going on, and isn't worn out. This checklist covers a shift from start to finish, plus what the team owes the people on call.

Before your shift

Access

• You can sign in to everything you might need, and you've checked this week, not assumed: cloud console, deploy tool, dashboards, logs, error tracker, database read access, status page admin, paging tool.
• MFA devices are with you and charged. Backup codes are somewhere you can reach.
• VPN or zero-trust access works from where you'll be.
• You can roll back the main services. You know the command or button, and you have the permissions.

Devices and setup

• Paging app installed, notifications allowed, set to override silent and do-not-disturb modes.
• Test page sent to yourself and received.
• Laptop charged, with a charger. A phone hotspot or second internet option if your connection is unreliable.
• Your contact details in the paging tool are current.

Knowledge

• Read the previous shift's handover notes.
• Skim incidents and alerts from the last week.
• Know what's being deployed or changed this week, and any planned maintenance.
• Know where the service runbooks are, and skim the ones for services that have been noisy.
• Know who your secondary and escalation contacts are, and how to reach them.

Life

• Your plans let you respond within the target time: within reach of your laptop, in signal, sober enough to make production changes.
• If something comes up, arrange a swap early and update the schedule in the paging tool. Telling someone in chat isn't enough.

Handover

At the start and end of every shift, ideally on a short call.

The outgoing person shares:

• Open incidents and their state
• Alerts that fired and what was done
• Anything flaky or being watched ("the queue backs up around 02:00 UTC; it has cleared itself so far")
• Changes planned during the next shift
• Temporary mitigations still in place, and when they need attention
• Tickets raised that need follow-up

Handover note template:

On-call handover: [outgoing] to [incoming], [date HH:MM UTC]

Open incidents: [none / list with links]
Pages this shift: [number], [short summary of each]
Watch list: [things that might fire, and what to do]
Planned changes: [deploys, maintenance, migrations with times]
Temporary fixes in place: [what, where, when to revisit]
Follow-ups raised: [tickets]
Anything else: [notes]

During your shift

When a page arrives

• Acknowledge within the target time, even if you need a minute to get to your laptop.
• Follow first-15-minutes.md.
• Declare an incident if customers are or may be affected. Don't wait until you're sure.
• If you're stuck after 15 to 30 minutes, escalate. Asking for help early is expected, not a failure.

For every alert, even ones that need no action

• Note it in the on-call log: time, alert, what you did, whether it was useful.
• If it didn't need action, raise a ticket to fix or remove the alert.

Changes

• Avoid risky changes while on call unless they're needed to fix an incident.
• Announce any production change in the team channel before you make it.

Look after yourself

• If you were up at night handling an incident, start late or take time back the next day. Tell your manager; don't wait to be offered.
• If the load is too much, say so during the shift, not at the end of it.

After your shift

• Write and send the handover note.
• Make sure every incident has a log and, where needed, a postmortem owner.
• Raise tickets for noisy alerts, missing runbooks and access problems you hit.
• Note your pages and out-of-hours time in the on-call log for the team's review.
• Take any time back you're owed.

What the team owes people on call

Review these monthly or quarterly.

• A rota with enough people that nobody is on call more than about one week in four, with primary and secondary cover.
• Alerts that are actionable. Track pages per shift; if most pages need no action, fix the alerts before adding people.
• A runbook for every alert that can page (use service-runbook-template.md).
• Access granted before the first shift, not during the first incident.
• New people shadow a shift before taking one alone, and do a reverse-shadow (they lead, an experienced person watches).
• Compensation or time off for on-call, agreed and written down.
• A regular review of pages, incidents and on-call load, with actions.

On-call log

Postmortem: [Incident title]

Incident ID: [INC-2026-0001] Severity: [SEV1 / SEV2] Date of incident: [YYYY-MM-DD] Author: [Name] Reviewers: [Names] Status: [Draft / In review / Final] Review meeting: [Date and time]

---

How to use this template

• This is a blameless review. We assume everyone acted reasonably with the information, tools and time they had. The goal is to find how the system (code, process, tooling, incentives) allowed this, and change it.
• Write about what happened and why, not who made a mistake. Use roles ("the on-call engineer") rather than names in the narrative, except in the timeline where names help accuracy.
• Draft within the deadline in severity-levels.md while memories are fresh. Build the timeline from the incident log and channel, not from memory.
• Delete these instructions before sharing.

---

Summary

[Three to five sentences a busy executive could read: what happened, how long, who was affected, what fixed it, and the main change we're making.]

Impact

Timeline

All times UTC. Include the lead-up (the change that introduced the problem may be days earlier), detection, key decisions and resolution.

What happened

[A narrative in plain language, a few paragraphs. Explain the system as much as a reader from another team needs. Describe the sequence of events and the state of the system at each point.]

Contributing factors

Incidents rarely have one root cause. List the conditions that together allowed this to happen and made it worse. For each, ask "why?" until you reach something you can change.

Factor types to consider: code, configuration, deploy process, testing, monitoring and alerting, capacity, dependencies, documentation, access and permissions, communication, on-call load, time pressure.

Detection

• How did we find out? [Alert / customer / colleague / vendor]
• Could we have found out sooner? How?
• Did the alert go to the right people?

Response

• What helped us respond quickly?
• What slowed us down? (Access, unclear ownership, missing runbook, noisy alerts, tooling)
• Was the severity right? Was it changed at the right time?
• Did communication (status page, internal updates) go out on time?

What went well

• [For example: rollback took 3 minutes once decided]
• [For example: support had a holding reply within 10 minutes]
• [For example: the new dashboard showed the error spike by region straight away]

What didn't go well

• [For example: the first alert was routed to a channel nobody was watching]
• [For example: two people made changes at the same time without announcing them]
• [For example: the status page wasn't updated for 50 minutes]

Where we got lucky

[Things that limited the impact by chance rather than by design. These are often the most useful action items.]

• [For example: the problem started outside peak hours]
• [For example: an engineer who knew the old system happened to be online]

Action items

Each item is specific, has one owner and a due date, and is tracked in [your issue tracker]. Prefer a few meaningful items over a long list nobody finishes.

Types: Prevent (stop it happening), Detect (find it sooner), Mitigate (reduce impact or speed recovery), Respond (improve the process).

Lessons

[Two or three sentences other teams could learn from, even if they don't use this system.]

Supporting material

• Incident channel: [link]
• Incident log: [link]
• Dashboards and graphs: [links or screenshots]
• Related postmortems: [links]

---

Review checklist

Before marking this final:

• Timeline is built from records and checked by people who were there.
• Impact numbers are measured, with the method stated.
• Language is blameless: no "should have", no individual named as a cause.
• Contributing factors go beyond the trigger to the conditions that allowed it.
• Every action item has one owner, a due date and a ticket.
• Reviewed in a meeting with the responders and the owning team.
• Shared with [audience: engineering, leadership, the whole company].
• Customer-facing summary written, if promised on the status page.
• A date is set to check action items are done: [date].

Runbook: [Service name]

One runbook per service. Write it for the on-call engineer at 03:00 who has never touched this service: short steps, exact commands, links that work. Review it after every incident that involved this service, and at least every [6] months.

Owner team: [Team] Escalation contact: [Name or rota], [how to reach] Last reviewed: [Date] by [Name] Repository: [link] Dashboard: [link] Logs: [link with useful filters saved] Error tracker: [link]

---

1. What this service does

[Two or three sentences. What it does, who uses it, and what customers notice if it breaks.]

Customer impact if down: [For example: "Customers can't check out. Browsing still works."] Usual severity if down: [SEV1 / SEV2], see severity-levels.md

2. How it fits together

Where it runs: [Cloud, region, cluster, number of instances] How it's deployed: [Tool and link]

3. Health checks

4. Alerts

One section per alert that can page. The alert's description should link straight here.

Alert: [High error rate]

• Means: [More than 5% of requests failed for 5 minutes]
• Customer impact: [Some checkouts fail]
• First checks:

1. Was there a deploy in the last hour? [Link to deploy history]
2. Is the database healthy? [Link]
3. Is the payments provider reporting problems? [Status page link]

• Mitigations:
• Recent deploy: roll back (section 5).
• Database: see [database runbook link].
• Provider: enable [fallback / queue mode] (section 6).
• Escalate if: not improving within [20] minutes.

Alert: [Queue backlog]

• Means: [More than 10,000 jobs waiting for 10 minutes]
• Customer impact: [Order confirmation emails delayed]
• First checks:

1. Are workers running? [Command]
2. Are workers erroring? [Log link with filter]
3. Has incoming volume spiked? [Dashboard link]

• Mitigations:
• Scale workers: [command]
• Pause non-urgent producers: [command or flag]
• Escalate if: backlog still growing after scaling.

5. Common procedures

Roll back a deploy

[exact command or steps, for example: deploy-tool rollback [service] --to [previous-version]]

• Takes about [x] minutes.
• Check: [what to watch to confirm it worked].
• Can't roll back if: [a database migration ran that the old version can't read; see section 7].

Restart

[exact command]

• Safe to do at any time? [Yes / No: describe the impact]

Scale up or down

[exact command]

• Normal count: [N]. Maximum safe count: [N], limited by [database connections / cost / quota].

Turn a feature off

6. Degraded modes and fallbacks

[How the service can keep partly working when something breaks. For example: "If the payments provider is down, set payments.queue-mode to on. Orders are accepted and charged when the provider recovers. Tell support."]

7. Data

• Main data stores: [names, links]
• Backups: [how often, where, retention]
• Restore procedure: [link]. Expected restore time: [x]. Data loss window: up to [x] minutes.
• Migrations: [how they're run, whether they're backwards compatible, how to tell if one ran]
• Never do this in production without a second person: [for example: run manual SQL updates, truncate tables, delete queues]

8. Known issues and gotchas

• [For example: "The health check passes even when the payments connection is broken. Check the error rate too."]
• [For example: "Restarting all instances at once causes a thundering herd on the cache. Restart in batches of 3."]
• [For example: "Logs before 10:00 UTC each day are in the archive bucket."]

9. Contacts

10. Change history

Severity levels

A severity level is a shared shorthand for "how bad is this and how fast do we move". Agree the levels before you need them, so nobody argues about labels while customers are waiting.

The response targets below are examples to adapt. Set numbers your team can actually meet, at the hours you actually cover. A target nobody meets is worse than a modest one people trust.

The levels

When in doubt, pick the higher level. Downgrading later costs nothing. Starting too low means the wrong people aren't there when it gets worse.

Response targets (examples)

Notes:

• "Acknowledge" means a person has seen the page and is looking, not that it's fixed.
• Public update times start when the incident is declared, not when it started.
• A SEV1 or SEV2 that runs outside covered hours still gets the full response. If you don't have 24/7 cover, say so in your SLA and status page, and set an out-of-hours phone tree for SEV1.

Who decides

• Anyone can declare an incident and set an initial level. Nobody should need permission to raise the alarm.
• The incident lead confirms or changes the level once they're in place.
• Change the level whenever the facts change, and say so in the incident channel: "Raising to SEV1: data loss confirmed in orders table."

Escalation triggers

Raise the level, or bring in more people, when:

• The incident has run for 30 minutes (SEV1) or 60 minutes (SEV2) with no clear mitigation in sight.
• Impact is spreading: more customers, more regions, more features.
• The cause is outside the on-call team's knowledge or access.
• There is any sign of a security cause.
• A customer with contractual commitments is affected and asking.
• Responders are tired. Incidents running longer than about four hours need a planned handover.

Customer-facing commitments

If you have SLAs or published uptime commitments, map them here so responders know which incidents count:

After the incident

Team decisions

Fill these in and keep them up to date.

• Paging tool: [name]
• Incident channel naming: [for example #inc-2026-10-22-checkout]
• Status page: [URL]
• Incident lead rota: [link or names]
• Out-of-hours phone tree for SEV1: [link]
• Last reviewed: [date] by [name]

Customer email templates

Send a direct email after an incident when the status page isn't enough: customers lost data or money, need to take action, have contractual commitments, or were affected for a long time. Most incidents don't need one. When in doubt, ask whether a customer would be surprised not to hear from you.

Before you send

• The incident is resolved, or the email clearly says it isn't.
• Every fact is confirmed: times, scope, what was and wasn't affected.
• Legal or privacy has reviewed it if personal data, security or contractual credits are involved.
• Support knows it's going out and has the same wording.
• It's going only to affected customers, unless the incident affected everyone.
• It's sent from a monitored address, and replies go to support.

Writing rules

• Lead with the apology and the facts, not with background on how much you care about reliability.
• Say what happened in plain words, at the level a customer needs.
• Be clear about what they need to do, if anything. Put it near the top.
• Say what you're changing. Specific steps are believable; "we're taking this very seriously" isn't.
• One person's name at the bottom, ideally someone senior enough to mean it.
• Keep it under 300 words if you can.

---

Template: service disruption

Subject: [Service] disruption on [date]: what happened

Hi [first name or team name],

On [date], between [HH:MM] and [HH:MM UTC], [service or feature] [was unavailable / returned errors / was slow] for [all customers / customers in [region] / some customers, including your account]. I'm sorry. We know you rely on [service] for [what they use it for], and we let you down.

What happened

[Two or three plain sentences. For example: "A configuration change we made to our database servers reduced their capacity. When traffic rose in the afternoon, requests began to fail. We reversed the change, and service recovered by [HH:MM UTC]."]

What this means for you

[For example: "No data was lost. Any requests that failed during this time can be retried." Or list what was affected.]

What you need to do

[Nothing. / Please [action] by [date]. / If you see [symptom], contact us at [link].]

What we're changing

- [Specific change, for example "Configuration changes to database servers now go through a staged rollout."]

- [Specific change]

- [Specific change]

[If applicable: "Under your agreement, this incident qualifies for a service credit of [amount or percentage], which will be applied to your next invoice. You don't need to do anything."]

If you have questions, reply to this email or contact [support link].

[Name]

[Title], Example Co

Template: action required

Use when customers must do something, such as retry payments, re-upload files or check data.

Subject: Action needed: [what they need to do] after the [date] incident

Hi [first name],

On [date], between [HH:MM] and [HH:MM UTC], an issue with [feature] meant that [what went wrong for them]. We're sorry.

Please [action] by [date]. [Step-by-step instructions, or a link to them.]

[Which records or items are affected, or how to find them: "You can see the affected [orders] in [location], marked [label]."]

If you don't take action, [what happens, honestly].

We've fixed the underlying problem and [what you're changing to stop it happening again].

If you need help, reply to this email and our team will help you directly.

[Name]

[Title], Example Co

Template: data affected

Use only after legal and privacy review. Data protection laws may set the content and timing of these notices.

Subject: Important information about your [Example Co] account

Hi [first name],

We're writing to tell you about an incident that affected some of your information.

What happened: On [date], [plain description, for example "a software error caused some account details to be visible to other signed-in users for about 40 minutes"].

What information was involved: [Exactly which fields, for example "your name and email address. Passwords and payment details were not affected."]

What we've done: [Fixed the error, removed the exposure, informed the relevant authority if applicable, and any other steps.]

What you can do: [Specific advice, for example "Be cautious of emails claiming to be from us that ask for your password. We will never ask for it." Or "You don't need to take any action."]

For more information: [Contact details, a dedicated page or reference number.]

We're sorry this happened.

[Name]

[Title], Example Co

After sending

• Log who received it and when, against the incident.
• Watch replies and route them to support or the account owner.
• Add any repeated questions to the support team's notes.

Internal update template

For leadership, support, sales, account managers and anyone else inside the company who needs to know but isn't in the incident channel. Send it to one place (an incidents channel or mailing list) so people stop asking the responders directly.

Internal updates can be more detailed and more candid than the status page, but keep them short enough to read on a phone in 30 seconds.

When to send

Template

[SEV level] [Status: Investigating / Identified / Monitoring / Resolved] - [Short title]
Update [number], [HH:MM UTC]

Impact: [Who is affected and what they experience. Numbers if known, and how sure we are.]

Current status: [One or two sentences on what we know and what we're doing.]

Customer comms: [Status page posted at HH:MM / Not public yet, because ... / Support has a holding reply.]

What we need: [Nothing / Decision on X from Y / Help from team Z / Approval to contact customer A.]

Next update: [HH:MM UTC]

Incident lead: [Name]   Channel: [#inc-...]   Status page: [link]

Example

SEV2 Identified - Delayed email notifications
Update 2, 15:10 UTC

Impact: Since about 14:20 UTC, password reset and receipt emails are delayed by up to 40 minutes for all customers. No emails are lost; they are queued.

Current status: Our email provider is rate-limiting our account after a sending spike. We've paused the marketing send that caused it and asked the provider to lift the limit.

Customer comms: Status page updated at 15:05. Support has a holding reply for password reset questions.

What we need: Nothing right now.

Next update: 16:00 UTC

Incident lead: [Name]   Channel: #inc-2026-10-22-email   Status page: [link]

Final update template

RESOLVED - [Short title]
[HH:MM UTC]

Duration: [start] to [end] UTC ([x] hours [y] minutes)

Impact: [Final summary: who, what, how many, any data or money affected.]

Cause (initial): [One or two sentences. Say it's initial; the postmortem will confirm.]

Fix: [What stopped the impact.]

Follow-up: [Temporary changes to clean up. Customers to contact. Credits to consider.]

Postmortem: [Owner], draft due [date].

Thanks to [names or teams].

Notes for support and account teams

Include these in the first update when customers are affected:

• What to tell customers: [one or two sentences that match the status page]
• What not to say: guesses at cause or fix time, anything about other customers
• Workaround, if any: [steps]
• Where to send escalations: [channel or person]
• Tagging: tag related tickets with [incident ID] so they can be followed up after resolution

Status page templates

Short, honest and on time beats detailed and late. Customers want to know three things: do you know, what's affected, and when will they hear more.

Writing rules

• Say what customers experience, not what broke inside: "Some payments are failing" rather than "Elevated 5xx on payments-api pods".
• Be specific about scope: which feature, which region, which customers, if you know.
• Give a next update time and keep it, even if the update is "no change, still working on it".
• Don't guess at causes or fix times in public. "We expect to have this fixed within the hour" becomes a broken promise if you're wrong.
• Don't blame a vendor, a customer or a person. "A third-party provider" is fine when it helps customers understand.
• Don't minimise. Avoid "minor", "brief" or "a small number" unless you're sure. Customers who are fully blocked read that as dismissive.
• Use plain words. No internal names, ticket numbers or jargon.
• Use UTC, or UTC plus your main customers' time zone, and always say which.
• Keep the same incident open from start to finish, adding updates, so the history reads as one story.

Fill-in fields are in [square brackets]. Delete any sentence you can't fill truthfully.

---

Investigating

Use as soon as the incident is declared and customers may be affected.

Title: [Feature or service] [errors / slowness / unavailable]

We're investigating reports of [errors / slow responses / failures] when [customers do what, for example "signing in" or "loading dashboards"]. [Customers in [region] / Some customers / All customers] may be affected. We'll post an update by [HH:MM UTC].

Shorter version for the first few minutes:

We're investigating [errors when signing in]. Next update by [HH:MM UTC].

Identified

Use when you know the cause or have a clear fix underway.

We've identified the cause of [the errors when signing in] and are [rolling back a recent change / applying a fix / moving traffic to healthy servers]. [Customers may still see errors until this is complete.] Next update by [HH:MM UTC].

If you have a workaround:

While we work on a fix, you can [use the mobile app / retry the request / use the export from Settings instead].

Monitoring

Use when the fix is in place and you're watching to confirm it holds.

A fix has been applied and [sign-in / payments / dashboards] [are working normally again / are recovering]. We're monitoring to make sure the issue is fully resolved. [If you still see problems, please [retry / refresh / contact support at [link]].] Next update by [HH:MM UTC].

Resolved

Use when impact has ended and the system is stable.

This issue is resolved. Between [HH:MM] and [HH:MM UTC], [some / all] customers [could not sign in / saw errors when paying / saw delayed data]. [Data was not lost / Delayed [emails / jobs] have now been processed / Affected [payments] can be retried.] We're reviewing what happened and will take steps to prevent it happening again. We're sorry for the disruption.

If you'll publish a summary:

We'll publish a summary of what happened and what we're changing by [date].

If customers need to act:

If you [made a payment / submitted a form] between [HH:MM] and [HH:MM UTC], please [check / retry / contact support at [link]].

Scheduled maintenance

Post at least [3 to 7] days ahead for disruptive work, and remind customers [24 hours] before.

Title: Scheduled maintenance: [system]

On [day, date] from [HH:MM] to [HH:MM UTC], we'll be carrying out maintenance on [system]. During this time, [describe the impact: "the dashboard will be read-only" / "the API may return errors for up to [5] minutes" / "no impact is expected"]. [No action is needed from you. / Please [action] before [time].] We'll update this page when the work starts and when it's complete.

When it starts:

Maintenance is now in progress. [Impact reminder.] We'll update this page when it's complete.

When it's complete:

Maintenance is complete and [system] is working normally. Thank you for your patience.

If it overruns:

Maintenance is taking longer than planned. [Impact reminder.] We now expect to finish by [HH:MM UTC] and will update this page by then.

Third-party outage

Use when a provider you depend on has a problem that affects your customers.

Title: [Feature] affected by a third-party outage

[Feature, for example "card payments" or "email notifications"] [is unavailable / is delayed / may fail] because of an outage at one of our service providers. We're monitoring their progress [and have [switched to a backup provider / paused [jobs] so nothing is lost]]. [Other features are working normally.] Next update by [HH:MM UTC].

When the provider recovers:

Our provider has resolved their issue and [feature] is working normally again. [Delayed [emails / jobs] are being processed now and should complete by [HH:MM UTC].]

Naming the provider is a judgement call. It helps customers if they use the same provider and can see its status page; it can read as passing blame. Either is fine; be consistent.

Partial or regional impact

[Feature] is [unavailable / slow] for customers in [region / on [plan] / using [integration]]. Other customers aren't affected. Next update by [HH:MM UTC].

Security-related incidents

Don't post detail about an active security incident on a public status page until your security lead and legal contact agree the wording. A neutral holding message is fine:

We're investigating an issue affecting [feature]. [Feature] [is temporarily unavailable] while we do this. Next update by [HH:MM UTC].

Update schedule by severity

Adjust to match severity-levels.md.