Skip to main content
An incident game day is a scheduled exercise where a team simulates a failure and runs their real incident response process end to end. The point is not to prove the system breaks; it’s to find out whether the people, tools, and playbooks that are supposed to handle a real incident actually work when you use them under pressure. Game days are how strong teams keep their response sharp between real incidents, and how new responders build the muscle memory to prepare for their first real one. This page covers why game days matter, how to plan one, and how to run them in Rootly using test incidents so your drill stays isolated from production reporting, customer-facing status, and paging.

​
Why game days matter

Most incident response programs look healthy on paper: there’s an on-call schedule, a severity matrix, a runbook folder, a retrospective template. The gap between those artifacts and what actually happens at 3 a.m. is where incidents get long and expensive. Game days close that gap by exercising the whole incident process in a controlled setting. Running them regularly:
  • Validates the process end to end. Paging, role assignment, Slack channel creation, status updates, workflows, and the retrospective flow all get exercised together, not one at a time in isolation.
  • Trains new incident commanders and responders. People learn command by commanding. A game day is a safe place to declare, delegate, and make timeboxed calls before a real SEV1 forces it.
  • Surfaces broken runbooks and stale ownership. Docs that no one has opened in a year, dashboards that 404, services with no clear owner: game days find them cheaply.
  • Tests tooling and integrations. Alert routes, escalation policies, workflow automations, status page publishing, and third-party integrations all get real traffic through them.
  • Builds cross-team coordination. Multi-team incidents are the hardest kind. Practicing hand-offs between platform, product, security, and support teams beats discovering the seams live.
  • Reduces mean time to recovery. Teams that drill respond faster, communicate more clearly, and make fewer avoidable mistakes when the real thing happens.

​
Industry best practices

The teams that get value from game days treat them like production incidents: planned, scoped, reviewed, and followed up on. A few patterns hold up across organizations:
  • Start small, then expand. The first game day should exercise the response process against a well-understood failure (a single service degraded, a dependency timing out). Save multi-team, multi-region, or security scenarios until the basics are boring.
  • Write a hypothesis before you start. “If the payments service returns 500s for 10% of requests, on-call is paged within 2 minutes, a SEV2 is declared within 5, and the status page updates within 10.” A game day either confirms the hypothesis or produces a specific finding.
  • Pick one primary objective per drill. Testing a new runbook, onboarding a new commander, validating a status page workflow, and rehearsing a security escalation are four different game days, not one.
  • Announce it, but not everything. Tell responders a game day is happening in a given window so they don’t page a real executive. Don’t tell them the exact scenario, the exact time, or which service will fail. Surprise is where the learning is.
  • Include everyone the real incident would include. On-call engineers, incident commander, communications lead, support, and anyone whose real job appears in your runbooks. A drill that skips the communications lead never tests communications.
  • Have an observer, not just participants. Assign someone to watch the response without joining it. They catch the things responders are too busy to notice, like decisions that stalled or steps that were skipped. AI transcription can help here, but don’t rely on it entirely; human observations are a critical piece of the learning.
  • Run a real retrospective afterwards. Use the same template, the same blameless framing, and the same action-item tracking you use for real incidents. If the game day doesn’t produce follow-through, it was theater.
  • Rotate scenarios. Repeating the same failure teaches the team to solve that failure, not to respond to novelty. Rotate across infrastructure, application, third-party, and security scenarios over time.
  • Keep the scope of impact controlled. Game days can and should touch real systems (that’s what makes them useful), but decide up front what is out of scope, who can call it off, and what “abort” looks like.
  • Measure the same things you measure for real incidents. Time to detect, time to page, time to declare, time to mitigate, time to communicate. Track them across drills to see whether the process is actually getting faster.

​
How to run a game day in Rootly

Rootly is designed to let you exercise the full response flow, including Slack channels, role assignment, workflows, and retrospectives, without polluting incident metrics or paging real customers. The key is using test incidents so the drill is clearly separated from production.

​
1. Plan the drill

Before the day of, write down:
  • Scenario — what fails, how, and when. Keep this to the game day leads.
  • Hypothesis — the specific outcomes you expect (detection time, who pages, which workflows run, which runbook applies).
  • Scope — which services, integrations, and teams are in play, and which are explicitly out.
  • Participants and observers — who is responding, who is watching, who is running the drill.
  • Abort criteria — the conditions under which you stop the game day early, and who has authority to call it.
  • Window — a defined start and end so responders know when “this could be a drill” applies and when it stops applying.
Share the window and the fact that a game day is happening with a broader audience (leadership, adjacent teams, support), so no one confuses the drill for a real event or vice versa.

​
2. Declare a test incident

When you kick off the drill, declare the incident the same way responders would in production, but flag it as a test. In Rootly, the Test Incident checkbox on the New Incident form marks the incident as a drill so it’s clearly labeled everywhere it appears: in the incident list, in Slack, in the timeline, and in analytics. Marking an incident as a test is what keeps game days safe to run against real tooling. Use it for every drill, without exception. See Creating Incidents via Web Interface for the full New Incident form, including where the Test Incident checkbox appears.

​
3. Run the response like it’s real

Once the incident is declared, the response should look identical to a production incident:
  • Roles get assigned (commander, technical lead, communications lead, scribe). See What Is an Incident Commander? for role guidance.
  • Responders investigate in the incident Slack channel and update the timeline as they go.
  • The commander drives status updates on the cadence your runbooks require.
  • Workflows fire the same way they would in production.
The goal is to exercise the muscle. Skipping steps because “it’s just a drill” defeats the purpose. If your workflows post to a real customer status page or page a real executive on SEV1, decide in planning whether to route those to test destinations or leave them live.
Have observers take notes in a separate channel or doc during the drill. In-channel observation tends to turn into participation, which is exactly what you’re trying to avoid.

​
4. Resolve and run the retrospective

Resolve the test incident through the normal lifecycle (mitigated, then resolved) so the timeline captures the full arc. Then run a retrospective using the same template you’d use for a production incident of the same severity. See How to Run an Incident Retrospective for the full flow. Focus the retro on the response, not the scenario:
  • Where did detection, paging, or declaration lag?
  • Which roles were unclear or unfilled?
  • Which runbooks were missing, wrong, or hard to find?
  • Which workflows didn’t fire, fired incorrectly, or produced noise?
  • What decisions stalled, and why?
  • What would the observer have done differently?
Every finding should turn into a tracked action item with an owner and a due date. A game day that doesn’t produce assigned follow-through is a rehearsal that never ships.

​
5. Close the loop

After the drill:
  • Confirm the incident is clearly marked as a test everywhere it surfaced (channel, timeline, analytics, status page if applicable).
  • Review whether your analytics dashboards exclude or segment test incidents the way you want. Test incidents in the same reports as real ones will distort MTTR and volume metrics.
  • Schedule the next game day before the current one is out of memory. Cadence matters more than perfection: a rough drill every quarter beats a perfect one every two years.

​
What game days are not

Game days sit in a family of related practices that teams often conflate. Each one exercises a different part of your resilience posture, and each answers a different question.

​
Game days vs. chaos engineering

Chaos engineering is the practice of injecting controlled failure into a running system to test whether it behaves the way you expect: does the circuit breaker trip, does the failover fire, does the retry back off. It’s a systems experiment with a hypothesis about the technology. A game day is an exercise about the people and process. The failure is the setup; the response is the point. The two pair naturally. A common pattern is to run a chaos experiment as the trigger for a game day: inject the failure, let on-call detect and respond as if it were real, and evaluate both the system’s behavior and the team’s response. But you can run either one without the other, and they have different success criteria: chaos engineering succeeds when the system holds up, a game day succeeds when the response does.

​
Game days vs. tabletop exercises

A tabletop exercise is a discussion-based drill. The team gathers (in a room or a call), someone reads out a scenario, and participants talk through what they would do at each step: who gets paged, what gets declared, what the commander decides, what goes on the status page. Nothing actually happens in the system, and no tooling is exercised. Tabletops are cheap, low-risk, and well-suited to:
  • Onboarding new responders or commanders to the process
  • Rehearsing rare scenarios (major security incident, data loss, regulatory event) where a live drill would be too disruptive
  • Cross-functional exercises that involve legal, communications, or executive stakeholders who don’t otherwise sit in a Slack channel
  • Reviewing a runbook or playbook before committing to it in a live drill
The tradeoff: tabletops don’t catch tooling gaps. A runbook that reads well in a conference room can still point to a dashboard that no longer exists. Treat tabletops as a way to pressure-test thinking, and use game days to pressure-test execution.

​
Game days vs. disaster recovery tests

A disaster recovery (DR) test validates that you can restore service after a catastrophic failure: a region goes down, a database is corrupted, a provider has a multi-hour outage. DR tests exercise backups, restore procedures, failover automation, and recovery time and point objectives (RTO/RPO). They are typically scheduled, heavily coordinated, and scoped narrowly to the recovery mechanism itself. Game days and DR tests overlap but aren’t the same:
  • A DR test asks, “can we recover?” Success is measured against RTO and RPO.
  • A game day asks, “can we respond?” Success is measured against detection, coordination, communication, and decision quality.
Many teams combine them: run a DR test as the scenario for a game day, so the recovery procedure and the incident response around it get exercised together. That combination is powerful, but plan it as one exercise with two sets of objectives, not one drill that happens to cover both by accident.

​
Quick comparison

A mature program uses all four: tabletops to teach the process, chaos engineering to harden the system, game days to exercise the response, and DR tests to prove you can recover. Game days are also not a substitute for real incident practice: real incidents happen at inconvenient hours, involve incomplete information, and carry real consequences. Game days build the reflexes; real incidents keep them honed.

​
Where to go next