> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rootly.com/llms.txt
> Use this file to discover all available pages before exploring further.
# AI SRE Hypothesis Testing
> How Rootly AI SRE drafts competing causes, tests them against live evidence, challenges the leading one, and settles the outcome by rule.
Rootly AI SRE reaches a root cause by drafting competing explanations, testing each one against evidence from your connected tools, and trying to break the strongest one before the result is final. Use this page to judge how much weight a verdict deserves and where to push back.
Four rules hold at every stage:
* **Findings cite evidence.** Code discards any hypothesis, piece of evidence or timeline entry that doesn't cite an observation from this investigation.
* **Code settles the outcome.** The model writes the verdict and scores each hypothesis, but code decides the outcome from the recorded evidence and caps confidence when no evidence is decisive.
* **It abstains rather than guesses.** When nothing clears the bar, the result says so and points to the strongest remaining lead.
* **Guidance isn't evidence.** Instructions steer where AI SRE looks, and earlier conclusions come back as hypotheses to recheck.
## How an Investigation Reasons
The steps below follow the default flow. [Stages You See While It Runs](#stages-you-see-while-it-runs) shows how the six-stage flow differs.
Before any model reasoning starts, AI SRE reads what Rootly already knows about the alert or incident. It looks up each service or resource the alert or incident names in the [Knowledge graph](/ai/atlas/knowledge-graph) by exact match. A name that fits several entities comes back as a list of candidates instead of a guess. For an alert, it identifies the monitor that fired, and when Datadog is connected and the alert names a metric, it pulls that metric's recent series. Your instructions apply to every reasoning stage that follows. When Memory is on, those stages can also recall relevant notes.
Next, AI SRE measures the symptom without naming a cause. It checks the signal's shape (a step, ramp, spike, flap or steady climb) and when the symptom actually started, which can differ from when the alert fired. It also checks whether one host, task or tenant is affected or the whole fleet, what changed near onset, which errors appeared and how dependencies looked. A query that ran and came back empty counts as a finding. A question no connected tool can answer becomes a blind spot, and later stages are told about it.
AI SRE drafts one hypothesis for each failure axis the evidence makes plausible: a change, configuration, capacity, contention, a dependency, data or infrastructure. Each hypothesis must cite something already observed, and code discards any that don't. Starting from measured facts keeps the investigation from anchoring on the alert's wording. When no connected tool can examine an axis, such as recent changes with no deploy source connected, AI SRE is directed to report the gap instead of raising a hypothesis it can't test.
Each hypothesis gets its own line of inquiry, and several run at once, up to a fixed limit. Each line is set up to run the hypothesis's decisive test first: the reading that would confirm or rule it out. Evidence is weighted decisive, supporting or contextual, and code discards evidence that doesn't cite an observation. The report shows decisive items as **Decisive** and all others as **Supporting**. A line that finds no cited evidence leaves its hypothesis inconclusive.
A scoring pass that can't call tools marks each hypothesis supported, refuted or inconclusive, records whether its decisive test ran, and gives it a confidence score. If nothing is supported and a decisive test refuted at least one hypothesis, AI SRE drafts a fresh round and is told not to raise the ruled-out ones again. For each hypothesis still standing whose decisive test ran, AI SRE then tries to name the specific query, endpoint, lock, setting or commit behind the failure.
Each hypothesis still standing faces an adversarial pass that argues against it. The pass tests timing (did the cause precede the onset, not only the alert), mechanism (does it explain the shape of the symptom), coincidence (would this evidence look as incriminating on a normal day), better explanations (does a competing hypothesis explain the same evidence and more), and unchecked consequences (what else would have to be true). It can lower confidence or refute the hypothesis, and it raises confidence only when the attack fails. Code then caps the confidence of any hypothesis without decisive evidence.
A final writer that can't call tools composes the summary, the narrative of what happened, and a timeline that marks each event established, suspected or ruled out. It's told to state only facts already in the evidence, and code discards any timeline entry that cites nothing. The writer doesn't set the outcome or the confidence. Code settles both afterward.
### Stages You See While It Runs
The **AI SRE** tab shows the current stage while an investigation runs. By default, the stages are **Gather facts**, **Triage signal**, **Draft hypotheses**, **Investigate**, **Score evidence**, **Attribute cause**, **Challenge findings** and **Write verdict**.
Rootly can set the selected team to a six-stage flow instead. It measures how far the signal moved from normal. In its first round it works two parallel tracks, one breaking down the alerting signal and the other reading changes, demand and dependencies. Later rounds use one track. It reviews the verdict after writing it, and the review can send the investigation back for another round.
| Step | Default Stages | Six-Stage Flow |
| ------------------ | ------------------------------------------ | ------------------------------------------------- |
| Establish context | **Gather facts** | **Gather facts** |
| Measure the signal | **Triage signal** | **Measure the alert** or **Measure the incident** |
| Draft and test | **Draft hypotheses**, **Investigate** | **Investigate** |
| Score | **Score evidence**, **Attribute cause** | **Combine findings** |
| Challenge | **Challenge findings**, before the verdict | **Review the verdict**, after the verdict |
| Write the verdict | **Write verdict** | **Write verdict** |
## What Counts as Evidence
Evidence is an observation: the result of a tool call AI SRE made during this investigation. Everything else is context that shapes where it looks. The Knowledge graph and Memory are parts of [Atlas](/ai/atlas/overview), Rootly's AI layer, currently used by AI SRE investigations.
* **Readings from your running systems confirm a root cause.** A root cause needs decisive evidence from tools that observe your systems at run time, such as metrics, logs and traces.
* **Corroboration matters.** In the default flow, a root cause needs decisive evidence from at least two different tools that observe your running systems. In the six-stage flow, a root cause that rests on one such tool is held below **High confidence**.
* **An absence can be evidence.** A query that ran and returned nothing, such as no deploys to a service in the window, rules out an axis. A query that couldn't run because an integration is missing or a credential was refused is a blind spot, and the verdict says so.
* **Instructions aren't evidence.** Your [instructions](/ai/ai-sre/instructions), and the instruction on a matched [investigation rule](/ai/ai-sre/investigation-rules), reach every reasoning stage. They steer where AI SRE looks and how it reports, and they never license a cause the cited evidence doesn't show.
* **Earlier conclusions are hypotheses to recheck.** When [Memory](/ai/atlas/memory) is on, AI SRE can recall **Active** notes that your team or earlier investigations recorded. AI SRE is told to verify each note against live evidence, and never to conclude a root cause or declare a false positive from a note alone.
Instructions, the Knowledge graph and Memory may not be enabled for every team. Your Rootly account team turns them on.
## How the Outcome Is Decided
The model doesn't choose the outcome. After the verdict is written, code applies fixed checks to the recorded evidence.
In the default flow, the result is **Root cause identified** only when the leading supported hypothesis meets every condition:
* Its confidence clears a fixed bar after the challenge.
* Its decisive test ran.
* Its decisive evidence comes from at least two different tools that observe your running systems.
* A specific cause was attributed, such as a query, endpoint, lock, setting or commit.
A hypothesis without decisive evidence has its confidence capped below that bar, so it can't become the root cause however convincing it reads. When no hypothesis meets every condition, the **AI SRE** tab shows **Inconclusive — needs human**.
In the six-stage flow, the writer proposes what kind of conclusion the evidence supports. The reviewer can downgrade that conclusion, or upgrade it when the cited evidence already earns more. Code then maps the conclusion to an outcome and caps confidence by how well the evidence is corroborated. A verdict that cites nothing, makes a causal claim without readings from your running systems, or cites only the alerting signal itself becomes inconclusive. This flow can also end as **Contributing factor identified** or **Blocked**. In this flow, **Root cause identified** can also mean the evidence shows the monitor itself misfired or is badly tuned, rather than a fault in your systems.
The **AI SRE** tab shows a confidence tier only on a **Root cause identified** result. For every outcome label and where confidence appears, see [Reading the Result](/ai/ai-sre/reading-the-result).
## When AI SRE Abstains
The verdict writer is told that reaching for a conclusion it didn't earn is worse than reaching none, and never to present a refuted or untested hypothesis as the cause. It's also told to name what it couldn't examine, because "couldn't look" and "looked and found nothing" point to different next steps.
* **Nothing cleared the bar.** The result reads **Inconclusive — needs human**. When a leading hypothesis exists, the report shows it as the **Suspected area**, marked **Suspected, not confirmed**.
* **No tool call succeeded.** The run's outcome is **Could not investigate**. In the default flow, the **AI SRE** tab can show **Inconclusive — needs human** for such a run; see [Reading the Result](/ai/ai-sre/reading-the-result).
* **No hypothesis could cite evidence.** In the default flow, the run stops and shows **Failed** instead of guessing.
* **Access was refused.** In the six-stage flow, when a provider refuses AI SRE's credentials or no connected tool can read the alert's data source, an otherwise inconclusive result can read **Blocked**. The next step tells you to re-authorize that connector or connect a tool that can read the data.
## Investigation Limits
Each investigation runs inside fixed budgets, so it finishes predictably.
| Limit | What It Does |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Tool calls per stage** | Each stage has its own limit on tool calls and model turns. When a stage spends its tool budget, AI SRE is told to stop measuring and report what it established. |
| **Overall model usage** | Each investigation has a cap on model usage across all stages. Stages that call tools stop at the cap, and the verdict can still be written after it. |
| **Rounds** | Triage repeats until its questions are settled or it stops making progress, up to a fixed number of rounds. Fresh rounds of hypotheses, and review rounds in the six-stage flow, are capped too. |
| **Parallel work** | Only a fixed number of hypotheses are tested at once. |
| **Time window** | Each investigation centers on a window that opens shortly before the alert or incident started and closes shortly after it, and never extends past the moment the investigation began. AI SRE treats anything outside the window as a separate event unless it can show a link, and it can widen a reading when a slow climb began earlier. A run started while the problem unfolds sees less of it. |
Because the search is bounded, an inconclusive result is still a completed search. Its blind spots and open questions tell you which source would have settled it.
## Related Pages
How AI SRE investigates, the outcomes it reports, and its settings.
Outcome labels, confidence, evidence and the causal chain in a finished report.
What an investigation reads, and the access boundaries around it.
Team guidance that steers where AI SRE looks and how it reports.
Decide which alerts AI SRE investigates and which rule instruction applies.
The services, infrastructure and relationships AI SRE looks up first.
Notes AI SRE recalls as hypotheses to recheck against live evidence.
## Frequently Asked Questions
The time window stays anchored on when the alert or incident started, so a later run centers on the same period. A run started while the problem was still unfolding sees less of it. Between runs, your tools can also return data that hadn't arrived when the earlier run looked, and a teammate can change connectors, instructions, investigation rules or Memory. **Rerun investigation** starts a fresh run and keeps the earlier results available. It may not be enabled for every team; your Rootly account team turns it on.
Code applies the outcome checks to the recorded evidence, and the narrative's wording carries no weight. In the default flow, a root cause needs a decisive test that ran, decisive evidence from at least two tools that observe your running systems, and a specific attribution. If only one tool saw it, or its decisive test couldn't run, the result stays **Inconclusive — needs human** and the lead shows as the **Suspected area**. Giving AI SRE a second way to observe the same system, such as traces or logs alongside metrics, gives the next investigation a way to corroborate it.
No. Every reasoning stage is told that an instruction isn't evidence and never licenses a cause the cited evidence doesn't show, and the outcome checks count only observations. An instruction can point AI SRE toward a cause, but it can't make that cause the root cause without evidence. Use instructions to point AI SRE at the right services, dashboards and checks.
Testing a hypothesis means calling the tools your AI connectors and private agents expose, and AI SRE isn't guaranteed to be read-only. Some built-in connector tools accept provider-defined commands or queries that can change data, and Custom MCP tools and private-agent capabilities can be write-capable. Private Agent calls use your role when you start the investigation; automatic runs use the `ai-sre` system actor. Expose only the tools, commands and credentials that are safe for unattended use. See [Evidence Sources](/ai/ai-sre/evidence-sources) for the access boundaries.
A question that no working tool can answer becomes a blind spot, and the verdict names what AI SRE couldn't examine. If no tool call succeeds at all, the run's outcome is **Could not investigate**. In the six-stage flow, a provider that refuses AI SRE's credentials can end the run as **Blocked**, with a next step to re-authorize that connector. Check the connection at **AI SRE → Atlas → Connectors** (**AI & Agents → Connectors** if your sidebar doesn't have an **AI SRE** item).