AI SRE Evidence Sources

Rootly AI SRE attributes a cause only when evidence gathered during the investigation supports it, so an investigation is only as good as the sources it can reach. This page covers what a live investigation reads, how it chooses which connection to query, what each source is allowed to do, and what an investigation can’t see. Most of these sources belong to Atlas, Rootly’s AI layer. Other Atlas entry points have different tool access. Each source has its own page, linked below.

​

Sources at a Glance

The paths below use the consolidated AI SRE navigation, which is rolling out. If your sidebar still shows AI & Agents, drop the Atlas step: connectors and private agents sit directly under AI & Agents, and the Knowledge graph and Memory under AI & Agents → AI SRE. See Where AI Settings Live. Connect the smallest useful set first: your primary observability provider, your code or deployment provider, and the knowledge source that holds service runbooks. Add other providers when an investigation shows a specific evidence gap.

​

The Alert or Incident

Every investigation starts from the record it was given.

• For an alert: the summary and description, the monitor’s message and query, tags, event details such as environment, release, and trace or request IDs, and when the alert started and ended.
• For an incident: the title, severity, status, start time and summary.

From there, AI SRE loads more Rootly context as it needs it: an incident’s details and timeline, the Rootly services attached to an alert, and an incident’s retrospective.

​

Past Alerts, Incidents and Conclusions

AI SRE can also look back through your Rootly history.

• Past alerts. It searches your alerts by keyword across summary, labels, source and description, over the last 180 days by default.
• Similar past incidents. AI SRE’s initial hypothesis work can retrieve up to three similar incidents from the team’s history even when the Related incidents feature is off. When Related incidents is turned on in AI Settings, AI SRE can also call an on-demand search tool to look for similar incidents by meaning rather than keyword. Matches are weighted by shared services, comparable severity, the same owning team and recency, and include only incidents the run is allowed to read. A match points AI SRE at an earlier incident but doesn’t say how it was fixed. To learn that, AI SRE can open the incident and its retrospective. AI SRE does not require the Opt in to Rootly AI capabilities switch for its investigation path.
• Nearby alerts and open incidents. Where this is enabled, an alert investigation also considers other alerts from your organization that fired within an hour of it, conclusions AI SRE already reached on those alerts, and open, non-private incidents that overlapped the alert. AI SRE treats each one as a lead to test against the current alert’s own evidence, never as the answer.
• Earlier conclusions on the same service. An alert investigation also receives AI SRE conclusions from the previous 30 days on alerts that name the same service or scoped entity, counting only conclusions reached before the current alert fired, plus resolution and mitigation notes from incidents linked to the current alert or to those earlier alerts. AI SRE treats each one as a candidate explanation and repeats the reading it rested on over the current alert’s window. Incident investigations don’t use this lookup, and it’s separate from Memory.

​

AI Connectors

AI connectors are the main source of live external evidence. AI SRE queries them for the specific readings a step of the investigation needs. The table shows examples; the Connectors page lists every provider by category. When an alert comes from Datadog or Sentry, AI SRE reads the alerting signal through that provider’s tools first. Ownership comes from your service catalog through the Knowledge graph, not from work-tracking tools.

​

Private Agents

A private agent reaches systems inside your network that have no public endpoint. You run the agent, it connects outbound to Rootly, and it exposes typed capabilities, such as a Prometheus range query or a Kubernetes pod-log read, within the scope its local policy sets: namespaces, indices, databases and schemas, HTTP paths, and Kafka topics. An investigation uses only agents that show Online on the Private agents page, and only providers whose last reported health is Healthy or Degraded. Investigations also read a private agent’s description when deciding which one to use, as the next section explains.

​

How AI SRE Picks a Connection

You can connect more than one account of the same provider, such as two Datadog organizations or a production and a staging AWS account. For every investigation, AI SRE gets a map of your connections: each one’s name, its configured scope, and what earlier investigations observed through it, such as services, environments, regions and repositories. It’s told to match the alert’s context to each connection’s scope, to run a discovery query when several connections could fit or the alert says little, and never to pick a connection just because it’s listed first. Give it clear signals to route on:

• Name each connection for what it covers. The Connection name hint reads “A clear, distinct name gives the AI context to route tool calls to the right account. For example, Production EU.”
• Set the scope you want investigated. The configured scope is part of the map: the Datadog site, AWS account, regions and services, and selected Google Cloud projects.
• Describe Custom MCP servers and private agents. A Custom MCP connection’s description and a private agent’s description reach AI SRE as that connection’s purpose. The private agent’s description hint reads “Help Rootly choose this connection when more than one private agent is online.” Keep secrets and personal data out of descriptions.

AI SRE queries only connections that are ready. It skips connections that failed, are still pending, or are missing a required setting, such as a Google Cloud connection with no project selected. A provider you haven’t connected is never queried.

​

Knowledge Graph

The Knowledge graph holds derived facts about your environment, built from your Rootly services and catalog, infrastructure as code, and supported connectors such as AWS, Google Cloud, Datadog and Grafana. You reach it from AI SRE → Atlas → Knowledge graph (AI & Agents → AI SRE → Knowledge graph if your sidebar doesn’t have an AI SRE item), even though these docs file it under Atlas. An investigation reads the graph before it queries telemetry:

• Resolve names. AI SRE matches the services and resources an alert or incident names to entities in the graph. A name that matches several entities comes back as a list of candidates rather than a guess.
• Look up an entity. It reads what the graph knows about an entity, including its owners from your catalog.
• Follow dependencies. It walks recorded dependencies upstream to find candidate causes and downstream to map what else could be affected. This is how an investigation can propose a cause in a service that never alerted.
• Take inventory. It lists what exists in a part of your environment, so it isn’t limited to the components the alert happened to name.

Investigations also record which provider held an entity’s telemetry, so a later investigation can query that entity and its neighbors directly. The graph holds data only where Rootly has turned on graph ingestion for your organization. It stores facts derived from your sources, not a copy of their raw telemetry.

​

Memory

Memory holds short, durable context notes about your systems. You reach it from AI SRE → Atlas → Memory (AI & Agents → AI SRE → Memory if your sidebar doesn’t have an AI SRE item), even though these docs file it under Atlas. During an investigation, AI SRE can recall the best-matching Active notes. It treats a recalled note as a hypothesis to verify against live evidence. A note can’t be cited as evidence and can’t settle a cause on its own. Proposed, Deprecated and Quarantined notes aren’t recalled, and private incidents never feed Memory. See Memory for how notes are proposed, activated and quarantined.

​

Steer Where It Looks

Instructions and a matching investigation rule’s instructions tell AI SRE which evidence to check first, for example “Check for recent deploys to the affected service and link the suspected change.” They change what AI SRE checks first. They don’t add or remove sources, and AI SRE doesn’t treat an instruction as evidence.

​

Access Boundaries

AI connectors and private agents keep your providers’ permission boundaries, and they aren’t guaranteed to be read-only. Enabling AI SRE doesn’t widen a provider account, a connector’s tool list, a Kubernetes role, or a private agent’s local policy. What each source can do depends on how it connects. Who acts depends on the source:

• AI connectors use each connection’s own credentials, not those of the person who started the investigation.
• Private agents use the initiating user’s Private Agent permissions for a responder-started investigation. An automatic investigation uses the ai-sre system actor instead. No private-agent capability pauses for confirmation.
• Rootly data is read with the access of the person who started the investigation. An automatic investigation has no initiating responder, so its Rootly reads use a background identity with owner or admin access.

Automatic investigations run with no responder in the loop. Expose only connector tools, provider commands, local policies and credentials that are safe for unattended use, and prefer read-only identities. See Investigation Rules for automatic runs. For who can connect, configure and manage each source, see Manage User Permissions.

​

What a Live Investigation Can’t See

These boundaries apply to a live investigation of an alert or incident. Where a root cause sits entirely inside one of them, an investigation can reach a consistent-looking conclusion that is still wrong.

• Systems nothing reaches. A system with no AI connector and no private agent is invisible to AI SRE.
• Slack conversations. A live investigation doesn’t search Slack or read channel history. A cause discussed only in a team channel isn’t evidence it can find. Memory can be seeded from past incidents’ Slack messages and retrospectives, which is separate from a live investigation; see Memory.
• Changes with no record. A console change, a manual restart or a vendor-side fix that no connected tool recorded leaves nothing to find.
• Vendor status pages. AI SRE doesn’t read third-party status pages. It can check a managed dependency, such as a cloud database or queue, through the metrics your connected tools collect.
• Pulses. AI SRE doesn’t read Pulses. It reaches deploy and change history through your code, CI and feature flag connectors instead.
• Datadog change and deploy correlation over OAuth. The Connect Datadog MCP dialog warns: “OAuth currently excludes Datadog change and deploy correlation. Use API and application keys if Rootly AI must investigate what changed before an incident.”
• Some Datadog data types. AI SRE’s Datadog tools don’t read Synthetics, Audit Trail, Error Tracking, CI Visibility pipelines and tests, or profiling, so an alert built on those can’t be measured directly. Database Monitoring metrics and spans are available with dbm_read; parameterized query-pattern access requires API and application keys rather than OAuth.
• Other Datadog organizations. AI SRE reads only the Datadog organizations you’ve connected. If an alert comes from a different organization, connect that one as its own connection.
• Connections that aren’t ready. Failed or pending connections, a private agent that is offline, and a private-agent provider whose last health report isn’t Healthy or Degraded contribute nothing.

When a missing or refused connection limits an investigation, the result can say so in its open questions or next steps. See Reading the Result.

​

Related Pages

​

Frequently Asked Questions