> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rootly.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Private Agent for Grafana Pyroscope (Early Access)
> Query private continuous profiles through bounded, read-only AI SRE tools.
**Early Preview:** Rootly Private Agent is under active development and available only to approved customers. Features, configuration, limits, and APIs may change before general availability. Confirm the approved agent and backend versions with your Rootly representative before production use.
The native Pyroscope provider gives Rootly AI SRE direct, read-only access to
continuous profiles inside your network. It can discover profile populations,
inspect a bounded value timeline, merge one profile window into a flame graph,
and compare two windows or populations. It uses Pyroscope's modern Querier API;
it does not install an MCP server or expose a generic API proxy.
One agent can connect to multiple Pyroscope instances or tenants. Each entry has
its own stable provider ID, endpoint, credentials, TLS trust, tenant, label
scope, limits, health, and execution capacity. Rootly routes each tool call to
one exact provider ID. AI SRE cannot supply or override the endpoint, tenant,
credentials, TLS settings, or enforced label matchers.
Provider IDs must remain stable and unique across every Private Agent provider
in your Rootly account so registration and tool routing stay unambiguous.
## Configure instances with Helm
Create a Secret separately from the Helm release. The `rootly-private-agent`
namespace must already exist; on a first install, run this after the base Helm
installation has created the namespace. This example uses a bearer token and
the `X-Scope-OrgID` value expected by a multi-tenant deployment:
```bash theme={null}
kubectl create secret generic rootly-private-agent-pyroscope \
--namespace rootly-private-agent \
--from-file=token=./pyroscope-token \
--from-file=tenant-id=./pyroscope-tenant-id \
--from-file=ca.pem=./pyroscope-ca.pem
```
Omit the CA file and `ca_bundle_file` when the endpoint certificate chains to a
CA already trusted by the agent image.
The agent image runs as UID/GID `65532`. For Kubernetes Secret volumes using mode `0440`, set Pod-level `securityContext.fsGroup: 65532` so the process can read the mounted files. Do not make credential files world-readable.
Add the provider and Secret mount to your values:
```yaml theme={null}
providers:
pyroscope:
- id: profiles-production
url: https://pyroscope.production.internal
bearer_token_file: /run/secrets/pyroscope-production/token
tenant_id_file: /run/secrets/pyroscope-production/tenant-id
ca_bundle_file: /run/secrets/pyroscope-production/ca.pem
enforced_label_matchers: '{environment="production"}'
policy:
maximum_concurrency: 2
maximum_range_seconds: 3600
maximum_nodes: 1024
extraVolumes:
- name: pyroscope-production
secret:
secretName: rootly-private-agent-pyroscope
defaultMode: 0440
extraVolumeMounts:
- name: pyroscope-production
mountPath: /run/secrets/pyroscope-production
readOnly: true
```
`url` can be a direct Pyroscope endpoint or a fixed Grafana data source proxy
prefix. It must use HTTPS by default and can include a path prefix. URLs cannot
contain embedded credentials, query parameters, or fragments. Plain HTTP
requires `allow_insecure_http: true` and is intended only for deliberately
trusted local test environments.
Choose one application-layer authentication method: bearer authentication,
basic authentication with paired `username_file` and `password_file`, or
intentionally unauthenticated access. Mutual TLS is a separate transport
control; configure its paired certificate and key files with any of those
choices. `tenant_id_file` supplies a fixed `X-Scope-OrgID` header; use one
provider entry per tenant when Rootly needs to route between tenants explicitly.
`enforced_label_matchers` is an additional, optional boundary. Configure a fixed
tenant or enforced matchers for a shared deployment; omit both only when the
endpoint itself is intentionally dedicated to the provider's full scope.
## Available tools
All capabilities are version `1` sensitive reads. Owner and admin investigations
can use them. Unattended AI SRE investigations can also use them when both AI SRE
and Private Agent are enabled, so scope credentials, tenants, labels, and limits
for unattended access.
| Tool | Diagnostic use |
| ---------------------------- | ----------------------------------------------------------------------------- |
| `pyroscope.profile_types` | Discover profile type IDs available in an explicit time window |
| `pyroscope.label_names` | Discover bounded profile label names |
| `pyroscope.label_values` | Discover bounded values for one label |
| `pyroscope.series` | List bounded profile populations and their labels |
| `pyroscope.query_series` | Locate CPU, allocation, memory, goroutine, or other profile changes over time |
| `pyroscope.query_profile` | Merge one population and time window into a bounded flame graph |
| `pyroscope.compare_profiles` | Produce a differential flame graph for two windows or populations |
Always call `pyroscope.profile_types` before profile queries and pass its exact
`id` as `profile_type`. Profile type IDs identify the profile name, sample type,
units, period type, and period units; do not guess one from a display label.
For example, query a CPU flame graph after discovering the profile type:
```json theme={null}
{
"profile_type": "process_cpu:cpu:nanoseconds:cpu:nanoseconds",
"selector": "{service_name=\"checkout\"}",
"start": "2026-09-21T10:00:00Z",
"end": "2026-09-21T10:10:00Z",
"max_nodes": 512,
"timeout_seconds": 15
}
```
To compare a baseline with an incident window:
```json theme={null}
{
"profile_type": "process_cpu:cpu:nanoseconds:cpu:nanoseconds",
"left_selector": "{service_name=\"checkout\"}",
"left_start": "2026-09-21T09:40:00Z",
"left_end": "2026-09-21T09:50:00Z",
"right_selector": "{service_name=\"checkout\"}",
"right_start": "2026-09-21T10:00:00Z",
"right_end": "2026-09-21T10:10:00Z",
"max_nodes": 512
}
```
## Local scope and result controls
`enforced_label_matchers` adds customer-controlled matchers to every
selector-based request. The configuration is parsed at startup and is not
reported to Rootly. Equality matchers are a clear default, for example
`environment="production",cluster="primary"` with or without surrounding
braces.
Series, timeline, flame-graph, and comparison calls must include a selective
positive matcher in addition to any enforced matchers. Empty, catch-all, and
negative-only selectors are rejected. A positive regular expression counts as
selective only when it has a non-empty literal prefix and does not match the
empty string. Profile-type discovery also honors enforced matchers; without
them, discovery operates across the fixed tenant and explicit time window. Use
a narrowly authorized tenant or proxy when tenant-wide discovery is too broad.
Label-name and label-value discovery also inherit enforced matchers; when no
matcher is configured or supplied, they can enumerate across the fixed tenant
and time window.
Every time window uses explicit RFC3339 `start` and `end` values. A future start
is rejected, a future end is capped to the current clock, and an empty or
over-limit window is rejected. `query_series` computes a safe default step when
none is supplied and rejects a step that would exceed the point budget.
Flame-graph node counts, discovery values, returned series, points per series,
response bytes, timeout, and per-instance concurrency are also bounded. See
[Private Agent Limits](/private-agent-limits#pyroscope) for defaults and hard
ceilings.
The agent sends compact profile names, levels, series, and points to Rootly as
sensitive evidence. It does not expose ingest, delete, raw profile download,
administrative, or arbitrary HTTP capabilities. Results can enter AI model
context, evaluation traces, and investigation or conversation history under the
[Private Agent retention model](/ai/data-privacy-for-rootly-ai#how-is-private-agent-data-logged-and-retained).
## TLS, health, and compatibility
* HTTPS verifies hostnames and certificates with TLS 1.2 or newer. There is no skip-verification option.
* A custom CA extends system trust. Paired client certificate and key files enable mutual TLS.
* Credential and tenant files are read for each request so projected-file rotation does not require editing configuration.
* Redirects are rejected, environment HTTP proxy settings are not inherited, and Rootly control-plane credentials are never forwarded upstream.
* Upstream error bodies and endpoint details are not returned to Rootly.
Health calls the authenticated `ProfileTypes` Querier method for the preceding
minute and validates the response shape. Results are cached for 15 seconds. An
unhealthy Pyroscope instance remains independently visible in **AI →
Configurations → Private Agent** and does not prevent healthy neighboring
providers from registering, although overall agent readiness remains strict.
Compatibility CI exercises every tool, tenant routing, and both Pyroscope
storage architectures against the latest patch of each upstream-maintained
minor. The current matrix covers Pyroscope 2.1 and 2.2. Re-test authentication,
tenancy, and response contracts before using another compatible service or
version.