> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rootly.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Private Agent for Prometheus (Early Access)

> Query private Prometheus instances through bounded, read-only AI SRE tools.

<Warning>
  **Early Preview:** Rootly Private Agent is under active development and available only to approved customers. Features, configuration, limits, and APIs may change before general availability. Confirm the approved agent and backend versions with your Rootly representative before production use.
</Warning>

The combined Private Agent can run Kubernetes and multiple native Prometheus adapters together. Rootly AI SRE selects one registered instance and a typed tool for each query.

<Info>
  Prometheus support requires matching early-access agent and Rootly backend builds. Confirm availability with your Rootly representative. This guide does not announce a public image or Helm release.
</Info>

## Configure Instances

Add this fragment to your existing agent configuration, preserving its Rootly enrollment settings and any Kubernetes provider. Omit `providers.prometheus` to disable the adapter. Each list entry is enabled, subject to the [instance limit](/private-agent-limits#prometheus), with IDs unique across all providers in the process. Use stable, distinct IDs across your Rootly account to avoid ambiguous routing.

```yaml theme={null}
providers:
  prometheus:
    - id: metrics-production
      url: https://prometheus.production.internal/prometheus
      bearer_token_file: /run/secrets/prometheus-production/token
      ca_bundle_file: /run/secrets/prometheus-production/ca.pem
      policy:
        maximum_concurrency: 2
        maximum_series: 100
    - id: metrics-staging
      url: https://prometheus.staging.internal
      username_file: /run/secrets/prometheus-staging/username
      password_file: /run/secrets/prometheus-staging/password
```

Mount credentials as regular files with absolute paths. Kubernetes projected-volume symlinks are supported when they resolve to regular files; named pipes, devices, and directories are rejected. Choose either a bearer-token file or both basic-auth files; authentication can be omitted for an intentionally unauthenticated upstream. Endpoint path prefixes are supported, but URLs cannot contain embedded credentials, query parameters, or fragments. `allow_insecure_http: true` explicitly permits unencrypted upstream transport, including credentials; avoid it outside deliberately trusted local environments.

<Warning>
  The agent image runs as UID/GID `65532`. For Kubernetes Secret volumes using mode `0440`, set Pod-level `securityContext.fsGroup: 65532` so the process can read bearer/basic-auth, CA, and client-certificate/key files. See the [Secret-volume and persistent credential-state example](/private-agent-kubernetes#mount-enrollment-and-credential-state). On other platforms, give only the agent identity equivalent read access; do not make credential files world-readable.
</Warning>

## Tools and Arguments

All six capabilities are version `1` and classified as sensitive reads. The initial Rootly policy requires an owner or admin, including for metric names, metadata, and labels. System or unattended actors cannot use them under that policy.

| Capability                   | Required arguments                      | Optional arguments                 |
| ---------------------------- | --------------------------------------- | ---------------------------------- |
| `prometheus.query`           | `query`                                 | `time`, `timeout_seconds`, `limit` |
| `prometheus.query_range`     | `query`, `start`, `end`, `step_seconds` | `timeout_seconds`, `limit`         |
| `prometheus.list_metrics`    | None                                    | `match`, `start`, `end`, `limit`   |
| `prometheus.metric_metadata` | `metric`                                | `limit`                            |
| `prometheus.label_names`     | None                                    | `match`, `start`, `end`, `limit`   |
| `prometheus.label_values`    | `label`                                 | `match`, `start`, `end`, `limit`   |

Timestamps must be RFC3339 strings. Discovery `start` and `end` must be supplied together. `step_seconds` is a positive integer. Queries, selectors, names, timestamps, and encoded GET request targets must fit the [Prometheus input limits](/private-agent-limits#prometheus). `match` maps to upstream `match[]`. Label names use `[a-zA-Z_][a-zA-Z0-9_]*`; UTF-8 name escaping is not supported. Unknown fields and nulls are rejected.

For example, select `metrics-production` and call `prometheus.query_range` with:

```json theme={null}
{
  "query": "sum by (namespace) (rate(container_cpu_usage_seconds_total[5m]))",
  "start": "2026-09-10T10:00:00Z",
  "end": "2026-09-10T10:15:00Z",
  "step_seconds": 30,
  "limit": 20
}
```

The provider ID is routing metadata, not a tool argument. URLs, credentials, headers, tenant IDs, and AWS profiles cannot be supplied by an AI tool call. Local configuration determines the destination. Instance isolation does not enforce label-level tenancy within an upstream; use isolated servers or an authenticated query gateway when needed.

## Limits and Upstream Performance

See [Private Agent Limits](/private-agent-limits#prometheus) for the canonical policy defaults, hard ceilings, and input bounds. Omitted or zero policy values use defaults, not unlimited mode. Tool limits and timeouts cannot exceed local policy. The agent advertises free execution slots per instance as well as the shared runtime capacity. Rootly leaves excess work for a busy instance queued until capacity becomes available or its deadline expires; local admission waiters do not consume the slots needed by healthy neighboring providers. Active requests still share the global cap, so this is not a dedicated-capacity guarantee. Cancellation reaches the upstream request. There is no automatic query retry, separate unbounded agent queue, offline result cache, or on-disk metric storage.

Upgrade the Rootly backend before deploying a capacity-aware multi-provider agent. The agent refuses a work stream that does not acknowledge per-provider capacity support. Older Kubernetes-only agents can continue using global capacity during this rollout. Coordinate rollback of the agent and backend together.

Oversized or truncated responses fail instead of silently giving AI SRE incomplete evidence. Narrow the selector, window, metric name, or requested limit. Successful results preserve the Prometheus envelope, sample values, and warning/info annotations.

<Warning>
  Response limits bound agent-side handling, not PromQL evaluation cost on the server. Expensive subqueries, inner range selectors, and high-cardinality expressions still require upstream sample, timeout, concurrency, and tenant limits. These defaults are not a production sizing guarantee.
</Warning>

## Credentials, TLS, and Data Handling

* HTTPS verifies hostnames and certificates with TLS 1.2 or newer; there is no skip-verification option. A `ca_bundle_file` extends system trust and requires an agent restart after trust-root changes.
* Bearer and basic-auth files are read on each request for rotation. Missing, empty, or malformed credentials fail closed.
* `client_certificate_file` and `client_key_file` enable mTLS. They are loaded at startup and on new handshakes; restart for an immediate cutover of established sessions.
* Redirects are rejected, and the adapter does not inherit environment HTTP proxy settings. Rootly control-plane credentials are never forwarded upstream.
* Registration excludes configured endpoint URLs and credential paths/content. Upstream HTTP error bodies and transport URL details are not returned to Rootly.
* Successful metric data and labels are sensitive evidence and are not generically redacted. They can enter AI context and history under the [Private Agent retention model](/ai/data-privacy-for-rootly-ai#how-is-private-agent-data-logged-and-retained).

There are no write/admin, remote-write, reload, delete-series, targets, rules, or alerts capabilities. Automatic OAuth exchange, AWS SigV4, fixed tenant headers, and custom proxy configuration are not included.

## Health and Compatibility

Health uses a small `vector(1)` query with [bounded probing and caching](/private-agent-limits#health-probes). Probes have independent admission and connection capacity, so user saturation alone does not mark an upstream unhealthy. An unavailable instance is reported as unhealthy without preventing healthy neighboring providers from registering or serving work. Overall `/readyz` returns `not_ready` when a provider is not healthy, while liveness and the shared work stream remain independent. Invalid initial configuration or unreadable credentials stop startup. See [provider status and dispatch semantics](/private-agent#manage-agents) for the distinction between reported availability and display-only freshness labels.

In **Settings → Private agents**, inspect each instance's type, status, last check, and reported capabilities. Agent connectivity is distinct from upstream health; stale observations are not shown as healthy. Refreshing the page retrieves the stored snapshot rather than initiating a probe.

The adapter targets the stable Prometheus `/api/v1` API and has been tested against Prometheus 3.5.0. This testing does not certify production load capacity or compatibility with Thanos, Mimir, VictoriaMetrics, and hosted services. Confirm their API and authentication requirements before use.