> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rootly.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Private Agent for Grafana Tempo (Early Access)

> Investigate private distributed traces with bounded TraceQL tools through Rootly Private Agent.

<Warning>
  **Early Preview:** Rootly Private Agent is under active development and available only to approved customers. Features, configuration, limits, and APIs may change before general availability. Confirm the approved agent and backend versions with your Rootly representative before production use.
</Warning>

Rootly Private Agent connects AI SRE to Grafana Tempo instances reachable from inside your network. The native adapter calls Tempo's query-frontend HTTP API directly; it does not require Tempo's optional MCP server and does not expose a generic HTTP proxy.

Each configured entry becomes a separate provider with its own stable ID, endpoint, tenant, credentials, TLS material, health, and query budgets. One agent can connect to multiple Tempo instances or tenants while also running Kubernetes, Prometheus, Loki, and other providers. Rootly routes every tool call to the exact registered provider ID.

## Configure instances

Add `providers.tempo` to your existing agent configuration. Provider IDs must be unique across all provider types in the process and across active agents in the same Rootly account.

```yaml theme={null}
providers:
  tempo:
    - id: traces-production
      url: https://tempo.production.internal
      bearer_token_file: /run/secrets/tempo-production/token
      tenant_id_file: /run/secrets/tempo-production/tenant-id
      ca_bundle_file: /run/secrets/tempo-production/ca.pem
      client_certificate_file: /run/secrets/tempo-production/client.crt
      client_key_file: /run/secrets/tempo-production/client.key
      policy:
        maximum_concurrency: 2
        maximum_result_bytes: 524288
        maximum_traces: 20
        maximum_spans_per_span_set: 3
        maximum_attribute_values: 100
        maximum_stale_values: 1000
        maximum_series: 100
        maximum_points_per_series: 1000
        maximum_exemplars: 20
        maximum_range_seconds: 3600
        maximum_timeout_seconds: 15
    - id: traces-staging
      url: https://tempo.staging.internal/tempo
      username_file: /run/secrets/tempo-staging/username
      password_file: /run/secrets/tempo-staging/password
      tenant_id_file: /run/secrets/tempo-staging/tenant-id
```

`url` can include a gateway path prefix. It must use HTTPS and cannot contain embedded credentials, a query string, or a fragment. `allow_insecure_http: true` permits plaintext only as an explicit local-testing opt-in; do not use it to send production credentials or trace data over a network you do not trust.

Choose one HTTP authentication mode per instance:

* `bearer_token_file`
* `username_file` with `password_file`, including Grafana Cloud basic authentication
* An intentionally unauthenticated endpoint

mTLS is transport authentication and can be layered on any of these HTTP authentication modes by configuring `client_certificate_file` and `client_key_file`.

`tenant_id_file` provides the fixed `X-Scope-OrgID` header used by multi-tenant Tempo deployments. Configure a separate provider entry for every endpoint or tenant that needs independent routing. AI SRE cannot select or override the endpoint, tenant, credentials, headers, or TLS settings.

Mount credentials, tenant IDs, and TLS material as absolute paths that resolve to regular files. Kubernetes projected-volume symlinks are supported when they resolve to regular files. Named pipes, devices, and directories are rejected.

<Warning>
  The agent image runs as UID/GID `65532`. For Kubernetes Secret volumes using mode `0440`, set Pod-level `securityContext.fsGroup: 65532`. Give only the agent identity equivalent read access on other platforms; do not make secret files world-readable.
</Warning>

## Available tools

All Tempo tools are capability version `1` and sensitive reads because trace attributes, errors, database statements, identifiers, and span payloads can contain private data. Attended use requires an owner or admin. Unattended AI SRE system investigations can use the server-reviewed Tempo capabilities when Private Agent is available for the account and AI SRE is enabled.

| Capability                  | Required arguments                      | Optional arguments                                                      |
| --------------------------- | --------------------------------------- | ----------------------------------------------------------------------- |
| `tempo.search_traces`       | `query`, `start`, `end`                 | `limit`, `spans_per_span_set`, `timeout_seconds`                        |
| `tempo.get_trace`           | `trace_id`                              | Paired `start` and `end`, `timeout_seconds`                             |
| `tempo.attribute_names`     | `start`, `end`                          | `scope`, `filter_query`, `limit`, `max_stale_values`, `timeout_seconds` |
| `tempo.attribute_values`    | `attribute`, `start`, `end`             | `filter_query`, `limit`, `max_stale_values`, `timeout_seconds`          |
| `tempo.query_metrics`       | `query`, `start`, `end`                 | `exemplars`, `timeout_seconds`                                          |
| `tempo.query_metrics_range` | `query`, `start`, `end`, `step_seconds` | `exemplars`, `timeout_seconds`                                          |

`query` and `filter_query` use TraceQL. Every window uses RFC3339 timestamps, must be at least one second, and cannot exceed the configured local range. A future start is rejected. Rootly and the agent clamp a future end to their current clock before dispatch. Range metrics require a positive `step_seconds` no greater than 86,400; increase it when needed to keep each series within the configured point budget.

For `tempo.attribute_names`, `scope` accepts `all`, `resource`, `span`, `intrinsic`, `event`, `link`, or `instrumentation` and defaults to `all` when omitted.

For example, search production checkout traces for server errors:

```json theme={null}
{
  "query": "{ resource.service.name = \"checkout\" && status = error }",
  "start": "2026-09-21T14:00:00Z",
  "end": "2026-09-21T14:10:00Z",
  "limit": 20,
  "spans_per_span_set": 3
}
```

Then pass one returned hexadecimal trace ID to `tempo.get_trace`. Supplying a paired time window can make a trace lookup more efficient. Do not send an endpoint, tenant ID, or credential as a tool argument; those fields are rejected before work is created.

## Query and result boundaries

The provider forwards bounded limits to Tempo and independently reapplies them to returned traces, spans, attribute values, metric series, samples, and exemplars. Malformed responses, duplicate JSON keys, redirects, unknown or null input fields, oversized request targets, and responses beyond the local byte ceiling fail closed.

See [Private Agent Limits](/private-agent-limits#grafana-tempo) for policy defaults and hard ceilings. The 128 KiB Rootly AI-context ceiling remains independent of the agent's local result limit. If a valid upstream response is too large, narrow the TraceQL expression, time window, result limit, span count, metric resolution, or trace lookup window.

Local concurrency is per Tempo instance and also consumes the shared runtime capacity. Rootly leaves excess work queued until capacity is available or its deadline expires. Cancellation reaches the upstream request. The adapter has no automatic query retry, unbounded local queue, offline result cache, continuous trace collection, or local trace storage.

## Credentials, TLS, and data handling

* HTTPS validates hostnames and certificates with TLS 1.2 or newer. There is no skip-verification option.
* A private `ca_bundle_file` extends system trust. Restart the agent after changing the CA bundle.
* `client_certificate_file` and `client_key_file` enable mTLS and must be configured together.
* Bearer, basic-auth, tenant, and client-certificate files are reread for rotation. Established TLS connections may require a restart for immediate certificate cutover.
* Redirects are rejected, and the adapter does not inherit environment proxy settings. Rootly control-plane credentials are never forwarded to Tempo.
* Registration includes provider identity, bounded policy values, health, and capability schemas. It excludes endpoints, credential paths and contents, tenant IDs, and TLS material.
* Upstream HTTP error bodies and transport URL details are not returned to Rootly.

Use a dedicated query-only Tempo or gateway identity scoped to the intended tenant. The agent exposes no ingestion, deletion, override, ring, status, profiling, usage, or other administrative Tempo APIs. Trace results are sensitive evidence and can enter AI context and investigation history under the [Private Agent retention model](/ai/data-privacy-for-rootly-ai#how-is-private-agent-data-logged-and-retained).

## Health and compatibility

Health sends an authenticated, tenant-scoped request to Tempo's `/api/echo` query-frontend endpoint. It verifies endpoint routing, TLS, credentials, and tenant headers without searching trace data. Results are cached for 15 seconds. One unhealthy Tempo instance does not prevent healthy neighboring providers from registering or serving work, but overall `/readyz` remains stricter and reports not ready when any configured provider is unhealthy.

In **AI → Configurations → Private Agent**, inspect each Tempo instance's status, last observation, and reported capabilities. Refreshing the page reads the last stored provider snapshot; it does not force a new Tempo probe.

CI runs every capability against digest-pinned Grafana Tempo 2.10 and 3.0 containers. The test sends deterministic OTLP/HTTP traces to two tenants and verifies tenant isolation, TraceQL search, trace retrieval, attribute discovery, instant metrics, and range metrics.

Tempo 2.x needs its `local-blocks` metrics-generator processor for TraceQL metrics over recent traces. Without it, search, trace retrieval, and attribute discovery can work while metrics queries fail upstream. Tempo 3.0 supports the tested single-binary metrics flow without that legacy block. The compatibility matrix verifies API behavior, not production capacity; test representative trace cardinality and query latency before raising local limits.