Early Preview: Rootly Private Agent is under active development and available only to approved customers. Features, configuration, limits, and APIs may change before general availability. Confirm the approved agent and backend versions with your Rootly representative before production use.
providers.tempo to your existing agent configuration. Provider IDs must be unique across all provider types in the process and across active agents in the same Rootly account.
url can include a gateway path prefix. It must use HTTPS and cannot contain embedded credentials, a query string, or a fragment. allow_insecure_http: true permits plaintext only as an explicit local-testing opt-in; do not use it to send production credentials or trace data over a network you do not trust.
Choose one HTTP authentication mode per instance:
bearer_token_fileusername_file with password_file, including Grafana Cloud basic authentication- An intentionally unauthenticated endpoint
mTLS is transport authentication and can be layered on any of these HTTP authentication modes by configuring client_certificate_file and client_key_file.
tenant_id_file provides the fixed X-Scope-OrgID header used by multi-tenant Tempo deployments. Configure a separate provider entry for every endpoint or tenant that needs independent routing. AI SRE cannot select or override the endpoint, tenant, credentials, headers, or TLS settings.
Mount credentials, tenant IDs, and TLS material as absolute paths that resolve to regular files. Kubernetes projected-volume symlinks are supported when they resolve to regular files. Named pipes, devices, and directories are rejected.
The agent image runs as UID/GID 65532. For Kubernetes Secret volumes using mode 0440, set Pod-level securityContext.fsGroup: 65532. Give only the agent identity equivalent read access on other platforms; do not make secret files world-readable.
1 and sensitive reads because trace attributes, errors, database statements, identifiers, and span payloads can contain private data. Attended use requires an owner or admin. Unattended AI SRE system investigations can use the server-reviewed Tempo capabilities when Private Agent is available for the account and AI SRE is enabled.
query and filter_query use TraceQL. Every window uses RFC3339 timestamps, must be at least one second, and cannot exceed the configured local range. A future start is rejected. Rootly and the agent clamp a future end to their current clock before dispatch. Range metrics require a positive step_seconds no greater than 86,400; increase it when needed to keep each series within the configured point budget.
For tempo.attribute_names, scope accepts all, resource, span, intrinsic, event, link, or instrumentation and defaults to all when omitted.
For example, search production checkout traces for server errors:
Then pass one returned hexadecimal trace ID to tempo.get_trace. Supplying a paired time window can make a trace lookup more efficient. Do not send an endpoint, tenant ID, or credential as a tool argument; those fields are rejected before work is created.
Query and result boundaries
The provider forwards bounded limits to Tempo and independently reapplies them to returned traces, spans, attribute values, metric series, samples, and exemplars. Malformed responses, duplicate JSON keys, redirects, unknown or null input fields, oversized request targets, and responses beyond the local byte ceiling fail closed.
See Private Agent Limits for policy defaults and hard ceilings. The 128 KiB Rootly AI-context ceiling remains independent of the agent’s local result limit. If a valid upstream response is too large, narrow the TraceQL expression, time window, result limit, span count, metric resolution, or trace lookup window.
Local concurrency is per Tempo instance and also consumes the shared runtime capacity. Rootly leaves excess work queued until capacity is available or its deadline expires. Cancellation reaches the upstream request. The adapter has no automatic query retry, unbounded local queue, offline result cache, continuous trace collection, or local trace storage.
Credentials, TLS, and data handling
- HTTPS validates hostnames and certificates with TLS 1.2 or newer. There is no skip-verification option.
- A private
ca_bundle_file extends system trust. Restart the agent after changing the CA bundle.
client_certificate_file and client_key_file enable mTLS and must be configured together.- Bearer, basic-auth, tenant, and client-certificate files are reread for rotation. Established TLS connections may require a restart for immediate certificate cutover.
- Redirects are rejected, and the adapter does not inherit environment proxy settings. Rootly control-plane credentials are never forwarded to Tempo.
- Registration includes provider identity, bounded policy values, health, and capability schemas. It excludes endpoints, credential paths and contents, tenant IDs, and TLS material.
- Upstream HTTP error bodies and transport URL details are not returned to Rootly.
Use a dedicated query-only Tempo or gateway identity scoped to the intended tenant. The agent exposes no ingestion, deletion, override, ring, status, profiling, usage, or other administrative Tempo APIs. Trace results are sensitive evidence and can enter AI context and investigation history under the Private Agent retention model.
Health and compatibility
Health sends an authenticated, tenant-scoped request to Tempo’s /api/echo query-frontend endpoint. It verifies endpoint routing, TLS, credentials, and tenant headers without searching trace data. Results are cached for 15 seconds. One unhealthy Tempo instance does not prevent healthy neighboring providers from registering or serving work, but overall /readyz remains stricter and reports not ready when any configured provider is unhealthy.
In AI → Configurations → Private Agent, inspect each Tempo instance’s status, last observation, and reported capabilities. Refreshing the page reads the last stored provider snapshot; it does not force a new Tempo probe.
CI runs every capability against digest-pinned Grafana Tempo 2.10 and 3.0 containers. The test sends deterministic OTLP/HTTP traces to two tenants and verifies tenant isolation, TraceQL search, trace retrieval, attribute discovery, instant metrics, and range metrics.
Tempo 2.x needs its local-blocks metrics-generator processor for TraceQL metrics over recent traces. Without it, search, trace retrieval, and attribute discovery can work while metrics queries fail upstream. Tempo 3.0 supports the tested single-binary metrics flow without that legacy block. The compatibility matrix verifies API behavior, not production capacity; test representative trace cardinality and query latency before raising local limits.
