Skip to main content
Early Preview: Rootly Private Agent is under active development and available only to approved customers. Features, configuration, limits, and APIs may change before general availability. Confirm the approved agent and backend versions with your Rootly representative before production use.
The combined Private Agent can run Kubernetes, Prometheus, and multiple native Loki adapters together. Rootly AI SRE selects one registered instance and a typed tool for each query. The adapter calls Loki directly over its HTTP API; it does not install or proxy a third-party MCP server.
Loki support requires matching early-access agent and Rootly backend builds. Confirm availability with your Rootly representative. This guide does not announce a public image or Helm release.

Configure instances

Add this fragment to your existing agent configuration, preserving Rootly enrollment and any other providers. Omit providers.loki or use an empty list to disable the adapter. Provider IDs must be stable and unique across every provider in your Rootly account to avoid ambiguous routing.
Mount credentials, tenant IDs, and TLS material as regular files with absolute paths. Kubernetes projected-volume symlinks are supported when they resolve to regular files; named pipes, devices, and directories are rejected. Choose bearer authentication, basic authentication, or intentionally unauthenticated access. tenant_id_file supplies a fixed X-Scope-OrgID header and is required for multi-tenant Loki deployments. AI tool calls cannot choose or override it. Endpoint path prefixes are supported, but URLs cannot contain embedded credentials, query parameters, or fragments. allow_insecure_http: true explicitly permits upstream transport that is not encrypted, including credentials and log data; avoid it outside deliberately trusted local test environments.
The agent image runs as UID/GID 65532. For Kubernetes Secret volumes using mode 0440, set Pod-level securityContext.fsGroup: 65532. Give only the agent identity equivalent read access on other platforms; do not make secret files world-readable.

Tools and arguments

All Loki capabilities are version 1 and classified as sensitive reads. The initial Rootly policy requires an owner or admin, including for label and series discovery. System and unattended actors cannot use them. start and end are explicit RFC3339 timestamps, and end must be after start. A future start is rejected. Rootly and the agent cap a future end to their current clock before enforcing the range. direction is backward by default and can be forward. Unknown fields, nulls, duplicate JSON keys, malformed UTF-8, and nonstandard label names are rejected. For example, select logs-production and call loki.query_logs with:
The provider ID is routing metadata, not a tool argument. URLs, credentials, tenant IDs, headers, and local policy cannot be supplied by AI SRE. Configure a separate provider entry for each Loki tenant or endpoint that needs distinct routing.

Query patterns

loki.query_patterns is registered only when patterns_enabled: true. Your Loki deployment must also enable pattern detection. The tool returns patterns with timestamp/count samples and a checked total_count; step_seconds defaults to 60. Use this tool to summarize repetitive messages after selecting a service and short time window, not as an unbounded search across all logs.

Local scope and scan-cost controls

enforced_matchers adds customer-controlled label matchers to every selector in every request. For example, cluster="production",environment="prod" prevents a tool call from escaping that label scope. The configuration is parsed at startup. When loki.label_names or loki.label_values omits its optional selector, the agent constructs one from enforced_matchers; when neither is present, discovery can enumerate labels across the configured Loki tenant within the explicit time window and result limits. Configure enforced matchers when tenant-wide discovery is broader than intended. Log and pattern queries must still include their own selective positive matcher: empty, catch-all, and negative-only selectors are rejected. A positive regex is treated as selective only when it has a fixed literal prefix (for example, app=~"api-.+"); use an equality matcher for other bounded expressions. Before retrieving log lines or patterns, the adapter calls Loki’s index stats endpoint for each distinct selector, sums the estimated bytes with overflow checks, and rejects the request when estimation fails or exceeds maximum_scan_bytes. Loki describes this estimate as probabilistic and excludes data not yet stored in the index, so it is an additional guardrail rather than an exact cost or billing value. loki.query_logs fetches one sentinel entry beyond the requested limit. metadata.has_more: true means AI SRE should narrow the selector or time window; it is not silent transport truncation and there is no synthetic cursor. Results group entries by stream and preserve timestamps, lines, labels, structured metadata, and parsed fields. Oversized or transport-truncated results fail instead of supplying incomplete evidence to the model. See Private Agent Limits for policy defaults and hard ceilings. Local concurrency is per Loki instance, is advertised to Rootly as per-instance capacity, and also consumes the shared runtime capacity. Rootly leaves excess work queued until capacity is available or its deadline expires. Cancellation reaches the upstream request. There is no automatic query retry, unbounded local queue, offline result cache, or on-disk log storage in the agent. Upgrade the Rootly backend before deploying a capacity-aware multi-provider agent. The agent refuses a work stream that does not acknowledge per-provider capacity support. Older Kubernetes-only agents can continue using global capacity during this rollout. Coordinate rollback of the agent and backend together.

Credentials, TLS, and data handling

  • HTTPS verifies hostnames and certificates with TLS 1.2 or newer; there is no skip-verification option. A ca_bundle_file extends system trust and requires an agent restart after trust-root changes.
  • Bearer, basic-auth, and tenant files are read on each request for rotation. Missing, empty, or malformed files fail closed.
  • client_certificate_file and client_key_file enable mTLS. They must be configured together, are loaded at startup and on new handshakes, and require a restart for an immediate cutover of established sessions.
  • Redirects are rejected, and the adapter does not inherit environment HTTP proxy settings. Rootly control-plane credentials are never forwarded to Loki.
  • Registration excludes endpoint URLs, credential paths/content, tenant IDs, TLS material, and enforced matchers. It includes only bounded policy values, their digest, health, and capabilities.
  • Upstream HTTP error bodies and transport URL details are not returned to Rootly.
  • Successful logs, labels, parsed fields, structured metadata, and patterns are sensitive evidence. They can enter AI context and history under the Private Agent retention model.
There are no Loki write, push, delete, ruler, or administrative capabilities.

Health and compatibility

Health uses a bounded request to Loki’s /loki/api/v1/index/stats endpoint with a fixed no-match selector, the configured authentication and tenant header, and a one-second window. This verifies the same API route and credentials used by tools without depending on label cardinality or reading customer logs; results are cached for 15 seconds. Probes have independent admission and connection capacity, so user saturation alone does not mark an upstream unhealthy. An unavailable Loki instance does not prevent healthy neighboring providers from registering or serving work. Overall /readyz is not ready whenever any provider is not healthy, including a degraded provider; readiness is stricter than dispatch eligibility. In Settings → Private agents, inspect each Loki instance’s status, last check, and reported capabilities. loki.query_patterns is absent when local pattern support is disabled. Agent connectivity is separate from provider health, and refreshing the page retrieves the stored snapshot rather than forcing a probe. CI exercises all six capabilities against digest-pinned Loki 3.6 and 3.7 releases. Other Loki-compatible services may work when they implement the same HTTP response contracts, including categorized log tuples, but are not certified by that matrix. Confirm authentication, tenancy, index stats, and pattern support before production use.