Rootly Private Agent

Rootly Private Agent gives Rootly AI SRE controlled access to infrastructure that is not reachable from the public internet. The agent runs inside your network, opens an outbound gRPC connection over TLS and HTTP/2 to Rootly, and executes only registered, typed capabilities allowed by your local policy. Enrollment, credential refresh, provider registration, and work use the same secure endpoint. It is separate from Rootly Edge Connector. Edge Connector runs event-triggered HTTP or script actions. Private Agent serves bounded infrastructure queries for an AI investigation and does not provide arbitrary script execution.

​

How it works

1. An owner or admin creates a short-lived, one-time enrollment token in Settings → Private agents.
2. The agent enrolls through gRPC over TLS, stores rotating credentials on its persistent volume, and registers its provider capabilities, health, and non-secret policy digest. It refreshes this snapshot every minute.
3. Rootly queues a typed invocation with an execution deadline.
4. The agent opens an authenticated bidirectional gRPC stream and advertises its available execution capacity, including per-instance slots when Prometheus is enabled. Rootly delivers eligible work over that connection and leaves excess work queued; the agent accepts only an exact provider, capability, and version it registered.
5. The provider enforces customer-local policy before calling the local service.
6. The agent sends a bounded structured result or typed error to Rootly through an authenticated gRPC call.

There is no inbound connection from Rootly to your network. Temporary disconnects leave eligible work queued until its deadline. Rootly dispatches new work only when an agent is recently online and its provider is healthy or degraded. Leased work uses heartbeats, bounded retries, and cooperative cancellation. The connection carries cancellation even when all execution slots are occupied. After a temporary disconnect, the agent reconnects with backoff and reports still-running work rather than restarting it solely because the connection changed. There is no HTTP polling fallback. Coordinate upgrades from any earlier polling-based agent with your Rootly representative. Terminal credential, local credential-storage, or malformed-protocol errors stop the work loop instead of reconnecting indefinitely. Check readiness and agent logs, correct the underlying configuration or storage problem, and coordinate credential recovery with an administrator. Do not delete persistent credential state as a routine reconnect step. Each registered capability includes typed input and output JSON schemas, subject to the serialized schema limit. Rootly rejects an invalid provider snapshot with gRPC INVALID_ARGUMENT and keeps the agent’s last accepted snapshot active. Messages above the transport size limit are rejected with RESOURCE_EXHAUSTED.

​

Security model

Access requires all of these independent checks:

1. Rootly authorizes the AI SRE request.
2. The agent accepts only a registered typed capability.
3. Customer-managed local policy permits the operation.
4. The local service authorizes the agent credential, such as a Kubernetes ServiceAccount.

Rootly can narrow local access, but it cannot expand it. In the Kubernetes provider, Secrets, workload mutations, exec, attach, port-forward, token creation, impersonation, and RBAC administration are not supported capabilities. Optional authorization diagnostics use Kubernetes self-review APIs to evaluate the agent’s own access without persisting objects. Private capabilities are also filtered by the initiating user’s Rootly role before they are shown to AI SRE and checked again at execution. Owners and admins can run safe and sensitive reads. Ordinary users can run safe reads, including metadata queries and authorization diagnostics, but sensitive reads such as Pod logs require an owner or admin. Observer, no-access, anonymous, and unrecognized-sensitivity contexts cannot execute private capabilities. The safe-read category intentionally includes Kubernetes API discovery and the agent’s authorization self-reviews. Ordinary users allowed to run safe-read investigations can inspect supported API resources and the agent ServiceAccount’s effective RBAC. Rules reviews require a locally allowed namespace, but their raw output can reveal write verbs, unsupported resources, and cluster-scoped rules. This information does not grant additional executable capabilities. These tools evaluate the agent’s identity rather than the initiating user’s Kubernetes identity. Treat access to safe-read investigations as access to this authorization information; see authorization diagnostics for scope and completeness limits. The enrollment token is displayed once and stored by Rootly only as a digest. Agent access credentials are short-lived and refreshed with a rotating refresh credential. Revoking an agent invalidates both credentials immediately. Private Connect encrypts invocation inputs, actor metadata, results, errors, and metrics at the application layer before storing them. Payload filtering applies to Private Connect request logs and transport tracing; it is not an exclusion from AI model traces or conversation history. Terminal transport records become eligible for scheduled deletion seven days after their last update, so job scheduling and backlog can delay removal. Data returned to AI SRE can enter model context, evaluation traces, and investigation or conversation history. Deleting a transport record does not delete those downstream copies. See Private Agent data and retention for the shared explanation of transport filtering versus AI logging and retention.

​

Deployment modes

The supported early-access topology is a combined process with core and its configured providers in one Pod. Supporting builds can run Kubernetes and multiple independent Prometheus instances together. For multiple clusters, deploy a separate agent in each cluster with a distinct, stable provider ID within your Rootly account. Rootly routes invocations to the selected agent and provider; it does not broadcast a query to every cluster. The agent source repository remains private and the agent is proprietary software. Helm charts and container images are separate distribution artifacts; using Helm does not require access to the application source. Obtain the approved image, chart, and any registry credentials through your Rootly representative. No public image or Helm release is implied by this guide. The agent exposes local health endpoints on port 8080 by default. If you override runtime.listen_address, configure your probes to use that address and port:

• GET /healthz for liveness
• GET /readyz for control-plane and provider readiness
• GET /version for build and component information

These endpoints are intended for local probes. They do not need an externally reachable Service or Ingress.

​

Manage agents

Owners and admins can use Settings → Private agents to:

• Create a one-time enrollment token
• Review online status, last-seen time, version, and registered providers
• Review each provider’s type, health, last check, and advertised capabilities
• Review the number of capabilities advertised by an agent
• Revoke an agent immediately

An agent is shown as online when it is active and has contacted Rootly within the last two minutes. Agent connectivity and provider health are separate: an online agent can have a healthy Kubernetes provider and an unhealthy Prometheus instance. Healthy means the provider reports availability; Degraded means it reports limited availability but can still receive authorized work. Unhealthy providers are excluded from new dispatch. Overall /readyz returns not_ready if any provider is not healthy, including a degraded provider; readiness is distinct from dispatch eligibility. The provider-health UI marks observations older than two minutes as Stale and does not present offline or revoked agents as healthy. Missing, invalid, or implausibly future-dated observations appear as Unavailable. These observation-freshness labels are display-only: dispatch separately checks recent agent connectivity, reported provider availability, authorization, and local policy. A stale display alone does not disable an otherwise eligible provider. Use Refresh to retrieve the latest stored snapshot; this does not force a new upstream health probe. Displayed capabilities are agent-reported, not a guarantee that the current user or local policy permits their execution.

​

Automate management through the public API

Deployment automation can use the existing Rootly API service at https://api.rootly.com with a Rootly API key whose user or service account has Private Agent management permission. The Private Agents feature must be enabled for the account. Existing OAuth scopes do not currently grant access to these endpoints. Send Authorization: Bearer <ROOTLY_API_KEY>, Accept: application/vnd.api+json, and Content-Type: application/vnd.api+json over HTTPS. See API authentication and conventions. Neither POST requires a request body. The enrollment-token response contains data.attributes.token and data.attributes.expires_at, uses Cache-Control: no-store, and returns the secret only once. The token expires after 24 hours and can be consumed once. Repeating token creation produces a different token; this operation is not idempotent. The two enrollment steps use different credentials:

1. Your trusted deployment automation calls the public management API with a Rootly API key to obtain an enrollment token.
2. It securely supplies only that one-time token to the agent. The agent exchanges it through gRPC Enroll for rotating agent credentials and creates its registered identity.

Do not place your Rootly management API key in the agent configuration, print the enrollment response in CI logs, or commit the token to source control. Agent access and refresh credentials cannot authenticate to the management API. Creating enrollment tokens through the API is optional; the UI remains available for manual setup. List requests accept page[number] and page[size] (default 50, maximum 1000) and return pagination metadata. Lists omit provider snapshots and do not load encrypted credentials or snapshots from the database; retrieve individual agents when provider details are needed. Provider observations can be stale, and online does not imply that every provider is healthy. Credentials and capability input/output schemas are never returned by inventory endpoints. Revocation is safe to repeat and retains the agent record and invocation history. It prevents new authenticated work, but an operation already executing inside the customer network may not stop immediately. A revoked installation needs a new enrollment token to enroll again. Management access is account-scoped; inaccessible agent IDs return 404 without revealing whether they exist.

​

Next step

See Private Agent for Kubernetes for local policy, RBAC, compatibility, Pod logs, and sizing guidance. See Private Agent for Prometheus for multi-instance configuration, query tools, authorization, and performance limits. See Private Agent Limits for the canonical runtime, adapter, schema, and AI-context defaults and ceilings.

​

Roadmap Scope