Early Preview: Rootly Private Agent is under active development and available only to approved customers. Features, configuration, limits, and APIs may change before general availability. Confirm the approved agent and backend versions with your Rootly representative before production use.
kubectl and does not need AWS, Google Cloud, or Azure credentials to reach the Kubernetes API.
Use this guide for on-demand AI SRE investigations: Private Agent runs bounded
queries against the cluster selected for an investigation. For automatic
cluster-event ingestion as pulses, use the existing
Kubernetes integration and its
configuration guide. That workflow uses kubewatch
and webhooks; its setup and credentials are separate from Private Agent.
This makes the same agent compatible with conformant clusters including EKS, GKE, AKS, self-managed Kubernetes, and local Kind clusters. Kubernetes API discovery selects a version served by the connected cluster; callers can optionally request a specific API group and version.
The automated Kind test matrix covers Kubernetes 1.34, 1.35, and 1.36. Discovery
does not make every Kubernetes version or optional API supported: a resource
must be served by the cluster, supported by the provider, and permitted by local
policy and Kubernetes RBAC. Argo Rollouts, KEDA, Istio, Gateway API, and resource
metrics require their corresponding APIs to be installed.
Supported read operations
The provider supports get, list, and bounded watch operations for:
- Pods, Deployments, ReplicaSets, StatefulSets, DaemonSets
- Jobs and CronJobs
- Events, Nodes, Namespaces
- Services, Endpoints, and EndpointSlices
- ConfigMaps
- PersistentVolumes and PersistentVolumeClaims
- ResourceQuotas and LimitRanges
- StorageClasses and CustomResourceDefinitions
- HorizontalPodAutoscalers and PodDisruptionBudgets
- Ingresses, IngressClasses, NetworkPolicies, IPAddresses, ServiceCIDRs, and supported Gateway API resources
- Pod and Node resource metrics
- APIServices, admission policies and bindings, and admission webhook configurations
- PriorityClasses and RuntimeClasses
- CSIDrivers, CSINodes, CSIStorageCapacities, VolumeAttachments, and VolumeAttributesClasses
- DeviceClasses, ResourceClaims, ResourceClaimTemplates, and ResourceSlices
- Argo Rollouts and KEDA ScaledObjects
- Resources discovered in the
networking.istio.io API group
Not every resource exposes every read verb. Use kubernetes.api_resources to
discover the served versions and supported get, list, and watch verbs.
It accepts optional group, resource, namespaced, and limit filters;
group: "" explicitly selects the core API. Results are bounded to 500 entries
and indicate partial discovery or truncation. For metrics, explicitly select
group: "metrics.k8s.io" and resource: "pods" or resource: "nodes"; those
resource names also exist in the core API and otherwise can resolve to ordinary
objects rather than metrics.
Pod logs are a separate sensitive_read capability. ConfigMap data and binaryData are removed from generic results. Literal container environment-variable values are redacted from Pods and supported workload templates; variable names and valueFrom references remain visible. The kubectl.kubernetes.io/last-applied-configuration annotation is removed from returned objects because it can contain an older, unredacted manifest. Kubernetes Secrets and mutation capabilities are not supported. Redaction is not a guarantee that all customer-authored metadata or log content is non-sensitive; scope access accordingly.
Diagnose authorization
Two safe-read capabilities inspect the agent’s own ServiceAccount:
kubernetes.self_access_review checks a supported resource operation. get
requires a nonempty name; list and watch must omit name. Provide a
namespace for namespaced resources. Customer-local policy and served-verb
checks apply before the request reaches Kubernetes. The only supported
subresource is pods/log with get, and local Pod-log access must be enabled.kubernetes.self_rules_review requires an allowed namespace and returns raw
effective Kubernetes RBAC, including incomplete and evaluation_error
indicators. Its output can include write verbs, unsupported resources, and
cluster-scoped rules. These are not additional operations the agent can
execute, and an incomplete response must not be treated as a complete access
inventory.
Neither capability inspects another identity or persists an object. They are
classified as safe reads because of their execution effects, not because every
verb mentioned in the response is read-only.
Mount a non-secret YAML configuration file and set ROOTLY_PRIVATE_AGENT_CONFIG_FILE to its path. Secret values do not belong in this file.
Choose a stable provider id that is unique within the Rootly account, such as the cluster name plus region. Rootly rejects the same ID from a second active Private Agent so work cannot be routed ambiguously between clusters. Revoke a retired installation before reusing its provider ID.
An empty allowed_namespaces list denies all namespaced operations. Use "*" only when you intentionally want every namespace. Cluster-scoped access and Pod logs are disabled independently by default.
Local policy is not a substitute for Kubernetes RBAC. Grant the agent ServiceAccount only the resources its investigations need:
- Use a namespace
Role and RoleBinding when access is limited to one namespace.
- For several namespaces, create a
Role and RoleBinding in each allowed namespace, or define one read-only ClusterRole for namespaced resources and reference it from a separate RoleBinding in each allowed namespace. A namespaced Role cannot be reused from a different namespace. Each binding’s ServiceAccount subject must reference the namespace where the agent runs.
- For cluster-scoped resources such as Nodes, Namespaces, PersistentVolumes, StorageClasses, or CRD definitions, use a
ClusterRole with a ClusterRoleBinding only when those reads are required and permitted by local policy. Namespace RoleBindings do not grant cluster-scoped access.
- Grant
get on pods/log only when Pod-log access is enabled.
- For authorization diagnostics, ensure the ServiceAccount can
create
selfsubjectaccessreviews and selfsubjectrulesreviews in
authorization.k8s.io. These cluster-scoped API calls evaluate access; they do
not persist objects or grant permission to modify workloads or RBAC.
- Both review resources use cluster-level API endpoints. For
SelfSubjectRulesReview, the required spec.namespace selects the namespace whose rules are evaluated; it does not make the review resource namespaced. Verify existing self-review permissions before adding bindings. The Kubernetes client API reference lists these create endpoints without a /namespaces/{namespace} path segment.
- Do not grant Secrets, workload write verbs, RBAC administration, exec, attach, port-forward, impersonation, or service-account token creation.
The common read verbs are get, list, and watch. The provider cannot exceed either RBAC or its local YAML policy.
Mount enrollment and credential state
The enrollment token is short-lived and used once. Mount it from a Secret at the configured token path. The agent stores its rotating credential state on durable writable storage so Pod restarts do not require a new enrollment token.
The production image runs as UID and GID 65532. When a Secret volume uses mode 0440, set a Pod-level group so the non-root process can read it. This is a Pod-spec fragment, not a complete installation manifest; the Secret, configuration mount, and writable persistent credential volume must also be supplied:
Without fsGroup or equivalent group ownership, a 0440 Secret file owned by root with group root is not readable by this non-root image. Keep the fsGroup: 65532 setting with the mount shown above; do not make the token world-readable to work around the issue.
Query Pod logs safely
Enable Pod logs only when investigations require them. Each request supports Kubernetes PodLogOptions, including container, previous, since_seconds, since_time, timestamps, tail_lines, limit_bytes, and stream.
Kubernetes does not provide an end-time argument for Pod logs. Keep requests targeted with:
since_seconds or RFC3339 since_timetail_lineslimit_bytes- A request timeout
The provider bounds log bytes and request duration. Rootly AI SRE applies a smaller per-call log limit so a single response cannot consume the investigation’s model context. See the canonical local and AI limits. It reports returned bytes and whether output was truncated. Insecure kubelet-backend TLS is a separate local-policy opt-in and should remain disabled.
Rootly AI SRE also rejects private-provider results above its AI context cap before placing them in model context. Narrow the namespace, selectors, pagination limit, time window, or log bounds and retry instead of requesting one large response.
Pod logs are sensitive data. Private Connect encrypts its transport records and filters payloads from transport logs and traces, but returned logs can still enter AI model context, evaluation traces, and investigation history. See Private Agent data and retention for the canonical storage, asynchronous cleanup, and downstream retention boundaries.
Start a combined agent with requests around 100m CPU and 128Mi memory and limits around 1 CPU and 512Mi memory, then tune from observed usage. These are starting points, not universal requirements.
Load is controlled by independent runtime concurrency, list pagination, watch duration, encoded result size, log byte/time, and AI-context bounds. See Private Agent Limits for defaults and hard ceilings. Request deadlines and cancellation propagate into Kubernetes calls.
Large object metadata, high event volume, and concurrent watches can still increase CPU and memory. Lower runtime.maximum_concurrency for a small Pod or an API server with a tight request budget; excess work stays queued in Rootly until capacity is available. This reduces peak CPU, memory, and Kubernetes API pressure at the cost of parallel investigation throughput. Monitor container CPU, working-set memory, restarts, readiness, Kubernetes API throttling, invocation latency, and truncated-result counts. Narrow selectors, namespaces, time windows, and page sizes before increasing resources.
Network requirements
Allow outbound TCP 443 from the agent to its configured Rootly gRPC origin and access from the Pod to the cluster’s Kubernetes API endpoint. The default origin is https://connect.rootly.com. The control-plane channel requires gRPC over TLS and HTTP/2, including long-lived bidirectional streams; opening TCP 443 alone is not sufficient if a firewall or proxy downgrades or blocks HTTP/2. Enrollment, token refresh, provider registration, and work use this same endpoint. No inbound internet access, public Service, or Ingress is required.
If Rootly supplies a regional or custom endpoint, set rootly.grpc_url (or ROOTLY_PRIVATE_AGENT_GRPC_URL) and allow outbound access to that origin instead of connect.rootly.com. The value must be an origin without a path, query, or credentials. When omitted, legacy rootly.api_url selects the same gRPC endpoint, defaulting to https://connect.rootly.com. This override changes only the Private Agent control-plane transport origin; it does not change the region where Rootly AI processes or retains investigation data. See Data Privacy for Rootly AI for the current processing-region policy.
For enterprise proxies or TLS inspection, validate HTTP CONNECT and streaming behavior with your Rootly representative. Use a configured rootly.ca_bundle_file for an additional trusted CA when needed; certificate and hostname verification remain enabled. There is no HTTP polling fallback if gRPC is blocked.
TLS certificates are validated normally. Plain HTTP is rejected unless an explicit development-only override is enabled.
