> ## Documentation Index > Fetch the complete documentation index at: https://docs.rootly.com/llms.txt > Use this file to discover all available pages before exploring further. # Private Agent for Kubernetes > Configure Rootly Private Agent for bounded, read-only Kubernetes investigation access. **Early Preview:** Rootly Private Agent is under active development and available only to approved customers. Features, configuration, limits, and APIs may change before general availability. Confirm the approved agent and backend versions with your Rootly representative before production use. The Kubernetes provider uses the official Kubernetes Go clients and in-cluster ServiceAccount authentication. It does not shell out to `kubectl` and does not need AWS, Google Cloud, or Azure credentials to reach the Kubernetes API. Use this guide for **on-demand AI SRE investigations**: Private Agent runs bounded queries against the cluster selected for an investigation. For **automatic cluster-event ingestion as pulses**, use the existing [Kubernetes integration](/integrations/kubernetes) and its [configuration guide](/configuration/kubernetes). That workflow uses kubewatch and webhooks; its setup and credentials are separate from Private Agent. This makes the same agent compatible with conformant clusters including EKS, GKE, AKS, self-managed Kubernetes, and local Kind clusters. Kubernetes API discovery selects a version served by the connected cluster; callers can optionally request a specific API group and version. The automated Kind test matrix covers Kubernetes 1.34, 1.35, and 1.36. Discovery does not make every Kubernetes version or optional API supported: a resource must be served by the cluster, supported by the provider, and permitted by local policy and Kubernetes RBAC. Argo Rollouts, KEDA, Istio, Gateway API, and resource metrics require their corresponding APIs to be installed. ## Supported read operations The provider supports get, list, and bounded watch operations for: * Pods, Deployments, ReplicaSets, StatefulSets, DaemonSets * Jobs and CronJobs * Events, Nodes, Namespaces * Services, Endpoints, and EndpointSlices * ConfigMaps * PersistentVolumes and PersistentVolumeClaims * ResourceQuotas and LimitRanges * StorageClasses and CustomResourceDefinitions * HorizontalPodAutoscalers and PodDisruptionBudgets * Ingresses, IngressClasses, NetworkPolicies, IPAddresses, ServiceCIDRs, and supported Gateway API resources * Pod and Node resource metrics * APIServices, admission policies and bindings, and admission webhook configurations * PriorityClasses and RuntimeClasses * CSIDrivers, CSINodes, CSIStorageCapacities, VolumeAttachments, and VolumeAttributesClasses * DeviceClasses, ResourceClaims, ResourceClaimTemplates, and ResourceSlices * Argo Rollouts and KEDA ScaledObjects * Resources discovered in the `networking.istio.io` API group Not every resource exposes every read verb. Use `kubernetes.api_resources` to discover the served versions and supported `get`, `list`, and `watch` verbs. It accepts optional `group`, `resource`, `namespaced`, and `limit` filters; `group: ""` explicitly selects the core API. Results are bounded to 500 entries and indicate partial discovery or truncation. For metrics, explicitly select `group: "metrics.k8s.io"` and `resource: "pods"` or `resource: "nodes"`; those resource names also exist in the core API and otherwise can resolve to ordinary objects rather than metrics. Pod logs are a separate `sensitive_read` capability. ConfigMap `data` and `binaryData` are removed from generic results. Literal container environment-variable values are redacted from Pods and supported workload templates; variable names and `valueFrom` references remain visible. The `kubectl.kubernetes.io/last-applied-configuration` annotation is removed from returned objects because it can contain an older, unredacted manifest. Kubernetes Secrets and mutation capabilities are not supported. Redaction is not a guarantee that all customer-authored metadata or log content is non-sensitive; scope access accordingly. ## Diagnose authorization Two safe-read capabilities inspect the agent's own ServiceAccount: * `kubernetes.self_access_review` checks a supported resource operation. `get` requires a nonempty `name`; `list` and `watch` must omit `name`. Provide a namespace for namespaced resources. Customer-local policy and served-verb checks apply before the request reaches Kubernetes. The only supported subresource is `pods/log` with `get`, and local Pod-log access must be enabled. * `kubernetes.self_rules_review` requires an allowed namespace and returns raw effective Kubernetes RBAC, including `incomplete` and `evaluation_error` indicators. Its output can include write verbs, unsupported resources, and cluster-scoped rules. These are not additional operations the agent can execute, and an incomplete response must not be treated as a complete access inventory. Neither capability inspects another identity or persists an object. They are classified as safe reads because of their execution effects, not because every verb mentioned in the response is read-only. ## Configure local policy Mount a non-secret YAML configuration file and set `ROOTLY_PRIVATE_AGENT_CONFIG_FILE` to its path. Secret values do not belong in this file. ```yaml theme={null} version: 1 runtime: listen_address: ":8080" log_level: info rootly: grpc_url: https://connect.rootly.com control_plane_enabled: true agent_name: production-us-west-2 credential_state_file: /var/lib/rootly-private-agent/credentials.json enrollment_token_file: /var/run/secrets/rootly-private-agent/enrollment-token providers: kubernetes: enabled: true id: production-us-west-2 display_name: Production US West 2 policy: allowed_namespaces: - production - payments allow_cluster_scoped: false default_list_limit: 500 maximum_list_limit: 1000 default_watch_seconds: 30 maximum_watch_seconds: 60 maximum_result_bytes: 2097152 allow_pod_logs: false allow_insecure_log_backend_tls: false default_log_bytes: 262144 maximum_log_bytes: 1048576 default_log_seconds: 30 maximum_log_seconds: 60 ``` Choose a stable provider `id` that is unique within the Rootly account, such as the cluster name plus region. Rootly rejects the same ID from a second active Private Agent so work cannot be routed ambiguously between clusters. Revoke a retired installation before reusing its provider ID. An empty `allowed_namespaces` list denies all namespaced operations. Use `"*"` only when you intentionally want every namespace. Cluster-scoped access and Pod logs are disabled independently by default. ## Configure Kubernetes RBAC Local policy is not a substitute for Kubernetes RBAC. Grant the agent ServiceAccount only the resources its investigations need: * Use a namespace `Role` and `RoleBinding` when access is limited to one namespace. * For several namespaces, create a `Role` and `RoleBinding` in each allowed namespace, or define one read-only `ClusterRole` for namespaced resources and reference it from a separate `RoleBinding` in each allowed namespace. A namespaced `Role` cannot be reused from a different namespace. Each binding's ServiceAccount subject must reference the namespace where the agent runs. * For cluster-scoped resources such as Nodes, Namespaces, PersistentVolumes, StorageClasses, or CRD definitions, use a `ClusterRole` with a `ClusterRoleBinding` only when those reads are required and permitted by local policy. Namespace `RoleBinding`s do not grant cluster-scoped access. * Grant `get` on `pods/log` only when Pod-log access is enabled. * For authorization diagnostics, ensure the ServiceAccount can `create` `selfsubjectaccessreviews` and `selfsubjectrulesreviews` in `authorization.k8s.io`. These cluster-scoped API calls evaluate access; they do not persist objects or grant permission to modify workloads or RBAC. * Both review resources use cluster-level API endpoints. For `SelfSubjectRulesReview`, the required `spec.namespace` selects the namespace whose rules are evaluated; it does not make the review resource namespaced. Verify existing self-review permissions before adding bindings. The [Kubernetes client API reference](https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/AuthorizationV1Api.md) lists these create endpoints without a `/namespaces/{namespace}` path segment. * Do not grant Secrets, workload write verbs, RBAC administration, exec, attach, port-forward, impersonation, or service-account token creation. The common read verbs are `get`, `list`, and `watch`. The provider cannot exceed either RBAC or its local YAML policy. ## Mount enrollment and credential state The enrollment token is short-lived and used once. Mount it from a Secret at the configured token path. The agent stores its rotating credential state on durable writable storage so Pod restarts do not require a new enrollment token. The production image runs as UID and GID `65532`. When a Secret volume uses mode `0440`, set a Pod-level group so the non-root process can read it. This is a Pod-spec fragment, not a complete installation manifest; the Secret, configuration mount, and writable persistent credential volume must also be supplied: ```yaml theme={null} spec: securityContext: runAsNonRoot: true fsGroup: 65532 fsGroupChangePolicy: OnRootMismatch containers: - name: rootly-private-agent securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"] volumes: - name: enrollment-token secret: secretName: rootly-private-agent-enrollment defaultMode: 0440 ``` Without `fsGroup` or equivalent group ownership, a `0440` Secret file owned by root with group root is not readable by this non-root image. Keep the `fsGroup: 65532` setting with the mount shown above; do not make the token world-readable to work around the issue. ## Query Pod logs safely Enable Pod logs only when investigations require them. Each request supports Kubernetes `PodLogOptions`, including `container`, `previous`, `since_seconds`, `since_time`, `timestamps`, `tail_lines`, `limit_bytes`, and `stream`. Kubernetes does not provide an end-time argument for Pod logs. Keep requests targeted with: * `since_seconds` or RFC3339 `since_time` * `tail_lines` * `limit_bytes` * A request timeout The provider bounds log bytes and request duration. Rootly AI SRE applies a smaller per-call log limit so a single response cannot consume the investigation's model context. See the [canonical local and AI limits](/private-agent-limits). It reports returned bytes and whether output was truncated. Insecure kubelet-backend TLS is a separate local-policy opt-in and should remain disabled. Rootly AI SRE also rejects private-provider results above its [AI context cap](/private-agent-limits#shared-runtime-and-ai-context) before placing them in model context. Narrow the namespace, selectors, pagination limit, time window, or log bounds and retry instead of requesting one large response. Pod logs are sensitive data. Private Connect encrypts its transport records and filters payloads from transport logs and traces, but returned logs can still enter AI model context, evaluation traces, and investigation history. See [Private Agent data and retention](/ai/data-privacy-for-rootly-ai#how-is-private-agent-data-logged-and-retained) for the canonical storage, asynchronous cleanup, and downstream retention boundaries. ## Resource sizing and performance Start a combined agent with requests around `100m` CPU and `128Mi` memory and limits around `1` CPU and `512Mi` memory, then tune from observed usage. These are starting points, not universal requirements. Load is controlled by independent runtime concurrency, list pagination, watch duration, encoded result size, log byte/time, and AI-context bounds. See [Private Agent Limits](/private-agent-limits) for defaults and hard ceilings. Request deadlines and cancellation propagate into Kubernetes calls. Large object metadata, high event volume, and concurrent watches can still increase CPU and memory. Lower `runtime.maximum_concurrency` for a small Pod or an API server with a tight request budget; excess work stays queued in Rootly until capacity is available. This reduces peak CPU, memory, and Kubernetes API pressure at the cost of parallel investigation throughput. Monitor container CPU, working-set memory, restarts, readiness, Kubernetes API throttling, invocation latency, and truncated-result counts. Narrow selectors, namespaces, time windows, and page sizes before increasing resources. ## Network requirements Allow outbound TCP 443 from the agent to its configured Rootly gRPC origin and access from the Pod to the cluster's Kubernetes API endpoint. The default origin is `https://connect.rootly.com`. The control-plane channel requires gRPC over TLS and HTTP/2, including long-lived bidirectional streams; opening TCP 443 alone is not sufficient if a firewall or proxy downgrades or blocks HTTP/2. Enrollment, token refresh, provider registration, and work use this same endpoint. No inbound internet access, public Service, or Ingress is required. If Rootly supplies a regional or custom endpoint, set `rootly.grpc_url` (or `ROOTLY_PRIVATE_AGENT_GRPC_URL`) and allow outbound access to that origin **instead of** `connect.rootly.com`. The value must be an origin without a path, query, or credentials. When omitted, legacy `rootly.api_url` selects the same gRPC endpoint, defaulting to `https://connect.rootly.com`. This override changes only the Private Agent control-plane transport origin; it does not change the region where Rootly AI processes or retains investigation data. See [Data Privacy for Rootly AI](/ai/data-privacy-for-rootly-ai#where-is-data-processed) for the current processing-region policy. For enterprise proxies or TLS inspection, validate HTTP CONNECT and streaming behavior with your Rootly representative. Use a configured `rootly.ca_bundle_file` for an additional trusted CA when needed; certificate and hostname verification remain enabled. There is no HTTP polling fallback if gRPC is blocked. TLS certificates are validated normally. Plain HTTP is rejected unless an explicit development-only override is enabled.