> ## Documentation Index > Fetch the complete documentation index at: https://docs.rootly.com/llms.txt > Use this file to discover all available pages before exploring further. # Private Agent for Kafka (Early Access) > Give AI SRE bounded, read-only access to Apache Kafka, Amazon MSK, and Redpanda clusters inside your network. **Early Preview:** Rootly Private Agent is under active development and available only to approved customers. Features, configuration, limits, and APIs may change before general availability. Confirm the approved agent and backend versions with your Rootly representative before production use. The native Kafka provider gives Rootly AI SRE typed diagnostics for Apache Kafka, Amazon MSK, and protocol-compatible Redpanda clusters. It uses the Kafka wire protocol directly and does not require JMX, a Kafka CLI, or an HTTP proxy. One agent can connect to multiple Kafka clusters. Each entry has its own stable provider ID, bootstrap servers, credentials, topic boundary, health, and execution capacity. Rootly routes a tool call to one exact registered instance; it cannot supply another broker, credential, or topic-policy configuration. ## Configure a cluster Use a dedicated read-only Kafka principal. Create credentials separately, mount them as files, and add one provider entry per cluster: ```yaml theme={null} providers: kafka: - id: events-production-us-east-1 display_name: Production events in us-east-1 bootstrap_servers: - broker-1.kafka.internal:9093 - broker-2.kafka.internal:9093 client_id: rootly-private-agent tls: enabled: true ca_bundle_file: /run/secrets/kafka-production/ca.pem sasl: mechanism: scram-sha-512 username_file: /run/secrets/kafka-production/username password_file: /run/secrets/kafka-production/password policy: allowed_topics: - payments.* - orders.* denied_topics: - payments.secrets* include_internal_topics: false allow_message_reads: false include_message_values: false maximum_concurrency: 2 maximum_timeout_seconds: 20 extraVolumes: - name: kafka-production secret: secretName: rootly-private-agent-kafka-production defaultMode: 0440 extraVolumeMounts: - name: kafka-production mountPath: /run/secrets/kafka-production readOnly: true ``` The agent image runs as UID/GID `65532`. For Kubernetes Secret volumes using mode `0440`, set Pod-level `securityContext.fsGroup: 65532`. Omit `ca_bundle_file` when the broker certificate chains to a CA already trusted by the image. Provider IDs are model-visible routing context. Include the application, environment, cluster, or region that distinguishes the instance, and keep IDs unique across the Rootly account. ## Authentication and TLS TLS is required by default. The provider supports: * TLS with the system trust store and an optional private CA bundle; * mutual TLS with `client_certificate_file` and `client_key_file`; * SASL/PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 with mounted username and password files; and * Amazon MSK IAM using the standard AWS credential chain. SASL authentication always requires TLS. Credential files are reread when Kafka establishes a SASL session. There is no skip-certificate-verification option. `allow_plaintext: true` is intended only for unauthenticated, disposable local testing. ### Amazon MSK IAM For Amazon MSK with IAM access control, use the TLS IAM bootstrap brokers, usually on port `9098`, and configure the cluster's AWS Region: ```yaml theme={null} providers: kafka: - id: events-staging-us-east-1 bootstrap_servers: - boot-example.c1.kafka-serverless.us-east-1.amazonaws.com:9098 tls: enabled: true sasl: mechanism: aws-msk-iam aws_region: us-east-1 policy: allowed_topics: - staging.events.* ``` The agent obtains credentials through the AWS default credential chain, such as an ECS task role, EKS Pod Identity or IRSA role, or EC2 instance profile. Grant only the `kafka-cluster` connect, describe, and read operations needed by the enabled diagnostics and allowed topics. The local topic policy is an additional boundary, not a replacement for MSK IAM authorization. ## Available tools | Area | Tools | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Cluster and topics | `kafka.cluster_metadata`, `kafka.api_versions`, `kafka.health`, `kafka.list_topics`, `kafka.describe_topic`, `kafka.describe_configs` | | Partitions and capacity | `kafka.partition_offsets`, `kafka.partition_state`, `kafka.partition_skew`, `kafka.log_dirs`, `kafka.capacity_summary`, `kafka.list_partition_reassignments` | | Consumer groups | `kafka.list_consumer_groups`, `kafka.describe_consumer_group`, `kafka.consumer_group_offsets`, `kafka.consumer_group_lag`, `kafka.replay_readiness`, `kafka.diagnose_consumer_group` | | Transactions and producers | `kafka.list_transactions`, `kafka.describe_transaction`, `kafka.describe_producers`, `kafka.stuck_transactions` | | Access controls | `kafka.list_acls`, `kafka.describe_client_quotas`, `kafka.describe_scram_users` | | Investigation summaries | `kafka.diagnose_connectivity`, `kafka.diagnose_cluster`, `kafka.diagnose_topic`, `kafka.topic_audit`, `kafka.topic_usage`, `kafka.incident_snapshot` | | Records | `kafka.read_messages`, only when locally enabled | All Kafka tools are sensitive reads. Administrative metadata tools can require additional broker permissions; omit those grants when they are not needed. SCRAM passwords are never returned. `kafka.read_messages` is not advertised unless `allow_message_reads` is true. It reads explicit partitions and offsets without joining a consumer group or committing offsets. Record values require the separate `include_message_values` opt-in; otherwise the result contains bounded metadata without values. Message evidence can contain sensitive customer data and can enter AI context and investigation history under the [Private Agent retention model](/ai/data-privacy-for-rootly-ai#how-is-private-agent-data-logged-and-retained). ## Topic scope and limits `allowed_topics` is required and uses local glob patterns. Omitting it or providing an empty list fails provider registration. Use `allowed_topics: ["*"]` only when the dedicated Kafka principal is intentionally allowed to inspect every non-internal topic. `denied_topics` uses the same glob syntax and a deny match always wins. Internal topics remain hidden unless `include_internal_topics` is true. Rootly cannot widen these settings remotely, and every topic-scoped operation checks them before issuing its Kafka request. Concurrency, result size, returned items, messages, message bytes, scanned records, timeout, and stuck-transaction age are independently bounded. See [Private Agent Limits](/private-agent-limits#kafka) for defaults and hard ceilings. Use broker-side quotas and a least-privilege identity as the primary resource and authorization boundaries. ## Health, routing, and compatibility Health performs authenticated Kafka metadata discovery and is cached for 15 seconds. A TLS, authentication, authorization, DNS, or broker-availability failure marks only that configured instance unhealthy. Other Kafka and provider instances remain independently routed. In **AI → Configurations → Private Agent**, verify the provider ID, health, last check, and advertised capabilities. `kafka.read_messages` appears only when message reads are locally enabled. Compatibility CI exercises Apache Kafka 3.9, 4.2, and 4.3 and Redpanda 25.3 and 26.2 with deterministic records and real protocol calls. Rootly also validates Amazon MSK IAM from an ephemeral ECS task in `us-east-1`, exercising the same TLS, task-role credential, and authenticated metadata path used by the agent.