Skip to main content
Early Preview: Rootly Private Agent is under active development and available only to approved customers. Features, configuration, limits, and APIs may change before general availability. Confirm the approved agent and backend versions with your Rootly representative before production use.
The native Kafka provider gives Rootly AI SRE typed diagnostics for Apache Kafka, Amazon MSK, and protocol-compatible Redpanda clusters. It uses the Kafka wire protocol directly and does not require JMX, a Kafka CLI, or an HTTP proxy. One agent can connect to multiple Kafka clusters. Each entry has its own stable provider ID, bootstrap servers, credentials, topic boundary, health, and execution capacity. Rootly routes a tool call to one exact registered instance; it cannot supply another broker, credential, or topic-policy configuration.

​
Configure a cluster

Use a dedicated read-only Kafka principal. Create credentials separately, mount them as files, and add one provider entry per cluster:
The agent image runs as UID/GID 65532. For Kubernetes Secret volumes using mode 0440, set Pod-level securityContext.fsGroup: 65532. Omit ca_bundle_file when the broker certificate chains to a CA already trusted by the image. Provider IDs are model-visible routing context. Include the application, environment, cluster, or region that distinguishes the instance, and keep IDs unique across the Rootly account.

​
Authentication and TLS

TLS is required by default. The provider supports:
  • TLS with the system trust store and an optional private CA bundle;
  • mutual TLS with client_certificate_file and client_key_file;
  • SASL/PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 with mounted username and password files; and
  • Amazon MSK IAM using the standard AWS credential chain.
SASL authentication always requires TLS. Credential files are reread when Kafka establishes a SASL session. There is no skip-certificate-verification option. allow_plaintext: true is intended only for unauthenticated, disposable local testing.

​
Amazon MSK IAM

For Amazon MSK with IAM access control, use the TLS IAM bootstrap brokers, usually on port 9098, and configure the cluster’s AWS Region:
The agent obtains credentials through the AWS default credential chain, such as an ECS task role, EKS Pod Identity or IRSA role, or EC2 instance profile. Grant only the kafka-cluster connect, describe, and read operations needed by the enabled diagnostics and allowed topics. The local topic policy is an additional boundary, not a replacement for MSK IAM authorization.

​
Available tools

All Kafka tools are sensitive reads. Administrative metadata tools can require additional broker permissions; omit those grants when they are not needed. SCRAM passwords are never returned. kafka.read_messages is not advertised unless allow_message_reads is true. It reads explicit partitions and offsets without joining a consumer group or committing offsets. Record values require the separate include_message_values opt-in; otherwise the result contains bounded metadata without values. Message evidence can contain sensitive customer data and can enter AI context and investigation history under the Private Agent retention model.

​
Topic scope and limits

allowed_topics is required and uses local glob patterns. Omitting it or providing an empty list fails provider registration. Use allowed_topics: ["*"] only when the dedicated Kafka principal is intentionally allowed to inspect every non-internal topic. denied_topics uses the same glob syntax and a deny match always wins. Internal topics remain hidden unless include_internal_topics is true. Rootly cannot widen these settings remotely, and every topic-scoped operation checks them before issuing its Kafka request. Concurrency, result size, returned items, messages, message bytes, scanned records, timeout, and stuck-transaction age are independently bounded. See Private Agent Limits for defaults and hard ceilings. Use broker-side quotas and a least-privilege identity as the primary resource and authorization boundaries.

​
Health, routing, and compatibility

Health performs authenticated Kafka metadata discovery and is cached for 15 seconds. A TLS, authentication, authorization, DNS, or broker-availability failure marks only that configured instance unhealthy. Other Kafka and provider instances remain independently routed. In AI → Configurations → Private Agent, verify the provider ID, health, last check, and advertised capabilities. kafka.read_messages appears only when message reads are locally enabled. Compatibility CI exercises Apache Kafka 3.9, 4.2, and 4.3 and Redpanda 25.3 and 26.2 with deterministic records and real protocol calls. Rootly also validates Amazon MSK IAM from an ephemeral ECS task in us-east-1, exercising the same TLS, task-role credential, and authenticated metadata path used by the agent.