> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rootly.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Private Agent for Internal HTTP APIs (Early Access)

> Connect Rootly AI SRE to fixed, locally allowlisted internal HTTP APIs.

<Warning>
  **Early Preview:** Rootly Private Agent is under active development and available only to approved customers. Features, configuration, limits, and APIs may change before general availability. Confirm the approved agent and backend versions with your Rootly representative before production use.
</Warning>

The HTTP provider lets Rootly AI SRE call an internal REST or HTTP API without exposing that API to the internet or building a dedicated adapter. The agent remains inside your network and carries bounded results over its outbound Private Connect gRPC connection.

This is not a general web fetcher or forward proxy. Each provider entry fixes one origin, credentials, allowed methods, allowed path prefixes, headers, and budgets. An AI tool call cannot select another hostname, attach credentials, enable a proxy, or follow a redirect.

## Configure multiple APIs

Add one array entry per API, tenant, or credential identity. IDs must be stable and unique across all providers registered by your Private Agents.

```yaml theme={null}
providers:
  http:
    - id: inventory-production
      base_url: https://inventory.production.internal
      allowed_methods: [GET, HEAD]
      allowed_path_prefixes:
        - /v1/assets
        - /v1/owners
      health_check_path: /health
      bearer_token_file: /run/secrets/inventory/token
      policy:
        maximum_concurrency: 2
        maximum_request_bytes: 32768
        maximum_response_bytes: 65536
        maximum_timeout_seconds: 15

    - id: incidents-production
      base_url: https://incidents.production.internal/api
      allowed_methods: [GET, POST, PATCH]
      allowed_path_prefixes: [/v2/incidents]
      health_check_path: /health
      header_files:
        X-API-Key: /run/secrets/incidents/api-key
        X-Tenant-Id: /run/secrets/incidents/tenant-id
      allowed_request_headers: [Accept, Idempotency-Key, If-Match]
      exposed_response_headers: [Content-Type, ETag, Location, Retry-After, X-Request-Id]
```

`base_url` may contain a fixed path prefix. The invocation contributes only a relative path, such as `/v2/incidents/42`. Absolute and scheme-relative URLs, embedded query strings, fragments, backslashes, and dot segments are rejected. Safe percent-encoded bytes are repeatedly decoded to one canonical path before both allowlist matching and dispatch, preventing proxies or upstream routers from interpreting an authorized path as another route. Prefix matching follows path-component boundaries: `/v1` does not grant `/v11`.

The provider ID and `allowed_path_prefixes` are non-secret routing context sent to Rootly so AI SRE can choose the correct provider and path. Use a concise, descriptive ID. Registration also includes the provider type, allowed methods, invocation request-header names, exposed response-header names, bounded policy, health, and capabilities. It excludes free-form provider prose, the base URL and health path, credential paths and values, secret header names, and TLS material. Never include credentials or sensitive operational data in registered values.

## Available tools

Only locally enabled verbs are registered as tools.

| Tool           | Rootly sensitivity | Request body |
| -------------- | ------------------ | ------------ |
| `http.get`     | Sensitive read     | Not accepted |
| `http.head`    | Sensitive read     | Not accepted |
| `http.options` | Sensitive read     | Not accepted |
| `http.post`    | Write              | Optional     |
| `http.put`     | Write              | Optional     |
| `http.patch`   | Write              | Optional     |
| `http.delete`  | Write              | Optional     |

Owners and admins can run sensitive reads and writes. AI SRE system investigations can use both when the Private Agent and AI SRE features are enabled. There is no interactive approval pause; the configured API identity, method list, path prefixes, and upstream authorization are the enforcement boundary. Use a least-privilege identity and omit write methods unless investigations genuinely require them.

Mutating calls have at-least-once delivery under uncertain failure. The agent does not automatically retry `POST`, `PUT`, `PATCH`, or `DELETE` inside one execution, including after an OAuth 401. However, if the upstream accepts a mutation and the agent process or network fails before Rootly accepts its completion, the invocation lease can expire and the same stored input can be dispatched again before its deadline. Enable only idempotent mutation endpoints, or supply a stable `Idempotency-Key` that the upstream durably deduplicates. Because a re-leased invocation keeps the same input, a key included in that input remains stable across attempts. Do not enable a non-idempotent mutation when neither control is available.

Inputs contain `path`, optional `query`, optional allowlisted `headers`, optional `body` and `content_type` for write verbs, and an optional `timeout_seconds`. Query names are limited to 255 bytes and the complete encoded query is capped at 24 KiB before dispatch. JSON bodies are encoded as JSON, including a literal `null` body. To send another non-JSON media type, provide the body as a JSON string and set its content type; the decoded UTF-8 string bytes are sent verbatim. The media type does not add a binary decoding mode, so arbitrary binary request bodies are not supported.

Responses include the status code, locally exposed headers, body encoding, truncation status, and a bounded body. JSON is returned structurally, UTF-8 as text, and other bytes as base64. `maximum_response_bytes` bounds the complete serialized tool result, including headers and encoding overhead. The agent omits exposed headers and truncates the body as needed to stay within that limit, marking the result as truncated. HTTP error statuses are returned as evidence so AI SRE can reason about authorization, validation, and upstream failures.

Returned response bodies and exposed headers are sensitive evidence and are not generically redacted, including bodies from non-2xx responses. They can enter AI context, evaluation traces, and investigation or conversation history under the [Private Agent retention model](/ai/data-privacy-for-rootly-ai#how-is-private-agent-data-logged-and-retained). Allowlist only endpoints whose responses are acceptable under that model, and avoid paths that can return credentials, secrets, stack traces, echoed input, or unnecessary personal data.

## Credentials and TLS

Secret values are mounted files, never inline YAML values. Bearer, custom authorization, basic-auth, and secret header files are reread for each request, allowing projected Secret rotation without restarting the agent.

Choose at most one mechanism that owns the `Authorization` header. OAuth 2.0 client credentials is one of these mutually exclusive mechanisms and cannot be combined with bearer, custom-scheme, or basic authentication:

```yaml theme={null}
# Bearer
bearer_token_file: /run/secrets/internal-api/token

# A custom scheme, producing: Authorization: Token <file contents>
authorization_scheme: Token
authorization_file: /run/secrets/internal-api/token

# Basic authentication
username_file: /run/secrets/internal-api/username
password_file: /run/secrets/internal-api/password
```

Secret-backed non-authorization headers can accompany any of those mechanisms:

```yaml theme={null}
header_files:
  X-API-Key: /run/secrets/internal-api/api-key
  X-Tenant-Id: /run/secrets/internal-api/tenant-id
```

These top-level headers are sent only to the resource API. They are never sent
to a distinct OAuth token endpoint.

For OAuth 2.0 client credentials:

```yaml theme={null}
oauth2_client_credentials:
  token_url: https://identity.internal/oauth2/token
  client_id_file: /run/secrets/internal-api/client-id
  client_secret_file: /run/secrets/internal-api/client-secret
  scopes: [incidents.read, incidents.write]
  audience: https://incidents.internal
  client_auth_method: client_secret_basic
  # Optional token-endpoint-only headers
  header_files:
    X-Identity-Tenant: /run/secrets/internal-api/identity-tenant
```

`client_auth_method` can be `client_secret_basic` or `client_secret_post`. Tokens are cached until shortly before expiry. A 401 invalidates the cached token. Safe `GET`, `HEAD`, and `OPTIONS` calls retry once with a newly fetched token. Mutating calls return the 401 without an automatic replay and use the new token on the next invocation. Client credential files are reread when fetching a token. The token endpoint is fixed local configuration and cannot be changed by an invocation. If that endpoint needs a tenant or API-key header, configure it inside `oauth2_client_credentials.header_files`; those values are reread on each token exchange and are never sent to the resource API.

Private roots and mutual TLS are supported:

```yaml theme={null}
ca_bundle_file: /run/secrets/internal-api/ca.pem
client_certificate_file: /run/secrets/internal-api/client.crt
client_key_file: /run/secrets/internal-api/client.key
```

HTTPS uses TLS 1.2 or newer with hostname and certificate verification. There is no skip-verification setting. Plain HTTP requires `allow_insecure_http: true`; use it only for a deliberately trusted local test path and never assume the agent proves that a hostname is private.

The CA bundle is loaded when the provider starts and requires an agent restart after rotation. The client certificate and key are reread for each new TLS handshake, but an established connection can continue using its existing certificate; restart the agent when an mTLS cutover must take effect immediately.

`Authorization`, proxy authorization, cookies, `Host`, forwarding headers, method/path override headers, content length, `Content-Type`, transfer encoding, and hop-by-hop headers cannot be configured as secret headers or supplied through the invocation `headers` object. Request `Content-Type` is set only through the dedicated `content_type` argument. Other invocation headers must appear in `allowed_request_headers`. A header name cannot appear in both `header_files` and `allowed_request_headers`; the agent rejects that configuration instead of making a fixed credential or tenant header model-controlled. Authentication challenges, `Authorization`, and `Set-Cookie` cannot be exposed as response headers. Environment `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` settings are ignored, and redirects are returned without being followed, preventing credentials from crossing to another origin.

<Warning>
  The agent image runs as UID/GID `65532`. For Kubernetes Secret volumes using mode `0440`, set Pod-level `securityContext.fsGroup: 65532`. Give only the agent identity equivalent read access on other platforms; do not make secret files world-readable.
</Warning>

## Local policy and limits

`allowed_methods` and `allowed_path_prefixes` are mandatory. Missing grants fail startup. When omitted, request headers default to `Accept`, `Idempotency-Key`, `If-Match`, and `If-None-Match`, while response headers default to `Content-Type`, `ETag`, `Last-Modified`, `Location`, `Retry-After`, and `X-Request-Id`. Any explicitly configured `allowed_request_headers` or `exposed_response_headers` list replaces its defaults; use `[]` to allow no invocation request headers or return no upstream response headers, and repeat every default that should remain when adding another name.

See [Private Agent Limits](/private-agent-limits#internal-http-apis) for concurrency, request, response, timeout, query, path, and header ceilings. Rootly independently applies the lower of its reviewed limits and the policy advertised by the agent. The agent checks the same local grants again immediately before sending an invocation request.

The provider has no disk cache, offline result store, unbounded queue, or cross-provider credential sharing. Configure a separate provider entry when the origin, tenant, credential, purpose, or grants differ.

## Health and troubleshooting

The health probe sends an authenticated `GET` to `health_check_path`, which defaults to `/`. It is explicit local configuration and is not constrained by invocation `allowed_methods` or `allowed_path_prefixes`; this permits a dedicated health endpoint without exposing it as an AI tool. The path is appended to any fixed path in `base_url`, so `/health` with a base URL ending in `/api` calls `/api/health`. Choose a cheap, side-effect-free path that is valid for the configured credential. Explicit health endpoints must return a 2xx response. A 404 from the default `/` probe is accepted as origin reachability because many internal APIs do not serve a root route. Redirects, other non-2xx responses, connection failures, TLS failures, and timeouts mark the provider unhealthy.

If a provider is unhealthy:

1. verify DNS and network reachability from the agent Pod;
2. verify mounted files exist, are regular files, and are readable by UID/GID `65532`;
3. verify the configured CA and certificate names match the endpoint;
4. call the health path locally with the same identity;
5. inspect the provider health in **AI → Configurations → Private Agent** and the agent logs.

One unhealthy HTTP instance does not prevent other configured providers from starting. Overall `/readyz` remains stricter and reports not ready when any provider is unhealthy.

## Validation coverage

The agent unit suite covers fixed-origin routing, path and header enforcement, request and response bounds, redirects, timeouts, JSON, text and binary responses, bearer rotation, OAuth refresh, private CA certificates, and mTLS. Its container-backed integration starts an authenticated CRUD API with SQLite, exercises persisted state and every supported verb, and runs the provider under the Go race detector.