Skip to main content

​
Overview

Braintrust is a one-click connector that gives Rootly AI read access to your Braintrust organization during investigations. If your team runs AI features in production, Braintrust holds the logs, traces, and eval results that tell you what your models are doing. When an AI-adjacent incident fires, Rootly AI can pull that context directly. Every query is read-only. Rootly AI can inspect logs, traces, and eval outputs but cannot modify anything on the Braintrust side.

​
Before You Start

  • A Braintrust organization with permission to authorize third-party OAuth apps.
  • Read access to the projects, logs, and eval runs you want Rootly AI to be able to see.

​
Setup

Open Configuration → AI → Connectors and click Connect on the Braintrust card. Sign in to Braintrust and authorize Rootly AI for the organization you want to connect. When the flow completes, the card flips to Connected.

​
What Rootly AI Can Read

  • Logs — production AI logs including inputs, outputs, and latency.
  • Traces — the full trace of a multi-step AI request, useful for debugging chain / agent failures.
  • Evals — recent eval run results, useful when an incident is about “did the last deploy regress model quality?”
Rootly AI reaches for Braintrust when the incident is AI-adjacent — a jump in hallucinations, a latency spike in a chain, an eval regressing after a deploy. Pair with GitHub so Rootly AI can also see the code / prompt change that caused it.

​
During an Incident

“The last eval on the checkout-agent shows a 12% hallucination increase after yesterday’s prompt change.” When the incident is AI-adjacent — a jump in bad model output, a chain that’s suddenly slower, an eval that regressed — Rootly AI pulls the most recent eval run, the trace of the failing request, and the metadata linking it to a specific deploy or prompt change. Responders see the AI-specific signal alongside the rest of the incident context, without having to switch to Braintrust.

​
Best Practices

  • Authorize with an account that has broad project access. Restricting the connector to one Braintrust project starves cross-project investigations.
  • Tag logs and eval runs meaningfully. Rootly AI leans on Braintrust metadata to filter to the right subset of logs. Useful tags include environment (production vs staging), deploy version, and model.
  • Pair with a code connector. Braintrust tells Rootly AI how the AI is behaving; GitHub tells it what changed. Together they cover “did prompt engineering, code, or the model itself cause this?”

​
Troubleshooting

The authorizing Braintrust user doesn’t have access to those projects. Confirm the account has read permission on every project Rootly AI should see, then disconnect and reconnect to refresh the OAuth grant.
Braintrust logs are retained for a limited window depending on plan. If you’re asking about older logs, they may have been purged upstream. Rootly AI can only read what Braintrust still stores.
If a Braintrust admin revokes the Rootly AI OAuth grant from Braintrust’s side, Rootly’s card still shows Connected but every query fails. Reconnect from Rootly to reauthorize.

​
Frequently Asked Questions

No. Braintrust is only useful if your team is already running AI features in production and using Braintrust to observe them. Rootly AI works fine without it — the connector just adds AI-observability signal when relevant.
No. Every connector is read-only. Rootly AI can read Braintrust data but never invoke write actions.
Rootly does not maintain a persistent copy of Braintrust data — queries run at investigation time and the response is used in reasoning only. Query results do appear in the LLM traces Rootly logs for quality monitoring, as with every tool call Rootly AI makes. See Data Privacy for Rootly AI for the retention boundary.

Connectors Overview

All connectors and how they fit together.

Datadog

The other observability connector — different signal shape, useful in combination.

Data Privacy for Rootly AI

What Rootly AI sees, retention, and model training controls.