Summary
Experiment.summarize() fetches the score/metric summary from experiment-comparison2 and, if that call raises for any reason, logs a warning and falls back to an empty {} — making a fetch failure indistinguishable from an experiment that genuinely has no scores. Combined with the HTTP adapter's retry logic only covering connection-level exceptions (not HTTP error responses), a single transient 5xx from the backend is enough to silently zero out ExperimentSummary.scores for an experiment that actually scored fine.
This is dangerous for anyone building pass/fail logic on experiment.summary.scores (e.g. CI gates) — that code sees the same shape (scores: {}) whether the eval genuinely bombed or Braintrust just had a blip fetching the summary, with no reliable signal to tell the difference.
Where
Experiment.summarize():
try:
summary_items = state.api_conn().get_json(
"experiment-comparison2",
args={"experiment_id": self.id, "base_experiment_id": comparison_experiment_id},
)
except Exception as e:
_logger.warning(
f"Failed to fetch experiment scores and metrics: {e}\n\nView complete results in Braintrust or run experiment.summarize() again."
)
summary_items = {}Any exception — network error, auth failure, a 5xx with a non-JSON body, anything response_raise_for_status in util.py#L128-L132 raises — is caught here and swallowed into an empty summary. There's no retry, no distinct exception type, and no field on ExperimentSummary indicating the fetch failed vs. legitimately returned nothing.
RetryRequestExceptionsAdapter.send() only retries on exceptions raised during super().send() (urllib3.exceptions.HTTPError, requests.exceptions.RequestException — connection resets, timeouts). A request that completes with a non-2xx status code returns a normal Response, so it never enters this retry loop; raise_for_status() fires later, downstream, outside the adapter entirely. So a transient 5xx gets zero retries, even though this adapter is explicitly built to smooth over "intermittent connectivity issues."
Repro
Any environment where the experiment-comparison2 call can transiently fail (a load balancer/proxy hiccup returning 502/503/504, or any other non-2xx response) reproduces this:
import braintrust
experiment = braintrust.init(project="my-project", experiment="my-experiment")
# ... log some results with scores ...
summary = experiment.summarize()
print(summary.scores) # {} if the comparison fetch happened to 5xx, even though real scores existWe hit this in practice: our eval run scored well (a locally-computed aggregate well above our CI threshold), but experiment.summarize() logged Failed to fetch experiment scores and metrics: <html>... (the backend returned an HTML error page for that one request) and returned scores={}. Our own CI gate — which trusted summary.scores — failed the build even though the run was actually fine, purely because of that one API hiccup.
Suggested fix
Either or both:
- Retry
experiment-comparison2 (and the base-experiment lookup) a bounded number of times with backoff before giving up, consistent with the retry behavior RetryRequestExceptionsAdapter already claims to provide — currently that only covers connection-level failures, not HTTP error statuses.
- Make the failure visible and distinguishable: raise instead of swallowing (or at minimum expose something like
ExperimentSummary.scores_fetch_error, or a sentinel other than {}) so callers building automation on summary.scores can tell "no scores" apart from "couldn't fetch scores," rather than silently treating both the same way.
Environment
braintrust (PyPI) 0.29.0- Confirmed the same code is still present on
main @ 6bee0d63fb5a80b5b37728df36e7409c27018246 (2026-07-30)
Summary
Experiment.summarize()fetches the score/metric summary fromexperiment-comparison2and, if that call raises for any reason, logs awarningand falls back to an empty{}— making a fetch failure indistinguishable from an experiment that genuinely has no scores. Combined with the HTTP adapter's retry logic only covering connection-level exceptions (not HTTP error responses), a single transient 5xx from the backend is enough to silently zero outExperimentSummary.scoresfor an experiment that actually scored fine.This is dangerous for anyone building pass/fail logic on
experiment.summary.scores(e.g. CI gates) — that code sees the same shape (scores: {}) whether the eval genuinely bombed or Braintrust just had a blip fetching the summary, with no reliable signal to tell the difference.Where
Experiment.summarize():Any exception — network error, auth failure, a 5xx with a non-JSON body, anything
response_raise_for_statusinutil.py#L128-L132raises — is caught here and swallowed into an empty summary. There's no retry, no distinct exception type, and no field onExperimentSummaryindicating the fetch failed vs. legitimately returned nothing.RetryRequestExceptionsAdapter.send()only retries on exceptions raised duringsuper().send()(urllib3.exceptions.HTTPError,requests.exceptions.RequestException— connection resets, timeouts). A request that completes with a non-2xx status code returns a normalResponse, so it never enters this retry loop;raise_for_status()fires later, downstream, outside the adapter entirely. So a transient 5xx gets zero retries, even though this adapter is explicitly built to smooth over "intermittent connectivity issues."Repro
Any environment where the
experiment-comparison2call can transiently fail (a load balancer/proxy hiccup returning 502/503/504, or any other non-2xx response) reproduces this:We hit this in practice: our eval run scored well (a locally-computed aggregate well above our CI threshold), but
experiment.summarize()loggedFailed to fetch experiment scores and metrics: <html>...(the backend returned an HTML error page for that one request) and returnedscores={}. Our own CI gate — which trustedsummary.scores— failed the build even though the run was actually fine, purely because of that one API hiccup.Suggested fix
Either or both:
experiment-comparison2(and the base-experiment lookup) a bounded number of times with backoff before giving up, consistent with the retry behaviorRetryRequestExceptionsAdapteralready claims to provide — currently that only covers connection-level failures, not HTTP error statuses.ExperimentSummary.scores_fetch_error, or a sentinel other than{}) so callers building automation onsummary.scorescan tell "no scores" apart from "couldn't fetch scores," rather than silently treating both the same way.Environment
braintrust(PyPI) 0.29.0main@6bee0d63fb5a80b5b37728df36e7409c27018246(2026-07-30)