Skip to content

fix: AnswerRelevancy silently ignores embedding_model (both sync and async) - #209

Open
Rehan (spacesheepinternet) wants to merge 1 commit into
braintrustdata:mainfrom
spacesheepinternet:fix/answer-relevancy-embedding-model
Open

fix: AnswerRelevancy silently ignores embedding_model (both sync and async)#209
Rehan (spacesheepinternet) wants to merge 1 commit into
braintrustdata:mainfrom
spacesheepinternet:fix/answer-relevancy-embedding-model

Conversation

@spacesheepinternet

Copy link
Copy Markdown

Fixes #166.

The bug is broader than #166 describes: embedding_model is ignored in both the sync and async paths, not just sync.

AnswerRelevancy passes the model at eval time:

EmbeddingSimilarity(client=self.client).eval(output=..., expected=..., model=self.embedding_model)

but EmbeddingSimilarity only reads model in its constructor (stored into self.extra_args). At eval time the kwarg lands in the **kwargs of _run_eval_sync / _run_eval_async and is never used — so every AnswerRelevancy run silently embeds with the default model, regardless of what the caller configured.

#164 fixed half of this (the sync path was passing self.model — the judge LLM — instead of self.embedding_model), but the corrected value still went to the call site, where it's dropped. #167 had the right approach (constructor, not call site) but no test coverage; this PR completes it.

Fix: pass model=self.embedding_model to the EmbeddingSimilarity constructor in both paths. embedding_model=None remains safe (the constructor falls back to get_default_embedding_model()).

Tests: two regression tests (sync + async) following the pattern of test_answer_correctness_uses_custom_embedding_model — respx captures the model field actually sent to /v1/embeddings. On main both fail with ['text-embedding-ada-002', ...]; with the fix both pass. The tests pin model="gpt-4o-mini" for question generation so the mocked chat-completions endpoint is used (gpt-5-class defaults route through the Responses API).

Found while preparing to use AnswerRelevancy with a custom embedding model for a RAG eval harness.

…rRelevancy

AnswerRelevancy passed model=self.embedding_model to EmbeddingSimilarity's
eval()/eval_async() call, but EmbeddingSimilarity only reads `model` in its
constructor (into extra_args); at eval time the kwarg lands in **kwargs of
_run_eval_sync/_run_eval_async and is silently dropped. As a result the
embedding_model parameter had no effect in either the sync or async path and
the default embedding model was always used.

Completes the partial fix from braintrustdata#164 (which corrected model -> embedding_model
at the call site but left the parameter unforwarded) and takes the same
approach as braintrustdata#167, adding regression tests for both paths.

Fixes braintrustdata#166

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AnswerRelevancy uses model parameter instead of embedding_model for Embedding call

1 participant