EmbeddingSimilarity._CACHE is a class attribute keyed only by the normalized input text (py/autoevals/string.py):
_CACHE = {} # class-level, shared across all instances
...
def _embed(self, value):
value = normalize_value(value, maybe_object=False)
with self._CACHE_LOCK:
if value in self._CACHE:
return self._CACHE[value]
result = run_cached_request(..., input=f"{self.prefix}{value}", **self.extra_args)
Two consequences:
- Cross-model poisoning: two scorers configured with different embedding models share cache entries — the second scorer silently reuses vectors produced by the first scorer's model, producing wrong similarity scores with no error.
- Prefix collisions: the request embeds
f"{self.prefix}{value}" but the cache key is the un-prefixed value, so instances with different prefix values collide the same way.
Suggested fix: key the cache on (model, prefix, value) (or the full request), e.g. (self.extra_args.get("model"), self.prefix, value).
Found while writing the regression tests for #166 (PR #209) — the tests had to use unique input strings to avoid the embeddings API call being skipped entirely.
EmbeddingSimilarity._CACHEis a class attribute keyed only by the normalized input text (py/autoevals/string.py):Two consequences:
f"{self.prefix}{value}"but the cache key is the un-prefixedvalue, so instances with differentprefixvalues collide the same way.Suggested fix: key the cache on
(model, prefix, value)(or the full request), e.g.(self.extra_args.get("model"), self.prefix, value).Found while writing the regression tests for #166 (PR #209) — the tests had to use unique input strings to avoid the embeddings API call being skipped entirely.