Skip to content

EmbeddingSimilarity class-level embedding cache ignores model and prefix — cross-model cache poisoning #211

Description

EmbeddingSimilarity._CACHE is a class attribute keyed only by the normalized input text (py/autoevals/string.py):

_CACHE = {}          # class-level, shared across all instances
...
def _embed(self, value):
    value = normalize_value(value, maybe_object=False)
    with self._CACHE_LOCK:
        if value in self._CACHE:
            return self._CACHE[value]
    result = run_cached_request(..., input=f"{self.prefix}{value}", **self.extra_args)

Two consequences:

  1. Cross-model poisoning: two scorers configured with different embedding models share cache entries — the second scorer silently reuses vectors produced by the first scorer's model, producing wrong similarity scores with no error.
  2. Prefix collisions: the request embeds f"{self.prefix}{value}" but the cache key is the un-prefixed value, so instances with different prefix values collide the same way.

Suggested fix: key the cache on (model, prefix, value) (or the full request), e.g. (self.extra_args.get("model"), self.prefix, value).

Found while writing the regression tests for #166 (PR #209) — the tests had to use unique input strings to avoid the embeddings API call being skipped entirely.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions