perf(cli): stop importing sentence_transformers for every command - #60
Conversation
5b94fe5 to
d726ec0
Compare
|
Rebased onto the rebased #56 (this PR is stacked on it, so #56 needs to merge first — until then the diff here shows #56's commits too). Resolved one conflict in Measured effect of this PR — importing
That's ~12x on every CLI invocation, including Incidental confirmation that this is worth having: while setting up to verify these PRs I couldn't run the parser tests without installing torch + sentence-transformers, purely because of this import chain. On this branch they run without the ML stack present. Verified against |
Importing ast_rag.cli took 3.65s, of which 3.05s was sentence_transformers (and transitively torch and transformers), reached via services/__init__ -> embedding_manager. Only semantic search needs it. Every other command -- goto, callers, refs, sig, blocks, cache-stats -- paid the full cost and never used the model. Moves the import into _get_model(), where it is actually needed, with TYPE_CHECKING for the annotations. import ast_rag.cli 3.65s -> 0.61s Measured end to end over 15 CLI scenarios against a live index: median command latency 3.068s -> 0.558s fastest command 2.864s -> 0.487s Semantic search is unaffected; the model loads on first use and 'query' still returns its full result set.
d726ec0 to
0237383
Compare
Found while running a full UAT sweep of the CLI against a live index — every command had a hard ~2.9s floor, including pure graph lookups.
Stacked on #50 and #56. Review those first; this branch contains them.
Cause
import ast_rag.clicosts 3.65s, and-X importtimeattributes 3.05s of it tosentence_transformers(and transitively torch/transformers), reached viaservices/__init__→embedding_manager:Only semantic search needs the model.
goto,callers,refs,sig,blocks,cache-statsare pure Neo4j lookups and never touch it — but they all paid for it.Fix
Move the import into
_get_model(), where it's actually used, withTYPE_CHECKINGfor the annotations. Same shape as thewatchdogfix in #50.Measured end to end
15 CLI scenarios against a live index (raged indexing itself — 965 nodes, 3,782 edges):
gotocallersquerystill pays the model load on first use, as it must — it's unchanged at ~12s cold, and it still returns its full result set, so the lazy path is exercised and works.For an interactive tool, and especially for an agent calling
goto/callersin a loop, a 5.5x cut on the common path is the difference between usable and not.Verification
Baseline on
mainis3 failed, 174 passed— same three pre-existing failures (fixed in #51). No test changes here; the existing suite covers that embeddings still work, and the UAT sweep confirmsquerybehaves identically.