[C API] Add support for filtered TopK search in C API#352
Conversation
- Introduced `svs_id_filter_interface` to define filtering operations. - Implemented `svs_index_search_topK` to support an optional ID filter for search operations. - Updated existing search functions to use the new filtered search capabilities. - Added a new source file `filtered_search.hpp` containing the logic for filtered top-K search. - Modified existing samples and tests to demonstrate and validate the new filtering functionality. - Marked the previous `svs_index_search` function as deprecated, directing users to use `svs_index_search_topK` instead.
|
Tick the box to add this pull request to the merge queue (same as
|
There was a problem hiding this comment.
Pull request overview
This PR extends the C API’s search functionality to support filtered top‑K queries via a user-provided ID filter interface, while also introducing a new svs_index_search_topK entrypoint and deprecating the older svs_index_search API.
Changes:
- Added a new C-facing ID filter interface (
svs_id_filter_interface) and a newsvs_index_search_topK(..., svs_id_filter_i id_filter, ...)API. - Implemented filtered top‑K search logic (
filtered_search.hpp) and plumbed it through the C++ index wrappers used by the C runtime. - Updated C samples and Catch2 tests to use
svs_index_search_topK, and added new tests exercising filtered searches (including restrictive filters and deprecated wrapper coverage).
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| bindings/c/include/svs/c_api/svs_c.h | Adds ID filter interface types and declares svs_index_search_topK; deprecates svs_index_search. |
| bindings/c/include/svs/c_api/svs_c_config.h | Introduces SVS_DEPRECATED(...) macro used by the public C header. |
| bindings/c/src/svs_c.cpp | Implements svs_index_search_topK and makes svs_index_search delegate to it. |
| bindings/c/src/types_support.hpp | Adds C→C++ adapter types for the new ID filter interface. |
| bindings/c/src/filtered_search.hpp | New filtered top‑K search implementation used by the C runtime’s Vamana wrappers. |
| bindings/c/src/index.hpp | Extends index wrapper virtual search to accept optional ID filter; wires filtered search path. |
| bindings/c/tests/c_api_index.cpp | Migrates tests to svs_index_search_topK; adds new filtered-search and deprecated-wrapper tests. |
| bindings/c/tests/c_api_dynamic_index.cpp | Migrates dynamic index tests to svs_index_search_topK; adds filtered-search tests. |
| bindings/c/samples/simple.c | Updates sample to use svs_index_search_topK (no filter). |
| bindings/c/samples/save_load.c | Updates save/load sample to use svs_index_search_topK (no filter). |
| bindings/c/samples/dynamic.c | Updates dynamic sample to use svs_index_search_topK (no filter). |
| bindings/c/CMakeLists.txt | Adds the new src/filtered_search.hpp to the C API sources list. |
| // if hit rate is less than filter_rate, return 0 - which means we should not even start | ||
| // the search | ||
| if (!hit_rate_sufficient(sample_size, hits, filter_rate)) { | ||
| return 0; | ||
| } | ||
| return estimate_batch_size(sample_size, hits, goal, hint, limit); |
There was a problem hiding this comment.
As expected: if actual 'sampled' filter rate is less than client's estimation, search should be interrupted to avoid too high latency.
See filter_stop approach in C++ Runtime.
|
@yuejiaointel, to help you review:
|
…functionality and memory management
Thx, this helps a lot! could we also add these in a comment to document the corresponding names, since they have different variable names and filter_stop and filter_estimate_batch are 2 controls but in c api they are both controlled by filter_rate |
| std::uniform_int_distribution<size_t> dist(0, max_id); | ||
| auto sample_generator = [&]() -> size_t { | ||
| static constexpr size_t max_attempts = 4; | ||
| for (size_t attempt = 0; attempt < max_attempts; ++attempt) { |
There was a problem hiding this comment.
Seems slightly different from #309 approach my understanding:
- 309 draws [0, size()-1] = [0, live_count-1]
- this pr draws [0, max_id] = [0, highest_id_ever_inserted]
they are same if id are continues, but if a delete is called and there is hole, - 309 will continue until it collected enough samples
- this pr will try 4 times and if all 4 missed, just return -1
Maybe we could add the logic to still continue to collect
There was a problem hiding this comment.
In case of DynamicVamana, vector id (label) is user-provided and can have any value, e.g. a value significantly bigger than index size. We cannot rely, that ids in dynamic index are continues anywhen. And we cannot rely that any of id will be in the [0, size() ) interval
The most reliable solution should collect all external ids from the index then select from them, but collecting all ids is expensive operation, so there is the trade-off between perfornmance and reliability.
svs_id_filter_interfaceto define filtering operations.svs_index_search_topKto support an optional ID filter for search operations.filtered_search.hppcontaining the logic for filtered top-K search.svs_index_searchfunction as deprecated, directing users to usesvs_index_search_topKinstead.