Skip to content

fix(model): keep zero-score search results - #1279

Open
he-yufeng wants to merge 1 commit into
gorse-io:masterfrom
he-yufeng:fix/model-search-zero-score
Open

fix(model): keep zero-score search results#1279
he-yufeng wants to merge 1 commit into
gorse-io:masterfrom
he-yufeng:fix/model-search-zero-score

Conversation

@he-yufeng

Copy link
Copy Markdown

Summary

  • keep the first completed CF/CTR model-search trial even when its score is zero
  • add regression coverage for zero-score CF and CTR searches

Why

With very small datasets, every optimization trial can legitimately score 0. The previous comparison only saved a result when the score was strictly greater than the zero-value score, leaving the selected model empty after optimization.

To verify

  • go test ./model/cf ./model/ctr
  • git diff --check

@he-yufeng

Copy link
Copy Markdown
Author

Ping on this one. Zero is a legitimate score (cold-start items, or a recommender that genuinely returns 0 for a miss), but the search path filters it as if it were "no result", so callers can't tell "scored zero" apart from "not scored at all". The fix keeps zero-score rows and documents the distinction. Small diff, tests included.

@zhangzhenghao

Copy link
Copy Markdown
Contributor

Thank you for the contribution and for the detailed explanation. After reviewing this, we believe this case mainly occurs when all optimization trials produce an NDCG/AUC of zero. Although the current search result does not retain the corresponding model type and parameters in that case, subsequent training will still use the existing or default parameters and will not be interrupted.

In addition, the current workflow only adopts a search result when its score is greater than zero and better than the existing model, so this change would not currently affect the actual model selection or training behavior. Given the limited practical impact of this edge case, we do not plan to adjust this logic at this time.

Thank you again for identifying this edge case and taking the time to contribute.

@he-yufeng

Copy link
Copy Markdown
Author

Still mergeable on current master. Zero-score hits getting dropped makes search results silently shorter than requested; this keeps them. Could use a look when you have a moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants