Score-tie ordering differs from trec_eval (docno ascending vs. descending)

[profsmucker]

Hi Charlie — a small heads-up about how compatibility.py orders the run prior to scoring.

In compatibility.py (currently lines 111–112):

run[topic].sort()
run[topic].sort(key=lambda docno: scores[topic][docno], reverse=True)

The first sort puts docnos in ascending lexical order, and the second (stable) sort by score descending preserves that for tied scores — so documents with equal scores end up ordered by docno ascending.

trec_eval uses the opposite tie-break. Its ranking comparator comp_sim_docno (in form_res_rels.c, and identically in form_res_rels_jg.c / form_prefs_counts.c) sorts by score descending and breaks ties with strcmp(ptr2->docno, ptr1->docno) — i.e. docno descending:

if (ptr1->sim > ptr2->sim) return (-1);
if (ptr1->sim < ptr2->sim) return (1);
return (strcmp(ptr2->docno, ptr1->docno));   /* note reversed args -> docno descending */

The consequence is that, for runs with tied scores, compatibility.py can order the tied documents in the opposite order from trec_eval, which can change the run sequence fed to RBO and therefore the reported compatibility scores. Matching trec_eval would just need run[topic].sort(reverse=True) on the first line.

Is the ascending tie-break intentional, or worth aligning with the trec_eval convention? Thanks!

---

Note: this issue was written by Claude (Anthropic's Claude Code) on behalf of, and reviewed by, the account owner (Mark D. Smucker).