fix(imap): confirm retired-UID tail via message count before flagging

[odiechoo]

Summary

The empty-enumeration guard added in 2.0.1 (#340) produces a persistent false positive on Gmail (and other servers that retire UIDs). When a folder's highest surviving UID sits below UIDNEXT - 1 — which happens routinely on Gmail whenever messages are relabeled, archived, or moved out of a folder — every sync logs a "Refusing to advance highest_uid" warning that never clears, and highest_uid stays pinned below UIDNEXT indefinitely.

Mailbox 'Work': UID FETCH <start>:* returned no UIDs but server UIDNEXT=<n> (<k> messages in range). Refusing to advance highest_uid to avoid skipping them; the next sync will retry.

The range is genuinely empty (the UIDs were retired, not withheld), so there is nothing for the retry to recover — the warning recurs forever.

Root cause

empty_enumeration_anomaly (crates/core/src/imap/executor.rs) decides purely on UIDNEXT > start_uid. That assumes a UID gap between highest_uid and UIDNEXT implies undelivered messages. On Gmail this is false: UIDNEXT only ever increases, and relabeling / archiving / moving a message retires its UID in that folder without lowering UIDNEXT. So a folder whose last surviving message is UID 337 can sit at UIDNEXT=344 with 338–343 being retired (empty) slots. UID FETCH 338:* correctly returns nothing, but the guard reads that as "6 messages I'm refusing to skip" and re-flags every sync.

The ({} messages in range) figure is UIDNEXT - start_uid, i.e. a slot count, not a confirmed message count — for a retired tail the real count is 0.

Reproduction

Gmail account synced folder-by-folder (individual labels).
Remove the label from (or archive/move) the message(s) holding the top UIDs of a folder, so the highest surviving UID is below UIDNEXT - 1.
No new mail arrives in that folder.
Every sync logs the "Refusing to advance highest_uid" warning for that folder; it never clears (only real new mail, which pushes a non-empty {start}:*, would).

Impact

No data loss — the archive is append-only and complete for these folders; the tail is genuinely empty. But it produces permanent per-folder error spam and pins highest_uid, which is indistinguishable at a glance from a real truncation/loss anomaly. Users cannot tell benign retired tails from a genuine problem.

Fix

The guard cannot distinguish a truncated/throttled empty (real mail — must not skip) from a retired-UID tail (benign) from UIDNEXT alone. Confirm with a signal immune to SEARCH/FETCH truncation: the message count. The EXAMINE response already carries exists (server count), and the local stored count is equally authoritative.

When the enumeration is empty and the guard would fire:

If examined.exists <= local_stored_count → the folder holds no more mail than we already store → retired tail → advance highest_uid to UIDNEXT - 1.
Else → server genuinely has more than we hold → real gap → keep the current refuse-and-retry behavior.

This preserves the anti-truncation protection while eliminating the false positive. It's conservative on the oversized-skipped edge: if a folder has messages skipped for max_email_size_bytes, exists can exceed the local count and the guard simply falls through to the retry path — never skipping mail.

Changes: adds a cheap IndexManager::count_for_mailbox (Tantivy Count collector, no ID materialization), and a resolve_empty_enumeration helper called from both fetch_new_mail_range and fetch_new_mail_with_before.

[cla-assistant]

All committers have signed the CLA.

[rustmailer]

A UID gap by itself should not be treated as an error. Has this issue already been fixed in version 2.0.2?

[odiechoo]

First picture with the error is v2.0.1. Second picture with the Info and not the Error is v2.0.2.

2.0.2 doesn't fix this — it changes the severity, not the logic

Tested 2.0.2 against the exact scenario in this report (Gmail, [Gmail]/Sent Mail, top surviving UID below UIDNEXT - 1). The false positive still fires. What changed is how it's reported, not whether it happens.

Diffed crates/core/src/imap/executor.rs between 2.0.1 and 2.0.2 (commit 02b8741, "fix(imap): downgrade empty tail-fetch from error to info", #342). The change:

tracing::warn! → tracing::info! on both call sites.
DownloadState::append_session_error(...) commented out.
Folder status changed from Failed to Success, with the message reworded to "UID FETCH range returned empty; tail UIDs appear to have been purged. highest_uid unchanged, will retry on next sync."

empty_enumeration_anomaly itself is untouched — still decides purely on uid_next > start_uid, no exists/stored-count comparison. I also checked for resolve_empty_enumeration and IndexManager::count_for_mailbox, the two pieces this report proposed adding — neither exists anywhere in the codebase.

So highest_uid still never advances past a retired-UID tail. The message even says so now, in plain text: "highest_uid unchanged." The only difference is it no longer shows up as an error, so it stops looking broken while remaining exactly as unresolved.

Before/after, same mailbox, same trigger:

2.0.1: lands in Errors (1), red warning card, session error appended. (screenshot attached)
2.0.2: same trigger, now sits under Mailboxes as an info Message, folder status Success, Errors (0) for the run. (screenshot attached)

This resolves the symptom (log spam, folders wrongly marked Failed) but not the bug (permanent highest_uid pinning on any Gmail folder with a retired tail). Reopening / requesting the original fix — comparing examined.exists against the locally stored count before deciding to advance highest_uid — be implemented rather than just muting the log line.

[rustmailer]

Yes, 2.0.2 handles it exactly as you described. But I don't think there's any problem with that approach. When new emails arrive in your mailbox, it will naturally move forward. Also, comparing against local data isn't necessarily accurate, because in the future it may be possible to import emails into an IMAP account.