Turnbull identifiability: replace the mass heuristic with the Vardi/Wang connectivity criterion

[derrynknife]

Follow-up to #308, which is closed. That issue is resolved as far as the bugs went; what remains is a question of rigour, and the literature turns out to answer it better than the heuristic now shipped.

What is in place

A Turnbull fit whose mass rests on a non-identifiable direction warns and reports model.exploitable_mass (#308, 8ec239c). The trigger is empirical: warn when more than 90% of the fitted mass sits in intervals that some observation could have failed in but that precede another observation's entry.

That threshold was chosen by measurement, not principle. Across 240 simulated samples:

	min	median	max
degenerate fits	0.374	0.994	0.9995
healthy fits	0.000	0.387	0.836

0.9 sits in the gap and cost no false alarms; 0.7 cost 9% and 0.5 cost 40%. It works, but it is a tuned cut-off standing in for a property that is actually decidable.

The literature has an exact criterion

Vardi (1985) and Wang (1991) give a graphical condition for the NPMLE under truncation. Build a directed graph from the observations and check its connectivity:

• not connected, but every subgraph strongly connected → the NPMLE exists but is not unique
• at least one subgraph not strongly connected → the NPMLE does not exist

The behaviour measured in #308 — the likelihood climbing towards a boundary it never attains, converging in value (-6.1377) while the parameters keep drifting — is the second case. "The EM never settles" and "the NPMLE does not exist" look like the same statement.

This would replace a threshold with a decision: exists / exists-but-not-unique / does not exist, computed from the data before any iteration, with no constant to tune.

Woodroofe (1985), Ann. Statist. 13:163-177, gives the underlying identifiability condition, π(t) = P(U ≤ t ≤ V) > 0 across the support — every survival time needs positive probability of being observed. Worth citing in whatever the code ends up saying, since it frames the non-identifiability as expected behaviour rather than a defect.

Our combination is a known hard case

There is a published note, "A Note on Inconsistency of NPMLE of the Distribution Function from Left Truncated and Case I Interval Censored Data" (Lifetime Data Analysis, Springer). I have only the title -- the fetch returned 403 -- but it establishes that left truncation combined with interval censoring is documented as yielding an inconsistent NPMLE.

So surpyval is not uniquely wrong here. This is a recognised limitation of the estimator for this data shape, which is an argument for reporting it precisely rather than trying to estimate around it.

Two honest caveats

• The Vardi/Wang condition is stated for the truncation-only setting (length-biased / multiplicative censoring). Extending it to left censoring plus truncation is real work, not a transcription. Frydman (1994) and Alioum & Commenges (1996) are the relevant extensions to interval-censored-and-truncated data and are the place to start.
• I have not verified that the configuration in Turnbull: left censoring combined with left truncation is wrongly rejected or fails to converge #308 formally maps to "not strongly connected". It fits the observed behaviour, but that is inference, not proof. Anyone implementing this should check it on the six-point reproducer first, which is in the suite.

Still open: what a non-identifiable fit should return

Independent of detection. Currently the boundary estimate is returned with a warning. The alternative is a constrained conditional estimate that assumes no mass between the entry times.

There is precedent for conditioning: delayed-entry Kaplan-Meier already evaluates survival conditional on having survived to entry, which is the basis of the (start, stop, event) counting-process form. So conditioning would not be an invention, though it would change results for existing truncated fits and needs its own verification pass.

Scope

Three xfails in test_turnbull_input_sweep.py point here (the all_censoring_types_truncated sweep cases and the six-point test_distinct_entry_times_with_left_censoring_round_trip). They are strict, so they report the day this changes.

References

• Woodroofe, Estimating a Distribution Function with Truncated Data, Ann. Statist. 13 (1985) 163-177
• Vardi/Wang graphical existence-and-uniqueness condition, summarised in On nonparametric maximum likelihood estimation with double truncation
• A Note on Inconsistency of NPMLE ... Left Truncated and Case I Interval Censored Data
• Delayed entry as conditional survival

🤖 Generated with Claude Code

https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts