Finish the type-hint ratchet for univariate.parametric, and fix what it surfaced - #346
Merged
Merged
Conversation
Fourteen signatures across the two distributions, both added to the
enforced list -- nineteen modules, eleven of the 25 distributions.
ExactEventTime confirms the OptimisedFitMixin split generalises. Its
fit(x, c, n, t) is still narrower than the estimation fit, but since
ExactEventTime_ does not inherit that mixin there is no wider fit above
it, so mypy reports nothing. It needed no type: ignore and no
accept-and-reject, unlike Bernoulli and Binomial before the split.
CustomDistribution needed restructuring rather than only annotating. It
assigned its distribution functions onto the instance -- self.Hf = fun,
then lambdas for hf, sf, ff and df -- which stopped being possible once
OptimisedFitMixin declared those names for its own use, because a
subclass inherits the declarations and assigning to an inherited method
is an error. The function is stored as _fun and the five are real
methods delegating to it. Equivalent by construction, since the old
self.Hf = fun was an unbound instance attribute; checked numerically
all the same, gradients included, because elementwise_grad is involved.
Surveying _parameter_initialiser for its return type found a bug.
Rayleigh is the only single-parameter distribution here and returned
its sigma seed as a bare scalar, so np.array(init) in _initial_guess
gave a 0-dimensional array rather than a length-1 one. The lfp and zi
paths append their p and f0 seeds with np.concatenate, which a 0-d
array cannot take:
Rayleigh.fit(x) OK
Rayleigh.fit(x, lfp=True) ValueError: zero-dimensional arrays
Rayleigh.fit(x, zi=True) ValueError: zero-dimensional arrays
Now a one-tuple. Plain and offset fits are unchanged. A sweep of all
fourteen continuous distributions across both paths confirmed Rayleigh
was the only one affected. Three regression tests, verified to fail
with the source fix stashed.
That survey also showed the contract is not one shape: Weibull returns
a tuple, five return arrays, CustomDistribution returns a list, and
five more return a fitted model's .params. Unifying them is next.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
The initial-guess seed a distribution hands the optimiser came back in four different containers across the 21 implementations: a tuple in nine, a numpy array in six, a Python list in one, a fitted model's .params in five, and a bare scalar in Rayleigh. Two files disagreed with themselves -- exponential returned a tuple in its offset branch and an array in the other, rayleigh a tuple and a scalar. It worked because the one caller, _initial_guess, does np.array(init), which flattens tuple, list and array alike. It stopped working at the scalar, because np.array of a scalar is 0-dimensional rather than length-1 and the lfp and zi paths concatenate onto the seed -- fixed for Rayleigh in the previous commit, but only there. All 28 return statements now construct a 1-D float array explicitly, so the shape is decided where the values are known rather than inferred downstream, and a 0-dimensional seed is no longer expressible. The three declared types that disagreed are updated too: weibull said tuple[float, ...], custom_distribution said list[float], discretize said npt.NDArray | tuple[float, ...]. With the base's TYPE_CHECKING stub that is one declaration across the package. No seed changed. All 38 -- every distribution with an initialiser, in both plain and offset form -- were captured before and compared after, and every value is identical. np.array(init) at the call site is now a no-op copy and is left alone. The layout is unchanged and remains flat rather than nested: [gamma] when an offset is requested, then the k distribution parameters, then [p] for a limited failure population and [f0] for zero inflation, appended by the caller. The arity depends on both k and the structural flags, which is why the type is variadic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
Twenty signatures across normal.py, added to the enforced list --
twenty modules, twelve of the 25 distributions. One shape not seen
before: _closed_form_mle takes a SurpyvalData and returns the parameter
vector or None when the closed form does not apply to that data.
Writing the annotation the base's own docstring already claimed found a
bug. ParametricFitter documents the initialiser signature as
(self, x, c=None, n=None, t=None, offset=False), but Normal tested
`2 in c` and indexed `x[c != -1]`, and Gumbel tested
`(2 in c) or (-1 in c)`, before either defaulted c:
Normal._parameter_initialiser(x) TypeError: argument of type
Gumbel._parameter_initialiser(x) 'NoneType' is not iterable
mypy reported it as an unsupported operand for `in` against
`ndarray | None`. Every caller inside the package passes c and n, which
is why nothing caught it. GumbelLEV is unaffected because it forwards c
to fit without inspecting it -- the reason this looked like a
Gumbel-family problem and is not.
Gumbel is fixed here too. It is the same one-line defect found by the
same check, and leaving a known TypeError in the tree to keep the batch
to one file was the worse trade. A sweep of all nineteen distributions
with an initialiser found these two and no others.
Seven parametrized regression tests across the affected and unaffected
distributions, verified to fail with both source fixes stashed. All 38
initialiser seeds -- every distribution, plain and offset -- are
unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
_parameter_initialiser took (x, c=None, n=None, t=None, offset=False), and all 21 implementations opened by re-establishing conventions that had already been established one layer up -- inconsistently, and in two cases wrongly. Normal defaulted c and n; Gumbel guarded c with `is not None`; Beta tested `(c is not None) and (c == 0).all()`; Beta4 tested both c and n; LogLogistic ran a whole xcnt_handler round trip in its offset branch. Two of those guards were added only this release, to fix TypeErrors the documented call raised. None of it was needed. The one production caller, _initial_guess, is reached from fit_from_surpyval_data, which is handed a SurpyvalData -- whose entire purpose is to guarantee x, c, n and t are present, validated and in xcnt form -- and destructured it into loose arrays on the first line of its body. The signature is now (self, data: SurpyvalData, offset: bool = False). offset stays separate because it describes the requested model, not the data. _initial_guess and _fit_numerically take the object for the same reason. Seven defaulting checks go, along with 63 optional data parameters, and the initialisers that round-tripped their arrays back through fit (re-running xcnt_handler to rebuild the object the caller already held) now call fit_from_surpyval_data directly. _initial_guess rewraps its own working copy rather than forwarding the caller's object: the imputation rewrites x and c and the support masks drop rows, so the caller's object no longer describes it. That copy is deliberately untruncated -- imputing a left-censored point to a midpoint can put it at or before its own left-truncation bound, which xcnt_handler rejects outright (#260). It is what every initialiser has always received; it is now explicit rather than accidental. Breaking for anyone with their own distribution class. No shim: a bare array fails at the first attribute access rather than being half accepted, which is the property that would have caught the c=None divergence. All 38 seeds -- every distribution, plain and offset -- are identical before and after. Suite and both doctest passes green on 3.11, 3.12 and 3.13; mypy clean on all three. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
All 21 signatures annotated and the module added to the
disallow_untyped_defs list, bringing the ratchet to 21 modules and 13
of the 25 parametric distributions.
Two things the annotations made explicit.
moment is typed n: Numeric, where Normal's is n: int. Both docstrings
promise "integer or numpy array of integers". LogNormal's closed form
is vectorised and delivers that; Normal, GumbelLEV and LogLogistic
delegate to scipy.stats, which raises on an array of orders:
LogNormal.moment(np.array([1, 2]), 3., 4.) -> [5.99e+04, 3.19e+16]
Normal.moment(np.array([1, 2]), 3., 4.) -> ValueError
So the two annotations differ because the behaviours differ. The three
overpromising docstrings are left alone here; narrowing them is a
separate decision.
mpp_x_transform takes a gamma that no caller passes -- the MPP fitter
subtracts the offset from x before calling it. Only CustomDistribution
has the same spare argument; every other distribution's takes x alone.
Annotated and commented rather than removed, since it is part of the
published surface.
Suite and both doctest passes green on 3.11, 3.12 and 3.13; mypy clean
on all three. Ratchet probe confirms an unannotated def in the module
is now an error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
A distribution is reached through a ParametricFitter reference all over
the package -- fit_best iterates a list of them, Discretize and
MixtureModel wrap one, the regression fitters hold one as self.dist --
so code written against that reference has to work for every member.
Three shared methods disagreed about what their leading argument was
called, which made a keyword call correct for a subset and a TypeError
for the rest:
Weibull.qf(p=0.5, alpha=10, beta=2) worked
Poisson.qf(p=0.5, mu=3) TypeError
Poisson.qf(u=0.5, mu=3) worked
Weibull.moment(n=2, alpha=10, beta=2) worked
Poisson.moment(n=2, mu=3) TypeError
Same defect as the narrow from_params overrides fixed earlier this
release, applied to the rest of the surface.
qf's first argument is u in all 22 implementations (was p in 14, u in
7, q in Binomial). p cannot be the shared name: it is a real parameter
of Bernoulli, Binomial, Geometric and NegativeBinomial, and q is one of
DiscreteWeibull's -- which is why the two obvious names had been
avoided piecemeal to begin with.
moment's is m in all 21 (was n in 13); n is Binomial's trial count.
mpp_x_transform takes x alone in all 15. Eleven also took a gamma they
subtracted. No caller ever passed it -- the MPP fitter subtracts the
offset from x before calling (fitters/mpp.py) -- so a caller that did
would have subtracted twice. Removed rather than added to the other
four.
moment is typed m: int uniformly, and nine docstrings promising
"integer or numpy array of integers" are narrowed to "integer". Only 6
of 20 implementations accepted an array of orders; the rest raise,
because they delegate to scipy.stats.
Positional calls -- every docstring example, every call in the package,
every notebook -- are unaffected, and no keyword call to any of the
three exists in the package, tests or docs. No deprecation shim:
keeping the old name as an alias preserves the ambiguity the change
removes.
test_shared_signatures.py reads signatures rather than asserting a list
of names, so a distribution added later is covered without touching it.
An open-ended guard fails on any method implemented by five or more
distributions whose leading data argument disagrees; parameter names
are excluded, since Weibull.mean(alpha, beta) against Poisson.mean(mu)
is not a divergence but what the distributions are. Verified by
injecting a divergence: three tests fail, including the open guard.
Suite and both doctest passes green on 3.11, 3.12 and 3.13; mypy clean
on all three.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
Twelve distributions defined a conditional survival function. Eleven had
the same body as ParametricFitter.cs, differing only in spelling the
parameters out instead of taking *params:
return self.sf(x + X, alpha, beta) / self.sf(X, alpha, beta)
The duplication had already rotted. Gamma.cs documented
R(x) = e^{-lambda x}
which is the exponential survival function, copy-pasted from
exponential.py where both methods sat at line 136. The body computed the
ratio correctly, so the code was right and the maths above it described
a different distribution. Gamma is not memoryless.
Removes the eleven pass-throughs (395 lines) and gives
ParametricFitter.cs the docstring it never had -- until now cs was
undocumented anywhere an override was absent, which was all seven
discrete distributions and ExactEventTime. The wrong Gamma formula goes
with the override it lived on, and Gamma inherits the correct generic
statement.
Exponential.cs stays: memorylessness makes R(x, X) = R(x), one exp
instead of two and a division, and no cancellation far into the tail.
Ten of the removed docstrings carried doctested examples, and those were
the only per-distribution numerical check on cs. test_conditional_
survival.py keeps their values and adds the properties they only
implied: that each cs equals the survival ratio (which is what holds
Exponential's shortcut to the long way), that cs(0, X) is 1, that the
exponential is memoryless for any conditioning time, and that the
discrete distributions reach a working inherited cs.
A scan of every .. math:: block in the distributions for the same
copy-paste turned up no others -- Gamma's sf and log_df are correct, and
the remaining shared blocks are genuinely generic (h = f/R, H = -ln R)
or genuinely equal (Gumbel and GumbelLEV entropy, Logistic and Normal
mean).
Suite and both doctest passes green on 3.11, 3.12 and 3.13; mypy clean
on all three; docs build clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
Both bodies were a single `raise NotImplementedError`, and neither could ever run. Refusing probability plotting is declarative -- supports_mpp = False, checked in fit() before the fitter is dispatched -- and both distributions already set it, so the guard raised a ValueError naming the distribution and the alternatives three frames before the method was reachable. Deleting them changes no behaviour. Beta, Beta4, Gamma and ExpoWeibull all still refuse how="MPP" from parametric_fitter.py:814 with the same message. mpp is now defined only by Exponential and Rayleigh, which is where the hook means something: absence sends a distribution to the *generic* plotting path, so the method is an override for a closed form, never a way to decline. An earlier reading of this had it backwards. Two invariants keep the mechanisms from drifting together again: no distribution may declare supports_mpp = False and define mpp as well, and every refusal must come through the shared guard. The second covers nine distributions and is scoped to those whose fit() takes a `how` -- Bernoulli, Binomial and ExactEventTime override fit() with a narrow signature that has none, so asking them for MPP is a TypeError from argument binding. That is the separate fit() divergence, still open. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
Two bugs with the same root: ParametricFitter's default relations assume
a continuous distribution, and reach two kinds that are not.
Binomial.log_df returned the wrong mass. The base defines log_df as
log(hf) - Hf, which encodes f = h R(x). On the integers the mass at k is
P(T = k) = h(k) R(k - 1)
-- the hazard there times the survival to just *before* it. The two
differ by R(k)/R(k-1), which is not a rounding difference:
Binomial.log_df(3, 10, 0.3) -> -1.887 (was)
log(Binomial.df(3, 10, 0.3)) -> -1.321
Across k = 1..7 the returned mass ran from 0.88 of the truth down to
0.15. Five of the six discrete distributions override log_df with a
closed-form log-pmf and were unaffected; Binomial did not.
DiscreteParametricFitter now supplies the discrete relation, so Binomial
is correct and any future discrete distribution inherits the right one.
The class already documented the convention -- hf is P(T = k)/R(k - 1) --
it just had no log_df to match.
Latent, not live: Binomial.fit is analytic (p is the observed mean over
the trial count) and never evaluates a log-density, so no fit was
affected. But log_df is public and generic code calling it got the wrong
numbers.
ExactEventTime answered df and hf with inf. It is a point mass, so its
density is a Dirac delta: zero everywhere, infinite at one point,
integrating to one, and not a function of x. df returned inf at T and 0
elsewhere, which integrates to inf rather than 1; hf returned inf at T
and at every x after it; inherited log_df then computed log(inf) - inf
and returned nan. All three now raise NotImplementedError explaining why
and naming what is defined. An inf propagates into a plot, a likelihood
or a mixture weight and surfaces far from its cause; a raise stops at
the call site. Bernoulli already omitted df, hf and Hf for the same
reason.
ExactEventTime.Hf is kept, unchanged in value -- it is -log R(x),
stepping from 0 to infinity at T, which is well defined. It had been
written as an alias for hf, which happened to take the same two values;
it is now written as itself. sf, ff, qf, fitting and serialisation are
untouched, and the tests prove the last two still work.
Tests cover the mass identity for all six discrete distributions,
Binomial against scipy.stats.binom.logpmf, that the discrete hazard is a
probability (a continuous-convention hazard can exceed one, which is how
the mix-up shows itself), and that Hf accumulates as -sum log(1 - h)
rather than sum h -- the latter fails on every discrete distribution and
is the wrong test, which is worth pinning so it is not "fixed" later.
Suite and both doctest passes green on 3.11, 3.12 and 3.13; mypy clean
on all three; docs build clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
The only continuous distribution without a public moment, while already having mean and entropy. The exponentiated Weibull has a closed form -- an infinite series in C(mu-1, i)(-1)^i (i+1)^-(1 + m/beta) -- but it only terminates when mu is a positive integer. For other mu it is alternating and slow to converge, losing significance to cancellation as mu grows. The integral is quadrature either way, so moment takes it directly, as entropy already does and as mean already did. mean now delegates to moment(1) rather than repeating the integral. Checked against two references with no integration in them: at mu = 1 the distribution collapses to the Weibull, whose m-th moment is alpha^m Gamma(1 + m/beta) exactly; and for integer mu the series terminates and can be summed. Both agree to ~1e-14. The exponentiated-exponential case (alpha = beta = 1) is pinned against the harmonic number H_mu, which is its mean. ExpoWeibull also joins the moment comparison against quantile-bounded numerical integration in test_distributions_math.py, which had excluded it by name. Not circular despite both sides integrating: the reference integrates between quantiles with breakpoints, moment integrates from zero to infinity. No change to fitting. ParametricFitter._moment already had a quadrature fallback for distributions without a moment, so how="MOM" worked for ExpoWeibull before this and still does -- what was missing was the public method. Suite and both doctest passes green on 3.11, 3.12 and 3.13; mypy clean on all three; docs build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
It was not one. F(x) returned p at every x -- including x = -100 --
which is a flat curve with no time axis, not a coin flip. Meanwhile
moment, entropy, random and fit all described a genuine {0,1} variable:
E[X^m] = p, the binary entropy, draws of 0 and 1, and a fit that rejects
anything else. The class was two models at once, and df, hf, Hf and mean
were missing because they are the four places the contradiction cannot
be papered over.
Bernoulli is now the coin flip the name promises. x is the outcome, so 0
and 1 are the only values accepted:
Bernoulli.sf([0, 1], 0.3) -> array([1. , 0.3])
Bernoulli.df([0, 1], 0.3) -> array([0.7, 0.3])
Bernoulli.hf([0, 1], 0.3) -> array([0.7, 1. ])
Bernoulli.sf(37.5, 0.3) -> ValueError
R(x) = P(X >= x), so R(0) = 1 and R(1) = p: read as a one-shot device, p
is the probability it works when demanded. df, hf, Hf and mean are added
and every internal identity holds -- mass sums to one, h = f/R,
H = -ln R, and E[X] from the mass equals both mean and moment(1).
moment, entropy, random and fit are untouched, because they already
described this model.
BREAKING: p has changed direction. It was the probability of failure; it
is now the probability of the 1 outcome, which under the survival
reading is the probability of surviving. Code coding failures as 1 now
fits the survival probability and wants 1 - p.
log_df is defined on the class rather than inherited. Neither base
relation fits: DiscreteParametricFitter uses f(k) = h(k) R(k - 1), which
assumes R(k) = P(X > k), and here the at-risk set at x is R(x) itself.
The flat model is not gone. FixedEventProbability was a second instance
of the same class and is now its own, unchanged. It is the two-point
mixture of InstantlyOccurs (weight p) and NeverOccurs (weight 1 - p),
which is why degenerate.py already described those two as its limits at
p = 1 and p = 0. Its df, hf, qf and mean stay absent, correctly: a
constant F has no density, no invertible quantile and no time to
average. Both names serialise and round-trip under their own identities.
binomial.py claimed Bernoulli was "the special case n = 1". That was
false of the old model and is now true of the mass function: the two
df's agree exactly. The survival functions stay offset by one by
convention and the docstring now says so, as does the rewritten
test_reduces_to_bernoulli_at_n_one, which had encoded the old flat
behaviour.
Suite and both doctest passes green on 3.11, 3.12 and 3.13; mypy clean
on all three; docs build clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
Bernoulli.qf([0.1, 0.7, 0.75, 0.99], 0.3) -> array([0., 0., 1., 1.])
It inverts P(X <= x), the ordinary CDF, stepping from 0 to 1 at
u = 1 - p. On the open interval it matches Binomial.qf(u, 1, p) and
scipy.stats.binom.ppf exactly. At u = 0 those answer -1, one below the
support; this answers 0, the smallest outcome there is.
It is deliberately not the inverse of this class's ff, and that follows
from the survival convention rather than being an oversight.
R(x) = P(X >= x) forces F(x) = P(X < x) if the two are to sum to one,
and P(X < x) never exceeds 1 - p anywhere on {0, 1} -- so once u passes
1 - p no x in the support satisfies F(x) >= u. The other discrete
distributions, whose R(k) is P(X > k), do not have this split and the
usual ff(qf(u)) >= u check still holds for them. A test pins the
difference in both directions so it stays a known consequence rather
than becoming a surprise later.
What the definition does buy is the property worth having: qf(U) for
uniform U reproduces the distribution, which is how
ParametricFitter.random samples. Tested at 200,000 draws, and at the
degenerate ends p = 0 and p = 1.
Suite and both doctest passes green on 3.11, 3.12 and 3.13; mypy clean
on all three; docs build clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
Type-hint coverage moves from 611/1755 (35%) to 646/1760 (37%), #143. Nine modules were fully annotated but not listed under disallow_untyped_defs, so nothing stopped them slipping back: fit_best, utils.recurrent_utils, utils.score, recurrent.tests, recurrent.parametric.counting_process, regression.regression_data, regression.tvc_fit, regression.frailty and distributions.fixed_event_probability. Only counting_process needed work -- four *params that an AST scan counts as annotated and mypy does not. Eleven of the twelve accelerated-life modules follow, and locking them in turned up four real problems the annotations made visible: - GeneralLogLinear's constructor arguments were swapped: the bounds lambda sat in the phi_param_map slot and the param-map lambda in the phi_bounds slot. Nothing consumed either, so no observable effect, but it would have bitten whoever finished the model. That module stays out of the ratchet -- its phi_param_map and phi_bounds are callables of the covariate dimension rather than the dict and tuple LifeModel declares, which is why it is already excluded from LIFE_MODELS (#345). - LifeModel.phi_bounds was annotated as a one-element tuple while every caller passes two or three. Now variadic. - Two dead branches around phi_init. The fitter chose between a "(Z)"-only signature selected by comparing str(inspect.signature(...)), the two-argument form, and a non-callable phi_init. All ten life models are callable with (life, Z), so only one branch could run. - AcceleratedLife deserialisation accepted a distribution it cannot fit. The guard established a ParametricFitter, which admits Bernoulli, Binomial and ExactEventTime. The dict is untrusted input, so such a name got through and failed deep inside the fitter on a missing attribute; it now raises where the mistake is. hf is also declared in OptimisedFitMixin's TYPE_CHECKING block, where sf, ff, df, Hf and qf already were. Its absence was invisible until a typed caller reached for it. Behaviour is unchanged: the accelerated-life fit, prediction, random and serialisation round-trip are bit-identical before and after. Suite and both doctest passes green on 3.11, 3.12 and 3.13; mypy clean on all three; docs build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
Logistic was the only distribution with a public mgf, which read as a
method the other twenty-two were missing.
It is not an orphan and is not removed: Logistic.moment differentiates
it m times with autograd to get the m-th raw moment, and the results are
exact --
Logistic(mu=3, sigma=2)
moment(1) = 3.0000000000 exact mu
moment(2) = 22.1594725348 exact mu^2 + s^2 pi^2/3
moment(3) = 145.4352528131 exact mu^3 + 3 mu s^2 pi^2/3
The general closed form for a logistic raw moment needs Bernoulli
numbers, so differentiating the MGF is both shorter and exact. What was
wrong was its visibility: it is machinery for moment, not part of the
distribution surface. It is _mgf now, alongside the other private
helpers on distributions (_closed_form_mle, _moment_estimate,
_gumbel_seed). Nothing outside the class ever referenced it.
The module is fully annotated and added to the ratchet (#143), which
moves to 665/1760 (38%). Two annotations had to follow the code rather
than the other way round: mpp_y_transform indexes y, so it takes an
npt.NDArray rather than a Numeric that includes float, and unpack_rr
returns a tuple of two values, not an array -- both matching how Weibull
already declares them.
New tests pin the three low-order Logistic moments against the algebra
rather than another numerical method, check the variance comes out as
sigma^2 pi^2 / 3, and assert no distribution exposes a public mgf.
Suite and both doctest passes green on 3.11, 3.12 and 3.13; mypy clean
on all three; docs build clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
Coverage moves from 665/1760 (38%) to 869/1760 (49%), #143. Every distribution module is now under disallow_untyped_defs. rayleigh, beta, beta4, gamma, gumbel, gumbel_lev, loglogistic, exponential, uniform, degenerate and expo_weibull -- 202 signatures. The bulk was mechanical, generated from each distribution's own param_names so x is a Numeric, a parameter is a Boxable and the return follows the method. What was not mechanical were the places the generated guess was wrong, and each is a small fact about the code: - Rayleigh.mpp and Exponential.mpp treat the output of mpp_y_transform as an array -- indexing it, and handing it to np.polyfit and np.linalg.lstsq -- while the transform is declared to return a Boxable. Wrapped at the call site rather than widening the shared transform. - Gamma._moment_estimate and the two _mom helpers return 2-tuples, not arrays. - Exponential._closed_form_mle and Uniform._closed_form_mle return None when the closed form does not apply, so they are npt.NDArray | None. - ExpoWeibull.unpack_rr returns three values where every other distribution's returns two. - degenerate's classes inherit Distribution, not ParametricFitter, so their signatures have to match that supertype instead. - ExpoWeibull._gumbel_seed reads gumb.res, which a Parametric only carries after an MLE fit. The branch reading it is the one that asked for MLE, so it is annotated as deliberate rather than made unconditional. Behaviour is unchanged, and checked rather than assumed: all eleven distributions fitted by MLE, MPP, MSE and MOM, plus entropy and the second moment, before and after. All 66 results bit-identical. Suite and both doctest passes green on 3.11, 3.12 and 3.13; mypy clean on all three; docs build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
Coverage moves from 869/1760 (49%) to 995/1771 (56%), #143. Every module in the package -- the fitters, the model, the base class and the mixture -- is now under disallow_untyped_defs. Two structural additions came out of it, both of the same kind. A TYPE_CHECKING block on ParametricFitter declares the distribution functions its own methods call: cs divides two sfs, log_sf negates Hf, random inverts qf, and the four ll_* methods are written in terms of hf, Hf and the log densities. The class docstring already stated that contract in prose ("a distribution needs only hf and Hf, or sf, ff and df"); this is the same statement in a form the checker reads, mirroring the block OptimisedFitMixin already carried for the estimation machinery. Declared rather than defined, so a distribution that forgets one still gets the AttributeError naming it instead of a silently wrong inherited implementation. MixtureModel's fitted state -- data, params, w, p, loglike -- is annotated where it is initialised to None. Three annotations had to follow the code rather than the reverse: probability_plot_data's ff is the failure *function*, not an array; bounds_convert returns five things, not three; fallback_minimize's jac and hess are declared optional but supplied by every caller. Where a value comes back from scipy or autograd and has no narrower type -- the confidence-bound closures, the mixture's prediction inputs -- it is Any rather than npt.ArrayLike. That is the trap the Numeric/Boxable comment in parametric_fitter already documents: ArrayLike admits str and bytes, so arithmetic on it does not check, and the np.asarray that clears the error destroys an autograd box. Behaviour is unchanged and was checked rather than assumed: four distributions fitted plain, right- and left-censored, interval censored, truncated, with a limited-failure population and with zero inflation, plus neg_ll, aic and a two-component mixture fit -- bit-identical before and after. Suite and both doctest passes green on 3.11, 3.12 and 3.13; mypy clean on all three; docs build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
With every distribution annotated, the annotations can be read as data and compared. Ten argument slots and thirteen returns disagreed across the twenty-two modules -- drift from typing them a batch at a time. Unified the cosmetic ones. The three mpp_* transforms take an npt.NDArray: every call site passes one, eight of fifteen implementations index their argument, and probability plotting is a regression on plotting positions that is never differentiated, so the input is never an autograd box and never a scalar. Their returns stay Boxable because the bodies delegate to qf; narrowing them would mean changing code to suit a hint. random returns npt.NDArray everywhere -- Geometric and DiscreteWeibull returned self.qf(...) straight through and now wrap it, honest for the same reason in reverse. _mom is tuple[float, float] throughout. One difference was an error rather than an inconsistency. Numeric and Boxable both exclude list, yet fit and from_params accept lists on every distribution, as their own docstring examples show (Binomial.from_params([5, 0.3])). Those four slots are now npt.ArrayLike, correct here precisely because the value is converted with np.asarray on the first line rather than used in arithmetic. Eight differences remain and each is deliberate: ExactEventTime's step functions return the narrower npt.NDArray, a stronger promise rather than a broken one, and ExpoWeibull.unpack_rr returns three values. Five guard tests added so none of this can return. No behaviour changed; the two np.asarray wraps were checked to give identical samples. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
The previous sweep compared annotations; this one compares what the distributions compute. Every identity that should hold for all of them was evaluated across all twenty-three, and the disagreements chased down. Six discrete distributions returned nonsense below their support. The closed forms are algebraic and did not know where the support started, so one step below it gave Geometric.df(0) == 0.43, a positive probability outside the distribution; BetaGeometric.sf(-1) == 2.0, a survival above one that hf divided by; DiscreteWeibull.df(0) == 0.0355+0.5468j, a complex number from a negative base to a fractional power; and NaN from the incomplete gamma and beta forms in Poisson and NegativeBinomial. The pmf now sums to one whether or not the sum starts below the support; it did not for three of them. The guards clamp the input, not just the result, so the discarded branch never evaluates the invalid expression. Three quantile functions did not invert their own CDF, answering k + 1 for a u that came straight out of their own ff: F(k) = 1 - R(k) is formed by cancellation, so recovering k lands a few ulp above the integer and ceil rounds away from it. BetaGeometric.moment reported finite values for moments that do not exist. The survival decays as k^-a, so E[T^m] converges only for a > m, the condition mean already applied at m = 1. At a = 2, b = 3 it returned about 25 for a second moment that is infinite. Two distributions were missing methods that are well defined. FixedEventProbability had no Hf, so log_sf and log_ff -- which the base class writes in terms of it -- raised AttributeError. ExactEventTime gained qf, mean and moment: a point mass has no density, but its quantile is T for every u. Behaviour on the support is unchanged and was checked rather than assumed: 58 fingerprints, covering every function over its support for all six discrete distributions plus each one's fitted parameters and neg_ll fitted plain and right-censored, are bit-identical. The only intended change is BetaGeometric.moment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
``support`` is a pair of exclusive bounds: ``_validate_fit_inputs`` rejects data with ``x <= support[0]`` or ``x >= support[1]``, so a distribution declares them one step outside its first and last mass points. That is why Poisson declares -1 and Geometric declares 0. Binomial had Geometric's lower bound with Poisson's first mass point. Its 0 said that zero events in n trials lies outside the distribution, when the probability of that outcome is 0.168 at n = 5, p = 0.3. ``fit`` and ``from_params`` set [0, n], excluding n events as well. Both are now (-1, n + 1). Nothing had observed this. The check lives on OptimisedFitMixin, which Binomial does not inherit -- it is one of the three closed-form distributions that validate their own inputs -- so the field was inert metadata that would have become live the moment anything else read it. Every value is unchanged, which was checked rather than assumed: 18 fingerprints across both constructors, covering sf, ff, df, hf, Hf, qf, mean, neg_ll and the fitted parameters from four below the support to four above it, are bit-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TRhKL2fJBiAAfNo9rihwts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nineteen commits that had accumulated on the branch since #339. The through-line is the type-hint ratchet (#143), but most of the value is in the defects annotating the code turned up: reading the distributions closely enough to type them exposed several that were computing the wrong thing.
The ratchet
univariate.parametricis finished — every module in the package (the distributions, the fitters, the model, the base class, the mixture) is now underdisallow_untyped_defs. Coverage moves 611/1755 (35%) → 995/1771 (56%).Two structural additions came out of it, both of the same kind: a
TYPE_CHECKINGblock onParametricFitterdeclaring the distribution functions its own methods call (csdivides twosfs,log_sfnegatesHf, the fourll_*methods are written in terms ofhf/Hf/the log densities). The class docstring already stated that contract in prose; this is the same statement in a form the checker reads. Declared rather than defined, so a distribution that forgets one still gets theAttributeErrornaming it rather than a silently wrong inherited implementation.Remaining blocks, unchanged by this PR:
copula(77 defs),recurrent/renewal(68),proportional_hazards(52),degradation/path_models.py(45),utils/__init__.py(42), andgeneral_log_linear(#345).Defects fixed
Six discrete distributions returned nonsense below their support. Their closed forms are algebraic and did not know where the support started.
Geometric.df(0)was0.43— a positive probability outside the distribution, growing without bound askdecreases.BetaGeometric.sf(-1)was2.0, a survival above one thathfdivided by.DiscreteWeibull.df(0)was0.0355+0.5468j, a complex number from a negative base to a fractional power. Poisson and NegativeBinomial returned NaN from the incomplete gamma and beta forms. The pmf now sums to one whether or not the sum starts below the support; it did not for three of them before.The fitter's interior check kept these out of a likelihood, which is why nothing failed — but
dfandsfare public, so anyone plotting a pmf from zero got them. Each is now guarded at the first mass point, clamping the input rather than just the result, so the discarded branch of thenp.wherenever evaluates the invalid expression.Three quantile functions did not invert their own CDF.
Geometric,DiscreteWeibullandBetaGeometricansweredk + 1for authat came straight out of their ownff.F(k) = 1 - R(k)is formed by cancellation, so recoveringklands a few ulp above the integer andceilrounds away from it.BetaGeometric.momentreported finite values for moments that do not exist. The survival decays ask^-a, soE[T^m]converges only fora > m— the conditionmeanalready applied atm = 1. A truncated sum cannot see divergence; ata = 2, b = 3it returned about 25 for a second moment that is infinite.Binomial's support excluded two of its own outcomes.supportis a pair of exclusive bounds, so a distribution declares them one step outside its first and last mass points.BinomialhadGeometric's lower bound withPoisson's first mass point — saying zero events in n trials lies outside the distribution when its probability is 0.168 at n = 5, p = 0.3. Now(-1, n + 1). This was inert rather than live: the check lives onOptimisedFitMixin, whichBinomialdoes not inherit.Two distributions were missing methods that are well defined.
FixedEventProbabilityhad noHf, solog_sfandlog_ffraisedAttributeError.ExactEventTimegainedqf,meanandmoment.Also: continuous identities were being applied to non-continuous distributions;
Rayleigh's seed was wrong;Normaldid not honour its documentedcdefault;ExpoWeibullhad nomoment.API-visible changes
Bernoulliis now actually a Bernoulli; the flat model it had been conflated with is split off asFixedEventProbability, andBernoulligained aqf.Logistic.mgf→_mgf. It was the only distribution with a publicmgf, which read as a method the other twenty-two were missing. It is machinery formoment, not distribution surface; nothing outside the class referenced it.Beta.mppandBeta4.mppremoved as unreachable.csis inherited rather than restated on every distribution._parameter_initialiser) requires aSurpyvalDataand has one return shape across all distributions.Verification
Behaviour was checked rather than assumed at each step, by fingerprint diffs across the working tree before and after:
entropyand second moment) — bit-identicalneg_ll,aicand a two-component mixture — bit-identicalBetaGeometric.momentBinomialsupport change — bit-identicalSuite green on this commit in a clean checkout: 2204 passed, 364 skipped, 5 xfailed. At commit time each change also passed the full matrix — suite plus two doctest passes on 3.11/3.12/3.13 — with mypy clean on 308 files, flake8/black/isort clean, and the sphinx build succeeding.
The changelog carries the full detail under
v0.19.1 (unreleased). Worth noting the version should probably be retitled 0.20.0 before release, given theBernoullisplit and themgf/mppremovals.🤖 Generated with Claude Code
Generated by Claude Code