diff --git a/docs/changelog.rst b/docs/changelog.rst index 1b72bf5..30aaa89 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -4,6 +4,605 @@ Changelog v0.19.1 (unreleased) -------------------- +- **A behavioural consistency sweep across the base distributions.** The + previous sweep compared annotations; this one compares what the + distributions actually compute. Every identity that should hold for all + of them -- ``sf + ff == 1``, ``Hf == -ln sf``, ``log_df == ln df``, + ``hf == df/R(k-1)``, ``qf(ff(x)) == x``, ``mean == moment(1)`` -- was + evaluated across all twenty-three, and the disagreements chased down. + + **Six discrete distributions returned nonsense below their support.** + Geometric, DiscreteWeibull, BetaGeometric and NegativeBinomial live on + :math:`\{1, 2, 3, \dots\}`; Poisson and Binomial on + :math:`\{0, 1, 2, \dots\}`. Their closed forms are algebraic and did not + know where the support started, so evaluating one step below it gave + ``Geometric.df(0) == 0.43`` -- a positive probability outside the + distribution, growing without bound as ``k`` decreases -- + ``BetaGeometric.sf(-1) == 2.0``, a survival above one that ``hf`` + divided by, ``DiscreteWeibull.df(0) == 0.0355+0.5468j``, a *complex + number* from a negative base to a fractional power, and NaN from the + incomplete gamma and beta forms in Poisson and NegativeBinomial. The + pmf now sums to one whether or not the sum starts below the support; + it did not for three of them before. + + The fitter's interior check kept these values out of a likelihood, + which is why nothing failed, but ``df`` and ``sf`` are public: anyone + plotting a pmf from zero got them. Each is now guarded at the first + mass point. The guards clamp the *input*, not just the result, so the + discarded branch of the ``np.where`` never evaluates the invalid + expression -- otherwise it still computes the NaN and warns before + throwing it away. + + **Three quantile functions did not invert their own CDF.** + ``Geometric``, ``DiscreteWeibull`` and ``BetaGeometric`` answered + ``k + 1`` for a ``u`` that came straight out of their own ``ff``. + :math:`F(k) = 1 - R(k)` is formed by cancellation, so recovering ``k`` + from it lands a few ulp above the integer and ``ceil`` rounds away from + it. The first two snap a near-integer before the ceiling; the third + compares with a relative slack in its bisection. + + **``BetaGeometric.moment`` reported finite values for moments that do + not exist.** The survival decays as :math:`k^{-a}`, so + :math:`E[T^m]` converges only for :math:`a > m` -- the condition + ``mean`` already applied at :math:`m = 1`. A truncated sum cannot see + divergence; at ``a = 2, b = 3`` it returned about 25 for a second + moment that is infinite. It now returns ``inf``, and ``moment(1)`` + uses the closed form, so it agrees with ``mean`` exactly rather than + to three decimal places. + + **Two distributions were missing methods that are well defined.** + ``FixedEventProbability`` had no ``Hf``, so ``log_sf`` and ``log_ff`` + -- which the base class writes in terms of it -- raised + ``AttributeError`` instead of returning constants. Its ``df``, ``hf``, + ``qf`` and ``mean`` remain absent deliberately: ``F`` is flat, so the + mass is an atom rather than a density. ``Hf`` is the exception, + exactly as for :class:`ExactEventTime`, whose ``Hf`` exists while its + ``hf`` does not. ``ExactEventTime`` itself gained ``qf``, ``mean`` and + ``moment``: a point mass has no density, but its quantile is ``T`` for + every ``u``, its mean is ``T`` and its m-th moment is ``T**m``. + + **Binomial's support excluded two of its own outcomes.** ``support`` is + a pair of *exclusive* bounds -- ``_validate_fit_inputs`` rejects + ``x <= support[0]`` and ``x >= support[1]`` -- so a distribution + declares them one step outside its first and last mass points, which is + why ``Poisson`` declares ``-1`` and ``Geometric`` declares ``0``. + ``Binomial`` had ``Geometric``'s lower bound with ``Poisson``'s first + mass point: ``0``, saying that zero events in n trials lies outside the + distribution when its probability is 0.168 at n = 5, p = 0.3. ``fit`` + and ``from_params`` set ``[0, n]``, excluding n events as well. The + bounds are now ``(-1, n + 1)``. + + Nothing had observed this: the check lives on ``OptimisedFitMixin``, + which ``Binomial`` does not inherit -- it is one of the three + closed-form distributions that validate their own inputs -- so the + field was inert metadata that would have become live the moment + anything else read it. All of its values are unchanged, which was + checked: 18 fingerprints across both constructors are bit-identical. + + Behaviour *on* the support is unchanged and was checked rather than + assumed: 58 fingerprints -- every function over its support for all six + discrete distributions, plus each one's fitted parameters and + ``neg_ll`` fitted plain and right-censored -- are bit-identical before + and after. The only intended change is ``BetaGeometric.moment``. Nine + new tests -- 37 cases once parametrised across the distributions -- + cover the below-support behaviour, the pmf total, the quantile round + trip, the divergence rule and the support bounds. + +- **A consistency sweep across the base distributions.** With every + distribution now annotated, the annotations themselves could be read + as data and compared. Ten argument slots and thirteen returns + disagreed across the twenty-two modules -- drift from having typed + them a batch at a time rather than a deliberate difference. + + Most of it was cosmetic and is now uniform. The three ``mpp_*`` + transforms take an ``npt.NDArray``: every call site in the package + passes one, eight of the fifteen implementations index their + argument, and probability plotting is a least-squares regression on + plotting positions that is never differentiated, so the input is + never an autograd box and never a scalar. Their returns stay + ``Boxable``, because the bodies delegate to ``qf``; narrowing them + would mean changing code to suit a type hint, which is the wrong way + round. ``random`` returns an ``npt.NDArray`` everywhere -- + ``Geometric`` and ``DiscreteWeibull`` returned ``self.qf(...)`` + straight through, and now wrap it, which is honest for the same + reason in reverse: ``qf`` is ``Boxable`` because a fit differentiates + it, and sampling never does. ``_mom`` is ``tuple[float, float]`` + throughout. + + One difference was a real error rather than an inconsistency. + ``Numeric`` and ``Boxable`` both exclude ``list``, and ``fit`` and + ``from_params`` were typed with them on four distributions -- yet + every one of those accepts a list, as their own docstring examples + show (``Binomial.from_params([5, 0.3])``). These are the entry points + a user reaches for with whatever data they have. They are now + ``npt.ArrayLike``, which is the correct type here precisely because + the value is converted with ``np.asarray`` on the first line rather + than used in arithmetic. ``Binomial.from_params`` already had it + right; ``Bernoulli``, ``FixedEventProbability`` and + ``ExactEventTime`` did not. + + Eight differences remain and each is deliberate: + ``ExactEventTime``'s ``sf``, ``ff``, ``df``, ``hf`` and ``Hf`` return + the narrower ``npt.NDArray``, which is a stronger promise rather than + a broken one -- they are step functions built with ``np.atleast_1d`` + and provably return a real array -- and ``ExpoWeibull.unpack_rr`` + returns three values where the two-parameter distributions return + two. + + Five tests were added to the shared-signature guard, so a future + distribution cannot reintroduce any of this: the distribution + functions take a ``Numeric`` and return a ``Boxable``, parameters are + ``Boxable``, the ``mpp_*`` family takes arrays, ``random`` returns + one, and the user entry points accept array-likes. Twenty-two tests + in that file now. No behaviour changed -- annotations are erased at + runtime, and the two ``np.asarray`` wraps were checked to produce + identical samples. + +- **Type-hint ratchet: ``univariate.parametric`` is finished.** Coverage + moves from 869/1760 (49%) to 995/1771 (56%), tracked in <#143>. Every + module in the package -- the fitters, the model, the base class and + the mixture -- is now under ``disallow_untyped_defs``. + + Two structural additions came out of it, both of the same kind. A + ``TYPE_CHECKING`` block on ``ParametricFitter`` now declares the + distribution functions its own methods call -- ``cs`` divides two + ``sf``\ s, ``log_sf`` negates ``Hf``, ``random`` inverts ``qf``, and + the four ``ll_*`` methods are written in terms of ``hf``, ``Hf`` and + the log densities. The class docstring already stated that contract in + prose ("a distribution needs only ``hf`` and ``Hf``, or ``sf``, ``ff`` + and ``df``"); this is the same statement in a form the checker reads, + and it mirrors the block ``OptimisedFitMixin`` already carried for the + estimation machinery. Declared rather than defined, so a distribution + that forgets one still gets the ``AttributeError`` that names it + instead of a silently wrong inherited implementation. + + ``MixtureModel``'s fitted state -- ``data``, ``params``, ``w``, ``p`` + and ``loglike`` -- is annotated where it is initialised to ``None``. + + Three annotations had to follow the code rather than the reverse, each + a small fact: ``probability_plot_data``'s ``ff`` is the failure + *function*, not an array of values; ``bounds_convert`` returns five + things, not three; and ``fallback_minimize``'s ``jac`` and ``hess`` are + declared optional but are supplied by every caller. + + Where a value comes back from scipy or autograd and genuinely has no + narrower type -- the confidence-bound closures, the mixture's + prediction inputs -- it is ``Any`` rather than ``npt.ArrayLike``. That + is the same trap the ``Numeric``/``Boxable`` comment in + ``parametric_fitter`` already documents: ``ArrayLike`` admits ``str`` + and ``bytes``, so arithmetic on it does not type check, and the + ``np.asarray`` that clears the error destroys an autograd box. + + Behaviour is unchanged and was checked rather than assumed: four + distributions fitted plain, right- and left-censored, interval + censored, truncated, with a limited-failure population and with zero + inflation, plus ``neg_ll``, ``aic`` and a two-component mixture fit -- + bit-identical before and after. + +- **Type-hint ratchet: the remaining eleven distributions.** Coverage + moves from 665/1760 (38%) to 869/1760 (49%), tracked in <#143>. Every + distribution module is now under ``disallow_untyped_defs`` except + ``general_log_linear``'s counterpart concerns (<#345>). + + ``rayleigh``, ``beta``, ``beta4``, ``gamma``, ``gumbel``, + ``gumbel_lev``, ``loglogistic``, ``exponential``, ``uniform``, + ``degenerate`` and ``expo_weibull`` -- 202 signatures. The bulk was + mechanical, generated from each distribution's own ``param_names`` so + that ``x`` is a ``Numeric``, a parameter is a ``Boxable`` and the + return follows the method. What was not mechanical were the places the + generated guess was wrong, and each of those is a small fact about the + code: + + - ``Rayleigh.mpp`` and ``Exponential.mpp`` treat the output of + ``mpp_y_transform`` as an array -- indexing it, and passing it to + ``np.polyfit`` and ``np.linalg.lstsq`` -- while the transform is + declared to return a ``Boxable``. Wrapped at the call site rather + than widening the transform, which is shared. + - ``Gamma._moment_estimate`` and the two ``_mom`` helpers return + 2-tuples, not arrays. + - ``Exponential._closed_form_mle`` and ``Uniform._closed_form_mle`` + return ``None`` when the closed form does not apply to the data, so + they are ``npt.NDArray | None``. + - ``ExpoWeibull.unpack_rr`` returns *three* values where every other + distribution's returns two. + - ``degenerate``'s classes inherit ``Distribution``, not + ``ParametricFitter``, and its signatures have to match that + supertype rather than the distribution convention. + - ``ExpoWeibull._gumbel_seed`` reads ``gumb.res``, which a + ``Parametric`` only carries after an MLE fit -- the branch that + reads it is the one that asked for MLE, so it is annotated as + deliberate rather than made unconditional. + + Behaviour is unchanged, and checked rather than assumed: every one of + the eleven distributions was fitted by MLE, MPP, MSE and MOM, and its + ``entropy`` and second moment evaluated, before and after. All 66 + results are bit-identical. + +- **``Logistic`` ratcheted, and ``mgf`` made private.** ``Logistic`` was + the only distribution with a public ``mgf``, which read as a method + the other twenty-two were missing. + + It is not an orphan and is not removed: ``Logistic.moment`` + differentiates it ``m`` times with autograd to get the m-th raw + moment, and the results are exact -- + + .. code-block:: text + + Logistic(mu=3, sigma=2) + moment(1) = 3.0000000000 exact mu = 3 + moment(2) = 22.1594725348 exact mu^2 + s^2 pi^2/3 + moment(3) = 145.4352528131 exact mu^3 + 3 mu s^2 pi^2/3 + + The general closed form for a logistic raw moment needs Bernoulli + numbers, so differentiating the MGF is both shorter and exact. What was + wrong was its visibility: it is machinery for ``moment``, not part of + the distribution surface. It is ``_mgf`` now, alongside the other + private helpers on distributions (``_closed_form_mle``, + ``_moment_estimate``, ``_gumbel_seed``). Nothing outside the class ever + referenced it. + + The module is now fully annotated and added to the ratchet (#143). + Two annotations had to follow the code rather than the other way + round: ``mpp_y_transform`` indexes ``y``, so it takes an + ``npt.NDArray`` rather than a ``Numeric`` that includes ``float``, and + ``unpack_rr`` returns a *tuple* of two values, not an array -- both + matching how ``Weibull`` already declares them. + + New tests pin the three low-order Logistic moments against the algebra + rather than against another numerical method, check the variance comes + out as :math:`\sigma^2\pi^2/3`, and assert that no distribution + exposes a public ``mgf``. + +- **Type-hint ratchet: the accelerated-life package, plus nine modules + that were already complete.** Coverage across the package moves from + 611/1755 (35%) to 646/1760 (37%), tracked in + <#143>. + + Nine modules were fully annotated but not listed under + ``disallow_untyped_defs``, so nothing stopped them slipping back. They + are listed now: ``fit_best``, ``utils.recurrent_utils``, + ``utils.score``, ``recurrent.tests``, + ``recurrent.parametric.counting_process``, + ``univariate.regression.regression_data``, + ``univariate.regression.tvc_fit``, ``univariate.regression.frailty`` + and ``distributions.fixed_event_probability``. Only + ``counting_process`` needed work -- four ``*params`` that an AST scan + counts as annotated and mypy does not. + + Eleven of the twelve accelerated-life modules follow, and locking them + in turned up four real problems that annotations made visible: + + - **``GeneralLogLinear``'s constructor arguments were swapped.** The + bounds lambda sat in the ``phi_param_map`` slot and the param-map + lambda in the ``phi_bounds`` slot. Nothing consumed either, so it had + no observable effect, but it would have bitten whoever finished the + model. That module stays out of the ratchet: its ``phi_param_map`` + and ``phi_bounds`` are callables of the covariate dimension rather + than the ``dict`` and ``tuple`` ``LifeModel`` declares, which is why + it is already excluded from ``LIFE_MODELS`` (<#345>). + + - **``LifeModel.phi_bounds`` was annotated as a one-element tuple** + while every caller passes two or three. Now variadic. + + - **Two dead branches around ``phi_init``.** The fitter chose between + three shapes -- a ``"(Z)"``-only signature selected by comparing + ``str(inspect.signature(...))``, the two-argument form, and a + non-callable ``phi_init``. All ten life models are callable with + ``(life, Z)``, so only one branch could ever run. + + - **``AcceleratedLife`` deserialisation accepted a distribution it + cannot fit.** The guard established a ``ParametricFitter``, which + admits ``Bernoulli``, ``Binomial`` and ``ExactEventTime`` -- none of + them fittable. Since the dict is untrusted input, such a name got + through and failed deep inside the fitter on a missing attribute; it + now raises where the mistake is. + + ``hf`` is also declared in ``OptimisedFitMixin``'s ``TYPE_CHECKING`` + block, where ``sf``, ``ff``, ``df``, ``Hf`` and ``qf`` already were. + Its absence was invisible until a typed caller reached for it. + + Behaviour is unchanged throughout: the accelerated-life fit, + prediction, ``random`` and serialisation round-trip all produce + bit-identical results before and after. + +- **``Bernoulli.qf``.** The quantile function, added after the rest of + the distribution:: + + Bernoulli.qf([0.1, 0.7, 0.75, 0.99], 0.3) -> array([0., 0., 1., 1.]) + + It inverts :math:`P(X \leq x)` -- the ordinary CDF -- stepping from 0 + to 1 at ``u = 1 - p``. On the open interval it matches + ``Binomial.qf(u, 1, p)`` and ``scipy.stats.binom.ppf`` exactly. At + ``u = 0`` those answer ``-1``, one below the support; this answers 0, + the smallest outcome there is. + + It is deliberately *not* the inverse of this class's ``ff``, and that + follows from the survival convention rather than being an oversight. + ``R(x) = P(X \geq x)`` forces ``F(x) = P(X < x)`` if the two are to + sum to one, and ``P(X < x)`` never exceeds ``1 - p`` anywhere on + ``{0, 1}`` -- so once ``u`` passes ``1 - p`` no ``x`` in the support + satisfies ``F(x) >= u``. The other discrete distributions, whose + ``R(k)`` is ``P(X > k)``, do not have this split, and the usual + ``ff(qf(u)) >= u`` check still holds for them. A test pins the + difference in both directions so it stays a known consequence rather + than becoming a surprise. + + What the definition does buy is the property worth having: ``qf(U)`` + for uniform ``U`` reproduces the distribution, which is how + ``ParametricFitter.random`` samples. Tested at 200,000 draws, and at + the degenerate ends ``p = 0`` and ``p = 1``. + +- **BREAKING: ``Bernoulli`` is now a Bernoulli distribution.** + It was not one. ``F(x)`` returned ``p`` at every ``x`` -- including + ``x = -100`` -- which is a flat curve with no time axis, not a coin + flip. Meanwhile ``moment``, ``entropy``, ``random`` and ``fit`` all + described a genuine ``{0, 1}`` variable: ``E[X^m] = p``, the binary + entropy, draws of 0 and 1, and a fit that rejects anything else. The + class was two models at once, and ``df``, ``hf``, ``Hf`` and ``mean`` + were missing because they are the four places the contradiction + cannot be papered over. + + ``Bernoulli`` is now the coin flip the name promises. ``x`` is the + outcome, so 0 and 1 are the only values accepted and anything else + raises:: + + Bernoulli.sf([0, 1], 0.3) -> array([1. , 0.3]) + Bernoulli.df([0, 1], 0.3) -> array([0.7, 0.3]) + Bernoulli.hf([0, 1], 0.3) -> array([0.7, 1. ]) + Bernoulli.sf(37.5, 0.3) -> ValueError + + The survival function is :math:`R(x) = P(X \geq x)`, so ``R(0) = 1`` + and ``R(1) = p``: read as a one-shot device, ``p`` is the probability + it works when demanded. ``df``, ``hf``, ``Hf`` and ``mean`` are added + and every internal identity now holds -- the mass sums to one, + ``h = f/R``, ``H = -ln R``, and ``E[X]`` from the mass equals both + ``mean`` and ``moment(1)``. ``moment``, ``entropy``, ``random`` and + ``fit`` are unchanged, because they already described this model. + + **``p`` has changed direction.** It was documented as the probability + of *failure*; it is now the probability of the ``1`` outcome, which + under the survival reading is the probability of *surviving*. Code + that coded failures as 1 now fits the survival probability and wants + ``1 - p``. + + ``log_df`` is defined on the class rather than inherited. Neither base + relation fits: ``DiscreteParametricFitter`` uses + ``f(k) = h(k) R(k - 1)``, which assumes ``R(k) = P(X > k)``, and here + the at-risk set at ``x`` is ``R(x)`` itself. + + **The flat model is not gone.** It survives unchanged as + :data:`FixedEventProbability`, which until now was a second instance + of the same class and is now its own. It is the two-point mixture of + ``InstantlyOccurs`` (weight ``p``) and ``NeverOccurs`` (weight + ``1 - p``) -- which is why ``degenerate.py`` already described those + two as its limits at ``p = 1`` and ``p = 0``. Its ``df``, ``hf``, + ``qf`` and ``mean`` remain absent, correctly: a constant ``F`` has no + density, no invertible quantile and no time to average. + + Both names serialise and round-trip under their own identities, so + stored models keep pointing at the model they were fitted with -- but + a stored ``Bernoulli`` fitted before 0.19.1 will now be read with the + new semantics, and its ``p`` reinterpreted as above. + + ``binomial.py`` claimed Bernoulli was "the special case ``n = 1``". + That was false of the old model and is now true of the mass function: + ``Bernoulli.df`` and ``Binomial.df(..., 1, p)`` agree exactly. The + survival functions remain offset by one by convention, and the + docstring now says so. + +- **``ExpoWeibull.moment``.** It was the only continuous distribution + without a public ``moment``, while already having ``mean`` and + ``entropy``. + + The exponentiated Weibull has a closed form -- an infinite series in + :math:`\binom{\mu-1}{i}(-1)^{i}(i+1)^{-(1+m/\beta)}` -- but it only + terminates when :math:`\mu` is a positive integer. For other + :math:`\mu` it is alternating and slow to converge, losing + significance to cancellation as :math:`\mu` grows. The integral is + quadrature either way, so ``moment`` takes it directly, as ``entropy`` + already does and as ``mean`` already did. ``mean`` now delegates to + ``moment(1)`` rather than repeating the integral. + + Checked against two references with no integration in them: at + :math:`\mu = 1` the distribution collapses to the Weibull, whose + m-th moment is :math:`\alpha^{m}\Gamma(1 + m/\beta)` exactly; and + for integer :math:`\mu` the series terminates and can be summed. Both + agree to about 1e-14. The exponentiated-exponential case + (:math:`\alpha = \beta = 1`) is also pinned against the harmonic + number :math:`H_{\mu}`, which is its mean. + + ``ExpoWeibull`` joins the ``moment`` comparison against quantile-bounded + numerical integration in ``test_distributions_math.py``, which had + excluded it by name. That check is not circular despite both sides + integrating: the reference integrates between quantiles with + breakpoints, ``moment`` integrates from zero to infinity. + + This does not change fitting. ``ParametricFitter._moment`` already had + a quadrature fallback for distributions without a ``moment``, so + ``how="MOM"`` worked for ``ExpoWeibull`` before this and still does. + What was missing was the public method. + +- **Fixed: ``Binomial.log_df`` returned the wrong mass, and + ``ExactEventTime`` answered ``df`` and ``hf`` with ``inf``.** + + Two consequences of continuous-distribution assumptions reaching + distributions that are not continuous. + + ``ParametricFitter.log_df`` is ``log(hf) - Hf``, which encodes the + continuous identity :math:`f = h R(x)`. On the integers the mass at + ``k`` is :math:`P(T = k) = h(k) R(k - 1)` -- the hazard there times + the survival to just *before* it. The two differ by a factor + :math:`R(k)/R(k-1)`, which is not a rounding difference:: + + Binomial.log_df(3, 10, 0.3) -> -1.887 (was) + log(Binomial.df(3, 10, 0.3)) -> -1.321 + + Across ``k = 1..7`` the returned mass ran from 0.88 of the truth down + to 0.15. Five of the six discrete distributions override ``log_df`` + with a closed-form log-pmf and were unaffected; Binomial did not, and + reached the continuous identity. ``DiscreteParametricFitter`` now + supplies the discrete relation, so Binomial is correct and any future + discrete distribution inherits the right one. The class already + documented the convention -- ``hf`` is ``P(T = k) / R(k - 1)`` -- it + simply had no ``log_df`` to match it. + + The bug was latent rather than live: ``Binomial.fit`` is analytic + (``p`` is the observed mean over the trial count) and never evaluates + a log-density, so no fit was affected. ``Binomial.log_df`` is public, + though, and generic code that calls it got the wrong numbers. + + Separately, ``ExactEventTime`` is a point mass, so its density is a + Dirac delta: zero everywhere, infinite at one point, integrating to + one. There is no function of ``x`` that represents it. ``df`` returned + ``inf`` at ``T`` and 0 elsewhere, which integrates to ``inf`` rather + than 1; ``hf`` returned ``inf`` at ``T`` *and at every x after it*; + and the inherited ``log_df`` computed ``log(inf) - inf`` and returned + ``nan``. All three now raise ``NotImplementedError`` explaining why + and pointing at the functions that are defined. An ``inf`` propagates + into a plot, a likelihood or a mixture weight and surfaces far from + its cause; a raise stops at the call site. ``Bernoulli`` already + omitted ``df``, ``hf`` and ``Hf`` for the same reason. + + ``ExactEventTime.Hf`` is kept and is unchanged in value -- it is + :math:`-\log R(x)`, stepping from 0 to infinity at ``T``, which is + well defined. It had been written as an alias for ``hf``, which + happened to take the same two values; it is now written as itself. + ``sf``, ``ff``, ``qf`` and fitting are untouched. + + New tests cover the discrete mass identity for all six distributions, + Binomial's log-pmf against scipy, that the discrete hazard is a + probability (a continuous-convention hazard can exceed one, which is + how the mix-up shows itself), that ``Hf`` accumulates as + :math:`-\sum \log(1 - h)` rather than :math:`\sum h`, and the + degenerate refusals alongside proof that fitting and serialisation + still work. + +- **``Beta.mpp`` and ``Beta4.mpp`` removed as unreachable.** + Both bodies were a single ``raise NotImplementedError``, and neither + could ever run. Refusing probability plotting is declarative -- + ``supports_mpp = False``, checked in ``fit`` before the fitter is + dispatched -- and both distributions already set it, so the guard + raised a ``ValueError`` naming the distribution and the alternatives + three frames before the method was reachable. + + Deleting them changes no behaviour. ``Beta``, ``Beta4``, ``Gamma`` and + ``ExpoWeibull`` all still refuse ``how="MPP"`` from the same guard, + with the same message. ``mpp`` is now defined only by ``Exponential`` + and ``Rayleigh``, which is where the hook means something: absence of + ``mpp`` sends a distribution to the *generic* plotting path, so the + method is an override for a closed form, never a way to decline. + + Two invariants in ``test_shared_signatures.py`` keep the two + mechanisms from drifting back together: no distribution may declare + ``supports_mpp = False`` and define ``mpp`` as well, and every + distribution that refuses must refuse through the shared guard rather + than an exception of its own. The second covers nine distributions and + is scoped to those whose ``fit`` takes a ``how`` at all -- ``Bernoulli``, + ``Binomial`` and ``ExactEventTime`` override ``fit`` with a narrow + signature that has none, so asking them for MPP is a ``TypeError`` + from argument binding. That is the separate ``fit`` divergence, still + open. + +- **``cs`` is inherited rather than restated on every distribution, + and Gamma's ``cs`` documentation no longer describes the exponential.** + Twelve distributions defined a conditional survival function. Eleven + of the twelve had the same body as ``ParametricFitter.cs``, differing + only in spelling the parameters out instead of taking ``*params``:: + + return self.sf(x + X, alpha, beta) / self.sf(X, alpha, beta) + + The duplication had already rotted. ``Gamma.cs`` carried + + .. math:: + R(x) = e^{-\lambda x} + + which is the *exponential* survival function -- copy-pasted from + ``exponential.py``, where both methods sat at line 136. The body + computed the ratio correctly, so the code was right and the + documentation above it described a different distribution. Gamma is + not memoryless and its conditional survival is not its survival. + + The eleven pass-through overrides are removed (395 lines), and + ``ParametricFitter.cs`` -- which had no docstring at all, so ``cs`` + was undocumented anywhere the override was absent -- now carries the + definition, the parameter descriptions and a worked example. The + wrong Gamma formula goes with the override it lived on, and Gamma + inherits the correct generic statement. + + ``Exponential.cs`` is kept. The exponential is memoryless, so + :math:`R(x, X) = R(x)`, which is one ``exp`` rather than two and a + division, and avoids the cancellation the ratio suffers far into the + tail. + + Ten of the removed docstrings carried doctested examples, and those + were the only per-distribution numerical check on ``cs``. Their values + are preserved in + ``surpyval/tests/univariate/parametric/test_conditional_survival.py``, + alongside tests that each distribution's ``cs`` equals the survival + ratio (which is what checks Exponential's shortcut against the long + way), that ``cs(0, X) == 1``, that the exponential is memoryless for + any conditioning time, and that the discrete distributions reach a + working inherited ``cs``. + +- **BREAKING: shared methods now have one signature across every + distribution.** + A distribution is reached through a ``ParametricFitter`` reference + all over the package -- ``fit_best`` iterates a list of them, + ``Discretize`` and ``MixtureModel`` wrap one, the regression fitters + hold one as ``self.dist`` -- so code written against that reference + has to work for every member. Three shared methods disagreed about + what their leading argument was called, which made a keyword call + correct for a subset and a ``TypeError`` for the rest:: + + Weibull.qf(p=0.5, alpha=10, beta=2) worked + Poisson.qf(p=0.5, mu=3) TypeError + Poisson.qf(u=0.5, mu=3) worked + Weibull.moment(n=2, alpha=10, beta=2) worked + Poisson.moment(n=2, mu=3) TypeError + + This is the defect that made the narrow ``from_params`` overrides on + ``Bernoulli``, ``Binomial`` and ``ExactEventTime`` worth fixing + earlier in this release, applied to the rest of the surface. + + - ``qf``'s first argument is ``u`` in all 22 implementations. It was + ``p`` in 14, ``u`` in 7 and ``q`` in ``Binomial``. ``p`` cannot be + the shared name because it is an actual parameter of ``Bernoulli``, + ``Binomial``, ``Geometric`` and ``NegativeBinomial``, and ``q`` is + one of ``DiscreteWeibull``'s -- which is why the two obvious + choices had been avoided piecemeal in the first place. + - ``moment``'s first argument is ``m`` in all 21. It was ``n`` in 13, + and ``n`` is ``Binomial``'s trial count. + - ``mpp_x_transform`` takes ``x`` alone in all 15. Eleven of them + also took a ``gamma`` they subtracted, and the other four did not. + No caller ever passed it: the MPP fitter subtracts the offset from + ``x`` before calling (``fitters/mpp.py``), so a caller that did + pass it would have subtracted the offset twice. Removed rather + than added to the other four. + + Positional calls -- which is what every docstring example, every call + inside the package, and every notebook uses -- are unaffected. No + keyword call to any of the three exists in the package, its tests or + its documentation. There is no deprecation shim: keeping the old name + as an alias would preserve exactly the ambiguity the change removes. + + ``moment`` is also now typed ``m: int`` uniformly, and nine + docstrings that promised "integer or numpy array of integers" are + narrowed to "integer". Only six of the twenty implementations + actually accepted an array of orders; the rest raised, because they + delegate to ``scipy.stats``:: + + LogNormal.moment(np.array([1, 2]), 3., 4.) -> [5.99e+04, 3.19e+16] + Normal.moment(np.array([1, 2]), 3., 4.) -> ValueError + + ``surpyval/tests/univariate/parametric/test_shared_signatures.py`` + reads the signatures rather than asserting a list of names, so a + distribution added later is covered without touching the test, and + an open-ended guard fails on *any* method implemented by five or + more distributions whose leading data argument disagrees. Parameter + names are excluded from that guard: ``Weibull.mean(alpha, beta)`` + against ``Poisson.mean(mu)`` is not a divergence, it is what the + distributions are. + - **API reference pages for the surfaces that only had narrative docs.** The multivariate copulas and the beta survival tree and forest had no autodoc coverage at all, and the degradation page stopped at the path @@ -108,11 +707,113 @@ v0.19.1 (unreleased) mypy cannot resolve them through that cycle. They name the concrete class instead. -- **Type-hint coverage is now enforced, for seventeen modules (#143).** +- **``Normal`` and ``Gumbel`` ignored their own documented default.** + ``ParametricFitter`` documents the initialiser signature as + ``(self, x, c=None, n=None, t=None, offset=False)``, but ``Normal`` + tested ``2 in c`` and indexed ``x[c != -1]``, and ``Gumbel`` tested + ``(2 in c) or (-1 in c)``, before either defaulted ``c``. Calling + either as documented raised ``TypeError: argument of type 'NoneType' + is not iterable``. Every caller inside the package passes ``c`` and + ``n``, which is why it went unnoticed; ``GumbelLEV`` is unaffected + because it forwards ``c`` to ``fit`` without inspecting it. A sweep of + all nineteen distributions found these two and no others. + +- **BREAKING: ``_parameter_initialiser`` takes a ``SurpyvalData``.** + The signature was ``(self, x, c=None, n=None, t=None, offset=False)``, + and every one of the 21 implementations spent its opening lines + re-establishing conventions that had already been established -- + inconsistently, and in some cases wrongly. ``Normal`` defaulted + ``c`` and ``n``; ``Gumbel`` guarded ``c`` with ``is not None``; + ``Beta`` tested ``(c is not None) and (c == 0).all()``; ``Beta4`` + tested both ``c`` and ``n``; ``LogLogistic`` ran a whole + ``xcnt_handler`` round trip in its offset branch. Two of those checks + were absent until this release and raised ``TypeError`` for the + documented call. + + None of it was ever needed. The one production caller, + ``_initial_guess``, is reached from ``fit_from_surpyval_data``, which + is *handed* a ``SurpyvalData`` -- an object whose entire purpose is to + guarantee that ``x``, ``c``, ``n`` and ``t`` are present, validated + and in xcnt form -- and destructured it into loose arrays on the first + line of its body. The convention was rebuilt three layers below the + object that had already established it. + + The signature is now ``(self, data: SurpyvalData, offset: bool = + False)``. ``offset`` stays a separate argument because it describes + the model being requested, not the data. ``_initial_guess`` and + ``_fit_numerically`` take the object rather than loose arrays for the + same reason. Seven defaulting checks are gone, along with 63 optional + data parameters (27 of them explicitly annotated ``| None``), and the + initialisers that used to + round-trip their arrays back through ``fit`` (re-running + ``xcnt_handler`` and rebuilding the object the caller already held) + now call ``fit_from_surpyval_data`` directly. + + ``t`` is not passed to the initialisers, and never was: no caller has + ever supplied it. ``_initial_guess`` imputes interval- and + left-censored points to midpoints before seeding, which can put an + observation at or before its own left-truncation bound -- data + ``xcnt_handler`` rejects outright (#260) -- so the working copy it + builds is deliberately untruncated. That is what every initialiser has + always received; it is now explicit rather than accidental. + + This is a breaking change for anyone who has written their own + distribution class. There is no shim: a bare array now fails at the + first attribute access rather than being half-accepted. Every one of + the 38 seeds -- each distribution, plain and offset -- is identical + before and after. + +- **Every ``_parameter_initialiser`` now returns the same thing.** + The initial-guess seed a distribution hands the optimiser came back in + four different containers across the 21 implementations: a tuple in + nine, a numpy array in six, a Python list in one, a fitted model's + ``.params`` in five -- and a bare scalar in ``Rayleigh``. Two files + disagreed with *themselves*: ``exponential`` returned a tuple in its + offset branch and an array in the other, ``rayleigh`` a tuple and a + scalar. + + It worked because the one caller, ``_initial_guess``, does + ``np.array(init)``, which flattens tuple, list and array alike. It + stopped working at the scalar, because ``np.array`` of a scalar is + 0-dimensional rather than length-1, and the ``lfp`` and ``zi`` paths + concatenate onto the seed. + + All 28 return statements now construct a 1-D float array explicitly, + so the shape is decided where the values are known rather than + inferred downstream, and a 0-dimensional seed is no longer + expressible. No seed changed: all 38 -- every distribution, plain and + offset -- were compared before and after and are identical. + + The seed itself is unchanged in layout, and it is flat rather than + nested: ``[gamma]`` when an offset is requested, then the ``k`` + distribution parameters, then ``[p]`` for a limited failure population + and ``[f0]`` for zero inflation, appended by the caller. The arity + therefore depends on both ``k`` and the structural flags. + +- **Limited-failure and zero-inflated Rayleigh models could not be fit.** + ``Rayleigh.fit(x, lfp=True)`` and ``Rayleigh.fit(x, zi=True)`` both + raised ``ValueError: zero-dimensional arrays cannot be concatenated``. + + Rayleigh is the only single-parameter distribution here, and its + ``_parameter_initialiser`` returned the sigma seed as a bare scalar + rather than a sequence. ``np.array(init)`` in ``_initial_guess`` then + produced a 0-dimensional array instead of a length-1 one, and the + ``lfp`` and ``zi`` paths append their ``p`` and ``f0`` seeds with + ``np.concatenate``, which a 0-d array cannot take. The seed is now a + one-tuple. Plain and offset fits are unchanged. + + Found by surveying every ``_parameter_initialiser`` in the library + after the type-hint work turned up three different return shapes; a + sweep of all fourteen continuous distributions across both paths + confirmed Rayleigh was the only one affected. + +- **Type-hint coverage is now enforced, for twenty-one modules (#143).** ``surpyval.distribution``, ``surpyval.serialisation``, ``surpyval.metrics``, ``surpyval.univariate.information_criteria``, - ``surpyval.datasets``, the Weibull and the eight discrete - distributions, and all of ``surpyval.univariate.nonparametric``, + ``surpyval.datasets``, the Weibull, the Normal, the LogNormal, the + eight discrete distributions, ``CustomDistribution`` and + ``ExactEventTime``, and + all of ``surpyval.univariate.nonparametric``, ``surpyval.recurrent.nonparametric`` and ``surpyval.univariate.regression.frailty`` have ``disallow_untyped_defs`` set in ``pyproject.toml``, so an @@ -120,9 +821,37 @@ v0.19.1 (unreleased) abstract base classes every model inherits from, the Kaplan-Meier, Nelson-Aalen, Fleming-Harrington and Turnbull estimators, the log-rank test, the plotting positions, the non-parametric MCF, the - shared-frailty fitter, the bundled datasets and nine of the 25 + shared-frailty fitter, the bundled datasets and thirteen of the 25 parametric distributions. + ``LogNormal.moment`` is annotated ``n: Numeric`` where ``Normal``'s + is ``n: int``, and the difference is real rather than an oversight. + Both docstrings promise "integer or numpy array of integers". + LogNormal's closed form is vectorised and delivers that; + ``Normal``, ``GumbelLEV`` and ``LogLogistic`` delegate to + ``scipy.stats``, which raises ``ValueError: The truth value of an + array ... is ambiguous`` on an array of orders. The annotations now + say which is which; the three docstrings that overpromise are not + yet corrected. + + ``CustomDistribution`` needed restructuring rather than only + annotating. It assigned its distribution functions onto the + instance -- ``self.Hf = fun``, then lambdas for ``hf``, ``sf``, + ``ff`` and ``df`` -- which stopped being possible once + ``OptimisedFitMixin`` declared those names for its own use, because + a subclass inherits the declarations and assigning to an inherited + method is an error. The function is stored as ``_fun`` and the five + are real methods delegating to it. Equivalent by construction: the + old ``self.Hf = fun`` was an unbound instance attribute, so + ``self.Hf(x, *params)`` called ``fun(x, *params)`` either way. The + autograd-derived ``hf`` and ``df`` were checked numerically against + the previous implementation, gradients included. + + Its ``_parameter_initialiser`` returns a *list*, where Weibull + returns a tuple and the discrete distributions return an array -- + three shapes for one contract the base never pinned down. Noted in + the signatures rather than unified, since every caller coerces. + ``handle_xicn`` gained ``@overload`` declarations as part of this. Its return shape is decided by ``as_recurrent_data``, but its signature only said "one or the other", so all nine callers taking diff --git a/pyproject.toml b/pyproject.toml index 31204c3..1ad0a67 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -137,6 +137,60 @@ module = [ 'surpyval.univariate.parametric.distributions.geometric', 'surpyval.univariate.parametric.distributions.negative_binomial', 'surpyval.univariate.parametric.distributions.poisson', + 'surpyval.univariate.parametric.distributions.custom_distribution', + 'surpyval.univariate.parametric.distributions.exact_event_time', + 'surpyval.univariate.parametric.distributions.normal', + 'surpyval.univariate.parametric.distributions.lognormal', + 'surpyval.univariate.parametric.distributions.logistic', + 'surpyval.univariate.parametric.distributions.rayleigh', + 'surpyval.univariate.parametric.distributions.beta', + 'surpyval.univariate.parametric.distributions.beta4', + 'surpyval.univariate.parametric.distributions.gamma', + 'surpyval.univariate.parametric.distributions.gumbel', + 'surpyval.univariate.parametric.distributions.gumbel_lev', + 'surpyval.univariate.parametric.distributions.loglogistic', + 'surpyval.univariate.parametric.distributions.exponential', + 'surpyval.univariate.parametric.distributions.uniform', + 'surpyval.univariate.parametric.distributions.degenerate', + 'surpyval.univariate.parametric.distributions.expo_weibull', + 'surpyval.univariate.parametric.fitters', + 'surpyval.univariate.parametric.fitters.closed_form', + 'surpyval.univariate.parametric.fitters.mle', + 'surpyval.univariate.parametric.fitters.mpp', + 'surpyval.univariate.parametric.fitters.mom', + 'surpyval.univariate.parametric.fitters.mps', + 'surpyval.univariate.parametric.fitters.mse', + 'surpyval.univariate.parametric.probability_plotting', + 'surpyval.univariate.parametric.discrete_fitter', + 'surpyval.univariate.parametric.royston_parmar', + 'surpyval.univariate.parametric.parametric_fitter', + 'surpyval.univariate.parametric.parametric', + 'surpyval.univariate.parametric.mixture_model', + 'surpyval.univariate.parametric.distributions.fixed_event_probability', + # Already fully annotated; listed so they cannot slip back. + 'surpyval.fit_best', + 'surpyval.utils.recurrent_utils', + 'surpyval.utils.score', + 'surpyval.recurrent.tests', + 'surpyval.recurrent.parametric.counting_process', + 'surpyval.univariate.regression.regression_data', + 'surpyval.univariate.regression.tvc_fit', + 'surpyval.univariate.regression.frailty', + # Every accelerated-life module except general_log_linear, whose + # phi_param_map and phi_bounds are callables of the covariate + # dimension rather than the dict and tuple LifeModel declares. It is + # excluded from LIFE_MODELS for the same reason (#345). + 'surpyval.univariate.regression.accelerated_life', + 'surpyval.univariate.regression.accelerated_life.accelerated_life', + 'surpyval.univariate.regression.accelerated_life.dual_exponential', + 'surpyval.univariate.regression.accelerated_life.dual_power', + 'surpyval.univariate.regression.accelerated_life.exponential', + 'surpyval.univariate.regression.accelerated_life.eyring', + 'surpyval.univariate.regression.accelerated_life.lifemodel', + 'surpyval.univariate.regression.accelerated_life.linear', + 'surpyval.univariate.regression.accelerated_life.parameter_substitution', + 'surpyval.univariate.regression.accelerated_life.power', + 'surpyval.univariate.regression.accelerated_life.power_exponential', ] disallow_untyped_defs = true diff --git a/surpyval/recurrent/parametric/counting_process.py b/surpyval/recurrent/parametric/counting_process.py index 33a0a2c..b859388 100644 --- a/surpyval/recurrent/parametric/counting_process.py +++ b/surpyval/recurrent/parametric/counting_process.py @@ -25,17 +25,17 @@ class CountingProcess(ABC): """ @abstractmethod - def iif(self, x: ArrayLike, *params) -> ArrayLike: + def iif(self, x: ArrayLike, *params: ArrayLike) -> ArrayLike: """Instantaneous intensity function (event rate) at ``x``.""" ... @abstractmethod - def cif(self, x: ArrayLike, *params) -> ArrayLike: + def cif(self, x: ArrayLike, *params: ArrayLike) -> ArrayLike: """Cumulative intensity (expected event count) by ``x``.""" ... @abstractmethod - def log_iif(self, x: ArrayLike, *params) -> ArrayLike: + def log_iif(self, x: ArrayLike, *params: ArrayLike) -> ArrayLike: """Natural logarithm of the instantaneous intensity at ``x``.""" ... @@ -66,7 +66,7 @@ class IntensityModel(CountingProcess): """ @abstractmethod - def inv_cif(self, N: ArrayLike, *params) -> ArrayLike: + def inv_cif(self, N: ArrayLike, *params: ArrayLike) -> ArrayLike: """Time by which ``N`` events are expected; the inverse of ``cif``.""" ... diff --git a/surpyval/tests/univariate/parametric/test_binomial.py b/surpyval/tests/univariate/parametric/test_binomial.py index dd8cff8..aae650d 100644 --- a/surpyval/tests/univariate/parametric/test_binomial.py +++ b/surpyval/tests/univariate/parametric/test_binomial.py @@ -12,7 +12,7 @@ import pytest from scipy.stats import binom -from surpyval import Bernoulli, Binomial, Parametric +from surpyval import Bernoulli, Binomial, FixedEventProbability, Parametric N, P = 5, 0.3 @@ -94,17 +94,30 @@ def test_fit_with_counts(): def test_reduces_to_bernoulli_at_n_one(): - # At n = 1 the event probabilities match the Bernoulli: P(K=1) = p and - # P(K=0) = 1 - p. Note surpyval's Bernoulli is a degenerate - # "fixed event probability" model whose survival is the constant - # probability of *no* event (1 - p), so it lines up with the binomial's - # P(K = 0) = ff(0), not its sf(0). + # At n = 1 the binomial *is* the Bernoulli, and since 0.19.1 the two + # agree exactly on the probability mass: binomial = Binomial.from_params([1, P]) bernoulli = Bernoulli.from_params(P) assert np.isclose(binomial.df(1), P) assert np.isclose(binomial.df(0), 1 - P) - assert np.isclose(binomial.ff(0), bernoulli.sf(0)) - assert np.isclose(binomial.sf(0), bernoulli.ff(0)) + np.testing.assert_allclose( + np.asarray(bernoulli.df([0, 1]), dtype=float), + np.asarray(binomial.df([0, 1]), dtype=float), + ) + + # The survival functions are offset by one, and that is a convention + # rather than a disagreement. Binomial follows the package's discrete + # rule R(k) = P(K > k); Bernoulli uses R(x) = P(X >= x), so that + # R(0) = 1 and R(1) = p read as a one-shot device. Hence: + for x in (0, 1): + assert np.isclose(bernoulli.sf(x), binomial.sf(x - 1)) + + # Before 0.19.1 Bernoulli was a flat "fixed event probability" model + # with F(x) = p at every x, which lined up with neither. That model + # still exists under its own name and is unchanged. + fixed = FixedEventProbability.from_params(P) + assert np.isclose(fixed.ff(0), P) + assert np.isclose(fixed.ff(37.5), P) @pytest.mark.parametrize( @@ -139,3 +152,28 @@ def test_to_dict_roundtrip(): restored = Parametric.from_dict(model.to_dict()) assert np.allclose(restored.params, [N, P]) assert np.isclose(restored.mean(), N * P) + + +def test_support_brackets_the_outcomes_exclusively(): + # ``support`` is a pair of exclusive bounds -- ``_validate_fit_inputs`` + # rejects ``x <= support[0]`` and ``x >= support[1]`` -- so both must + # sit one step outside the outcomes {0, ..., n}. Zero events and n + # events are ordinary outcomes with real mass, and the bounds used to + # exclude both. Nothing observed it because Binomial does not inherit + # OptimisedFitMixin, where that check lives. + n_trials = 5 + for model in ( + Binomial.from_params([n_trials, 0.3]), + Binomial.fit([0, 2, 3, 5, 1], n_trials=n_trials), + ): + lower, upper = model.support + for k in (0, n_trials): + assert lower < k < upper, k + assert Binomial.df(k, n_trials, 0.3) > 0 + + +def test_class_level_support_admits_zero_events(): + # The class-level bound is checked before n is known, so only its + # lower end is meaningful; it must still admit k = 0, as Poisson's + # does. It read 0 -- Geometric's value, whose first mass is at k = 1. + assert Binomial.support[0] < 0 diff --git a/surpyval/tests/univariate/parametric/test_conditional_survival.py b/surpyval/tests/univariate/parametric/test_conditional_survival.py new file mode 100644 index 0000000..d1a1a75 --- /dev/null +++ b/surpyval/tests/univariate/parametric/test_conditional_survival.py @@ -0,0 +1,188 @@ +""" +Numerical coverage for ``cs``, the conditional survival function. + +Eleven distributions used to carry their own ``cs``, each with the same +one-line body as ``ParametricFitter.cs`` and a docstring example that +pinned its numbers. The bodies were duplication -- and duplication that +had already rotted, since Gamma's docstring stated the *exponential* +survival function above a body that computed the ratio correctly. The +overrides are gone; the numbers they pinned are here, so deleting the +docstrings did not delete the only per-distribution check on ``cs``. + +Expected values are the ones those docstrings recorded, which the +doctest run verified on every supported interpreter. +""" + +import numpy as np +import pytest + +from surpyval import ( + Beta, + Exponential, + ExpoWeibull, + Gamma, + LogLogistic, + LogNormal, + Normal, + Rayleigh, + Uniform, + Weibull, +) + +X = np.array([1, 2, 3, 4, 5]) + +# (distribution, x, X, params, expected) -- lifted from the docstrings +# the overrides used to carry. +CASES = [ + ( + Beta, + np.array([0.1, 0.2, 0.3, 0.4, 0.5]), + 0.4, + (3, 4), + [0.6315219, 0.32921811, 0.12946429, 0.03115814, 0.00233319], + ), + ( + ExpoWeibull, + X, + 1, + (3, 4, 1.2), + [ + 8.77367129e-01, + 4.25451775e-01, + 5.09266354e-02, + 5.37452200e-04, + 1.35732908e-07, + ], + ), + ( + Exponential, + X, + 5, + (3,), + [ + 4.97870684e-02, + 2.47875218e-03, + 1.23409804e-04, + 6.14421235e-06, + 3.05902321e-07, + ], + ), + ( + Gamma, + X, + 5, + (3, 4), + [ + 2.59402488e-02, + 6.39048747e-04, + 1.51519143e-05, + 3.48776510e-07, + 7.79933496e-09, + ], + ), + ( + LogLogistic, + X, + 5, + (3, 4), + [0.51270879, 0.28444803, 0.16902083, 0.10629329, 0.07003273], + ), + ( + LogNormal, + X, + 5, + (3, 4), + [0.97287811, 0.9496515, 0.92933892, 0.91129122, 0.89505592], + ), + ( + Normal, + X, + 5, + (3, 4), + [0.73452116, 0.51421702, 0.34242113, 0.2165286, 0.1298356], + ), + ( + Rayleigh, + X, + 5, + (3,), + [0.54274748, 0.26359714, 0.11455884, 0.04455143, 0.01550385], + ), + ( + Uniform, + X, + 4, + (0, 10), + [0.83333333, 0.66666667, 0.5, 0.33333333, 0.16666667], + ), + ( + Weibull, + X, + 5, + (3, 4), + [ + 2.52537548e-04, + 3.00394073e-10, + 2.45288508e-19, + 1.48999440e-32, + 5.42544000e-51, + ], + ), +] + +IDS = [c[0].name for c in CASES] + + +@pytest.mark.parametrize("dist, x, cond, params, expected", CASES, ids=IDS) +def test_cs_matches_the_documented_values(dist, x, cond, params, expected): + # The docstrings printed eight decimal places, so a value like + # 0.01550385 pins the result to about 2e-7 relative -- atol carries + # the fixed-decimal entries, rtol the ones in scientific notation. + got = np.asarray(dist.cs(x, cond, *params), dtype=float) + np.testing.assert_allclose(got, np.array(expected), rtol=1e-6, atol=5e-9) + + +@pytest.mark.parametrize("dist, x, cond, params, expected", CASES, ids=IDS) +def test_cs_equals_the_survival_ratio(dist, x, cond, params, expected): + # The property the inherited implementation encodes. Exponential is + # included deliberately: its override returns sf(x) on the strength + # of memorylessness, and this is what checks that shortcut is the + # same function the others compute the long way. + got = np.asarray(dist.cs(x, cond, *params), dtype=float) + ratio = np.asarray( + dist.sf(x + cond, *params) / dist.sf(cond, *params), dtype=float + ) + np.testing.assert_allclose(got, ratio, rtol=1e-9) + + +def test_cs_at_zero_is_one(): + # Surviving a further nothing is certain, whatever has been survived. + for dist, _, cond, params, _ in CASES: + got = np.asarray(dist.cs(0.0, cond, *params), dtype=float) + np.testing.assert_allclose(got, 1.0, rtol=1e-9, atol=1e-12) + + +def test_exponential_cs_is_memoryless(): + # The reason Exponential keeps an override. Conditioning on any + # amount of prior survival leaves the distribution unchanged. + x = np.array([0.5, 1.0, 2.0, 4.0]) + base = np.asarray(Exponential.sf(x, 3), dtype=float) + for cond in (0.0, 1.0, 10.0, 100.0): + got = np.asarray(Exponential.cs(x, cond, 3), dtype=float) + np.testing.assert_allclose(got, base, rtol=1e-12) + + +def test_discrete_distributions_inherit_a_working_cs(): + # These never defined cs and reach the base implementation. Before + # the base gained one they raised AttributeError. + from surpyval import Geometric, NegativeBinomial, Poisson + + for dist, params in ( + (Poisson, (3.0,)), + (Geometric, (0.3,)), + (NegativeBinomial, (2.0, 0.4)), + ): + got = np.asarray(dist.cs(np.array([1, 2, 3]), 2, *params), dtype=float) + assert got.shape == (3,) + assert np.all(np.isfinite(got)) + assert np.all((got >= 0) & (got <= 1 + 1e-12)) diff --git a/surpyval/tests/univariate/parametric/test_degenerate.py b/surpyval/tests/univariate/parametric/test_degenerate.py index 6b5af52..788b171 100644 --- a/surpyval/tests/univariate/parametric/test_degenerate.py +++ b/surpyval/tests/univariate/parametric/test_degenerate.py @@ -61,3 +61,51 @@ def test_import_paths_preserved(): assert from_parametric_n is NeverOccurs is from_module_n assert from_parametric_i is InstantlyOccurs + + +def test_exact_event_time_refuses_density_and_hazard(): + # A point mass has no density: all of its probability sits at T, so + # the density is a Dirac delta rather than a function of x. These + # used to return inf at T (and, for hf, at every x after it), which + # integrates to inf rather than 1 and propagates silently into + # whatever consumes it. + import pytest + + from surpyval import ExactEventTime + + x = np.array([4.0, 5.0, 6.0]) + with pytest.raises(NotImplementedError, match="no density"): + ExactEventTime.df(x, 5.0) + with pytest.raises(NotImplementedError, match="no hazard rate"): + ExactEventTime.hf(x, 5.0) + # log_df is inherited and reaches hf, so it refuses too rather than + # returning the nan that log(inf) - inf used to give. + with pytest.raises(NotImplementedError): + ExactEventTime.log_df(x, 5.0) + + +def test_exact_event_time_keeps_the_functions_that_are_defined(): + # sf, ff and Hf are genuine step functions and are unaffected. + from surpyval import ExactEventTime + + x = np.array([4.0, 4.999, 5.0, 6.0]) + np.testing.assert_array_equal(ExactEventTime.sf(x, 5.0), [1, 1, 0, 0]) + np.testing.assert_array_equal(ExactEventTime.ff(x, 5.0), [0, 0, 1, 1]) + Hf = np.asarray(ExactEventTime.Hf(x, 5.0), dtype=float) + np.testing.assert_array_equal(Hf, [0.0, 0.0, np.inf, np.inf]) + # Hf is -log R(x), and used to be an alias for hf that happened to + # take the same two values. + with np.errstate(divide="ignore"): + expected = -np.log(np.asarray(ExactEventTime.sf(x, 5.0), dtype=float)) + np.testing.assert_array_equal(Hf, expected) + + +def test_exact_event_time_still_fits_and_serialises(): + # The estimator brackets T between the censoring bounds and never + # touches a density, so refusing df and hf cannot affect it. + from surpyval import ExactEventTime + + model = ExactEventTime.fit(x=[1.0, 2.0, 8.0, 9.0], c=[1, 1, -1, -1]) + np.testing.assert_allclose(model.params, [5.0]) + restored = surpyval.from_dict(model.to_dict()) + np.testing.assert_allclose(restored.params, model.params) diff --git a/surpyval/tests/univariate/parametric/test_discrete.py b/surpyval/tests/univariate/parametric/test_discrete.py index 2ba7a6c..0f3da55 100644 --- a/surpyval/tests/univariate/parametric/test_discrete.py +++ b/surpyval/tests/univariate/parametric/test_discrete.py @@ -12,7 +12,9 @@ from scipy.stats import geom, nbinom, poisson from surpyval import ( + Bernoulli, BetaGeometric, + Binomial, DiscreteWeibull, Discretize, Gamma, @@ -338,3 +340,345 @@ def test_discretize_rejects_negative_support(): def test_discretize_name_is_distinct_from_discrete_weibull(): assert DiscretizedWeibull.name == "Discretize(Weibull)" assert DiscretizedWeibull.name != DiscreteWeibull.name + + +def test_log_df_uses_the_discrete_mass_identity(): + # P(T = k) = h(k) R(k - 1). ParametricFitter.log_df encodes the + # continuous f = h R(x) instead, which puts R(k) where R(k - 1) + # belongs. Binomial inherited it and returned the mass scaled by + # R(k)/R(k - 1) -- 0.88 of the truth at k=1, falling to 0.15 by k=7. + x = np.arange(1, 8, dtype=float) + cases = [ + (Binomial, (10, 0.3)), + (Poisson, (3.0,)), + (Geometric, (0.3,)), + (NegativeBinomial, (4.0, 0.4)), + (DiscreteWeibull, (0.6, 1.2)), + (BetaGeometric, (2.0, 3.0)), + ] + for dist, params in cases: + df = np.asarray(dist.df(x, *params), dtype=float) + log_df = np.asarray(dist.log_df(x, *params), dtype=float) + np.testing.assert_allclose( + np.exp(log_df), df, rtol=1e-9, err_msg=f"{dist.name}" + ) + + +def test_binomial_log_df_matches_scipy(): + # Binomial is the one that reached the inherited implementation, so + # it is the one worth pinning against an independent source. + from scipy.stats import binom + + x = np.arange(0, 11, dtype=float) + np.testing.assert_allclose( + np.asarray(Binomial.log_df(x, 10, 0.3), dtype=float), + binom.logpmf(x, 10, 0.3), + rtol=1e-9, + ) + + +def test_discrete_hazard_is_a_probability(): + # The discrete hazard is P(T = k)/P(T >= k), which is a probability + # and cannot exceed one. The continuous form P(T = k)/P(T > k) can, + # which is how a mixed-up convention shows itself. + x = np.arange(1, 12, dtype=float) + for dist, params in [ + (Poisson, (3.0,)), + (Geometric, (0.3,)), + (DiscreteWeibull, (0.6, 1.2)), + (NegativeBinomial, (4.0, 0.4)), + (BetaGeometric, (2.0, 3.0)), + (Binomial, (10, 0.3)), + ]: + hf = np.asarray(dist.hf(x, *params), dtype=float) + finite = hf[np.isfinite(hf)] + assert np.all(finite >= 0.0), dist.name + assert np.all( + finite <= 1.0 + 1e-12 + ), f"{dist.name}: max {finite.max()}" + + +def test_cumulative_hazard_accumulates_the_discrete_way(): + # For a discrete distribution R(k) = prod(1 - h), so the cumulative + # hazard is -sum log(1 - h), not the sum of the hazards. Hf and hf + # must agree through that relation. + x = np.arange(1, 9, dtype=float) + for dist, params in [ + (Poisson, (3.0,)), + (Geometric, (0.3,)), + (DiscreteWeibull, (0.6, 1.2)), + (Binomial, (10, 0.3)), + ]: + hf = np.asarray(dist.hf(x, *params), dtype=float) + Hf = np.asarray(dist.Hf(x, *params), dtype=float) + seed = -np.log(float(np.asarray(dist.sf(0.0, *params)))) + np.testing.assert_allclose( + np.cumsum(-np.log1p(-hf)) + seed, + Hf, + rtol=1e-7, + err_msg=f"{dist.name}", + ) + + +# --------------------------------------------------------------------------- +# Bernoulli: a true coin flip since 0.19.1 +# --------------------------------------------------------------------------- + +P_BERN = 0.3 + + +def test_bernoulli_functions_at_the_two_outcomes(): + x = np.array([0, 1]) + np.testing.assert_allclose(Bernoulli.sf(x, P_BERN), [1.0, P_BERN]) + np.testing.assert_allclose(Bernoulli.ff(x, P_BERN), [0.0, 1 - P_BERN]) + np.testing.assert_allclose(Bernoulli.df(x, P_BERN), [1 - P_BERN, P_BERN]) + np.testing.assert_allclose(Bernoulli.hf(x, P_BERN), [1 - P_BERN, 1.0]) + np.testing.assert_allclose(Bernoulli.Hf(x, P_BERN), [0.0, -np.log(P_BERN)]) + + +def test_bernoulli_rejects_anything_but_zero_and_one(): + # x is the outcome of the flip, not a time. Before 0.19.1 every x + # returned the same number, so nothing marked 37.5 as meaningless. + for bad in (0.5, 2, -1, 37.5): + for method in ( + Bernoulli.sf, + Bernoulli.ff, + Bernoulli.df, + Bernoulli.hf, + Bernoulli.Hf, + ): + with pytest.raises(ValueError, match="x = 0 and x = 1 only"): + method(bad, P_BERN) + + +def test_bernoulli_internal_identities(): + x = np.array([0, 1]) + sf = np.asarray(Bernoulli.sf(x, P_BERN), dtype=float) + ff = np.asarray(Bernoulli.ff(x, P_BERN), dtype=float) + df = np.asarray(Bernoulli.df(x, P_BERN), dtype=float) + hf = np.asarray(Bernoulli.hf(x, P_BERN), dtype=float) + Hf = np.asarray(Bernoulli.Hf(x, P_BERN), dtype=float) + log_df = np.asarray(Bernoulli.log_df(x, P_BERN), dtype=float) + + np.testing.assert_allclose(sf + ff, 1.0) + np.testing.assert_allclose(df.sum(), 1.0) + np.testing.assert_allclose(hf, df / sf) + np.testing.assert_allclose(Hf, -np.log(sf)) + np.testing.assert_allclose(np.exp(log_df), df) + # E[X] from the mass equals the parameter, and equals mean/moment. + np.testing.assert_allclose((x * df).sum(), P_BERN) + np.testing.assert_allclose(Bernoulli.mean(P_BERN), P_BERN) + np.testing.assert_allclose(Bernoulli.moment(1, P_BERN), P_BERN) + # X is 0 or 1 so X**m == X, and every moment is p. + for m in (1, 2, 5): + np.testing.assert_allclose(Bernoulli.moment(m, P_BERN), P_BERN) + + +def test_bernoulli_log_df_needs_its_own_relation(): + # DiscreteParametricFitter.log_df is f(k) = h(k) R(k - 1), which + # assumes R(k) = P(X > k). Bernoulli's R is P(X >= x), so the + # at-risk set at x is R(x) itself. Inheriting the discrete relation + # would give df(1) = 1 instead of p. + x = np.array([0, 1]) + np.testing.assert_allclose( + np.exp(np.asarray(Bernoulli.log_df(x, P_BERN), dtype=float)), + np.asarray(Bernoulli.df(x, P_BERN), dtype=float), + ) + # At x = 1 the discrete relation would read h(1) * R(0) = 1 * 1 = 1, + # where the mass is p. (R(-1) is not even askable here, which is the + # other half of why that relation does not transfer.) + h1 = float(np.ravel(Bernoulli.hf(1, P_BERN))[0]) + R0 = float(np.ravel(Bernoulli.sf(0, P_BERN))[0]) + assert np.isclose(h1 * R0, 1.0) + assert not np.isclose(h1 * R0, P_BERN) + with pytest.raises(ValueError): + Bernoulli.sf(-1, P_BERN) + + +def test_bernoulli_fit_recovers_the_fraction_of_ones(): + data = np.array([1, 1, 0, 1, 0, 0, 1, 1, 1, 0]) + model = Bernoulli.fit(data) + np.testing.assert_allclose(model.params, [data.mean()]) + + +def test_fixed_event_probability_is_unchanged_and_separate(): + # The flat model Bernoulli used to be. It is now its own class, so + # making Bernoulli a real Bernoulli did not take it away. + from surpyval import FixedEventProbability + + assert type(FixedEventProbability) is not type(Bernoulli) + for x in (0.0, 1.0, 37.5, -12.0): + np.testing.assert_allclose(FixedEventProbability.ff(x, P_BERN), P_BERN) + np.testing.assert_allclose( + FixedEventProbability.sf(x, P_BERN), 1 - P_BERN + ) + # Its F is constant, so it still has no density, hazard rate, quantile + # or mean: the mass is an atom rather than a density, and there is no + # time axis to invert or average over. + for absent in ("df", "hf", "qf", "mean"): + assert not any( + absent in k.__dict__ for k in type(FixedEventProbability).__mro__ + ), absent + # ``Hf`` is the exception, and is present. -ln R(x) is a perfectly good + # constant, exactly as for ExactEventTime, whose Hf exists while its hf + # does not. Its absence was not a design decision but an omission: the + # base class writes log_sf and log_ff in terms of Hf, so both raised + # AttributeError instead of returning the constants they should. + np.testing.assert_allclose( + FixedEventProbability.Hf(np.array([1.0, 9.0]), P_BERN), + -np.log(1 - P_BERN), + ) + np.testing.assert_allclose( + FixedEventProbability.log_sf(np.array([1.0, 9.0]), P_BERN), + np.log(1 - P_BERN), + ) + np.testing.assert_allclose( + FixedEventProbability.log_ff(np.array([1.0, 9.0]), P_BERN), + np.log(P_BERN), + ) + + +def test_both_models_round_trip_under_their_own_names(): + import surpyval + from surpyval import FixedEventProbability + + for dist in (Bernoulli, FixedEventProbability): + model = dist.from_params(P_BERN) + restored = surpyval.from_dict(model.to_dict()) + assert restored.dist.name == dist.name + np.testing.assert_allclose(restored.params, model.params) + + +def test_bernoulli_qf_is_the_standard_quantile(): + from scipy.stats import binom + + # On (0, 1) it is exactly binom.ppf(u, 1, p) -- the smallest outcome + # k with P(X <= k) >= u. + u = np.array([0.01, 0.1, 0.5, 0.699, 0.7, 0.701, 0.9, 0.99]) + np.testing.assert_allclose( + np.asarray(Bernoulli.qf(u, P_BERN), dtype=float), + binom.ppf(u, 1, P_BERN), + ) + np.testing.assert_allclose( + np.asarray(Bernoulli.qf(u, P_BERN), dtype=float), + np.asarray(Binomial.qf(u, 1, P_BERN), dtype=float), + ) + # It steps at 1 - p, not at p. + assert float(np.ravel(Bernoulli.qf(1 - P_BERN, P_BERN))[0]) == 0.0 + assert float(np.ravel(Bernoulli.qf(1 - P_BERN + 1e-9, P_BERN))[0]) == 1.0 + # At u = 0 scipy answers -1, one below the support; this answers 0. + assert float(np.ravel(Bernoulli.qf(0.0, P_BERN))[0]) == 0.0 + + +def test_bernoulli_qf_drives_inverse_transform_sampling(): + # The property that makes qf worth having: qf(U) reproduces the + # distribution, which is what ParametricFitter.random does. + rng = np.random.default_rng(0) + U = rng.uniform(size=200_000) + draws = np.asarray(Bernoulli.qf(U, P_BERN), dtype=float) + assert set(np.unique(draws)) <= {0.0, 1.0} + assert np.isclose(draws.mean(), P_BERN, atol=0.005) + + +def test_bernoulli_qf_does_not_invert_this_ff_and_says_so(): + # Documented consequence of R(x) = P(X >= x): the failure function is + # P(X < x), which never exceeds 1 - p on {0, 1}, so the usual + # discrete check ff(qf(u)) >= u cannot hold once u passes 1 - p. The + # other discrete distributions, whose R(k) is P(X > k), are fine. + u = 0.9 # above 1 - p = 0.7 + k = float(np.ravel(Bernoulli.qf(u, P_BERN))[0]) + assert k == 1.0 + assert float(np.ravel(Bernoulli.ff(k, P_BERN))[0]) < u + # Whereas for a distribution using the package's R(k) = P(X > k): + k_pois = float(np.ravel(Poisson.qf(u, 3.0))[0]) + assert float(np.ravel(Poisson.ff(k_pois, 3.0))[0]) >= u - 1e-9 + + +@pytest.mark.parametrize("p", [0.0, 1.0]) +def test_bernoulli_qf_at_the_degenerate_ends(p): + u = np.array([0.01, 0.5, 0.99]) + expected = 0.0 if p == 0.0 else 1.0 + np.testing.assert_allclose( + np.asarray(Bernoulli.qf(u, p), dtype=float), expected + ) + + +# --------------------------------------------------------------------------- +# Behaviour below the support +# --------------------------------------------------------------------------- + +# (distribution, params, first mass point). Every one of these is defined on +# consecutive integers from the third entry upwards; below it there is no +# mass, so the pmf is zero and the survival is one. The closed forms are +# algebraic and do not know that -- left alone they returned a positive +# "probability" (Geometric 0.43 at k = 0), a survival above one +# (BetaGeometric 2.0 at k = -1), a complex number (DiscreteWeibull) or a +# NaN (Poisson, NegativeBinomial). +_DISCRETE_SUPPORTS = [ + (Geometric, (0.3,), 1), + (DiscreteWeibull, (0.6, 1.4), 1), + (BetaGeometric, (2.0, 3.0), 1), + (NegativeBinomial, (3.0, 0.4), 1), + (Poisson, (2.5,), 0), + (Binomial, (5.0, 0.3), 0), +] + + +@pytest.mark.parametrize("dist, params, first", _DISCRETE_SUPPORTS) +def test_below_the_support_is_real_and_finite(dist, params, first): + below = np.arange(-4.0, float(first)) + for method in ("sf", "ff", "df", "hf", "Hf", "log_sf", "log_df"): + value = np.asarray(getattr(dist, method)(below, *params)) + assert np.isrealobj(value), f"{dist.name}.{method} returned complex" + assert not np.isnan( + np.asarray(value, dtype=float) + ).any(), f"{dist.name}.{method} returned NaN below its support" + + +@pytest.mark.parametrize("dist, params, first", _DISCRETE_SUPPORTS) +def test_no_mass_below_the_support(dist, params, first): + below = np.arange(-4.0, float(first)) + as_float = lambda v: np.asarray(v, dtype=float) # noqa: E731 + np.testing.assert_allclose(as_float(dist.df(below, *params)), 0.0) + np.testing.assert_allclose(as_float(dist.hf(below, *params)), 0.0) + np.testing.assert_allclose(as_float(dist.sf(below, *params)), 1.0) + np.testing.assert_allclose(as_float(dist.ff(below, *params)), 0.0) + np.testing.assert_allclose(as_float(dist.Hf(below, *params)), 0.0) + + +@pytest.mark.parametrize("dist, params, first", _DISCRETE_SUPPORTS) +def test_the_pmf_sums_to_one_over_the_support(dist, params, first): + # Summed from below the first mass point, so any spurious mass there + # would push the total past one. BetaGeometric's tail decays as k^-a, + # hence the looser tolerance rather than a longer sum. + k = np.arange(-4.0, 4000.0) + total = float(np.sum(np.asarray(dist.df(k, *params), dtype=float))) + assert total == pytest.approx(1.0, abs=1e-4) + + +@pytest.mark.parametrize("dist, params, first", _DISCRETE_SUPPORTS) +def test_the_quantile_inverts_the_cdf_on_the_support(dist, params, first): + # u = F(k) is formed by cancellation, so it lands a few ulp off the + # value the quantile is looking for. Geometric, DiscreteWeibull and + # BetaGeometric all answered k + 1 for a u that came straight out of + # their own ff. + last = 5 if dist is Binomial else 11 + k = np.arange(float(first), float(last) + 1.0) + inverted = np.asarray(dist.qf(dist.ff(k, *params), *params), dtype=float) + np.testing.assert_array_equal(inverted, k) + + +def test_beta_geometric_moment_diverges_when_the_tail_is_too_heavy(): + # R(k) decays as k^-a, so E[T^m] exists only for a > m. A truncated + # sum cannot see that: it reported about 25 for a second moment that + # is infinite. + assert BetaGeometric.moment(2, 2.0, 3.0) == np.inf + assert BetaGeometric.mean(1.0, 3.0) == np.inf + assert np.isfinite(BetaGeometric.moment(2, 4.0, 3.0)) + + +def test_beta_geometric_mean_agrees_with_its_first_moment(): + for a, b in [(2.0, 3.0), (4.0, 1.5), (3.0, 7.0)]: + assert BetaGeometric.moment(1, a, b) == pytest.approx( + BetaGeometric.mean(a, b) + ) diff --git a/surpyval/tests/univariate/parametric/test_distribution_fixes.py b/surpyval/tests/univariate/parametric/test_distribution_fixes.py index efff251..3af927d 100644 --- a/surpyval/tests/univariate/parametric/test_distribution_fixes.py +++ b/surpyval/tests/univariate/parametric/test_distribution_fixes.py @@ -15,13 +15,18 @@ Exponential, ExpoWeibull, Gamma, + Gumbel, GumbelLEV, Logistic, LogNormal, + Normal, + Rayleigh, + Weibull, ) from surpyval.univariate.parametric.parametric_fitter import ( ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData def test_lognormal_fits_negative_mu(): @@ -165,3 +170,84 @@ def test_from_params_signature_matches_the_base(): for dist in (Bernoulli, Binomial, ExactEventTime): own = set(inspect.signature(type(dist).from_params).parameters) assert base <= own, dist.name + + +# --- Rayleigh's initial guess had to be a sequence ------------------------ +# +# Rayleigh is the only single-parameter distribution here, and its +# _parameter_initialiser returned a bare scalar for the non-offset case. +# `np.array(init)` in _initial_guess then produced a 0-dimensional array +# rather than a length-1 one, and the lfp and zi paths concatenate the p +# and f0 seeds onto it -- which a 0-d array cannot do. + + +def test_rayleigh_initial_guess_is_a_sequence(): + seed = Rayleigh._parameter_initialiser( + SurpyvalData(np.array([1.0, 2.0, 3.0, 4.0])) + ) + assert np.array(seed).ndim == 1 + + +@pytest.mark.parametrize("structural", ["lfp", "zi"]) +def test_rayleigh_fits_with_lfp_and_zi(structural): + np.random.seed(0) + x = Rayleigh.random(200, 10.0) + if structural == "zi": + x = np.concatenate([x, np.zeros(10)]) + model = Rayleigh.fit(x, **{structural: True}) + # The sigma estimate is unaffected; the point is that it runs at all. + assert model.params[0] == pytest.approx(9.92, abs=0.5) + + +# --- _parameter_initialiser takes a SurpyvalData ------------------------- +# +# It used to take (x, c=None, n=None, t=None, offset=False), and every +# implementation re-established the conventions that SurpyvalData had +# already guaranteed -- inconsistently. Normal indexed with c and Gumbel +# tested membership on it before either was defaulted, so both raised +# TypeError for the signature the base class documented; every caller +# inside the package passed c and n, which is why it went unnoticed. +# There is now nothing to default: the argument is the normalised +# object, so c, n and t are always arrays. + + +@pytest.mark.parametrize( + "dist", + [Normal, Gumbel, GumbelLEV, Weibull, LogNormal, Logistic, Rayleigh], +) +def test_parameter_initialiser_takes_surpyval_data(dist): + x = np.array([1.0, 2.0, 3.0, 4.0, 5.5]) + seed = dist._parameter_initialiser(SurpyvalData(x)) + assert np.asarray(seed).ndim == 1 + assert np.isfinite(np.asarray(seed, dtype=float)).all() + + +@pytest.mark.parametrize( + "dist", + [Normal, Gumbel, GumbelLEV, Weibull, LogNormal, Logistic, Rayleigh], +) +def test_parameter_initialiser_rejects_loose_arrays(dist): + # The old signature is gone rather than deprecated. Passing a bare + # array reaches the ``.x`` attribute access and fails loudly, which + # is the point: a silent partial acceptance is what let the c=None + # divergence above survive. + with pytest.raises(AttributeError): + dist._parameter_initialiser(np.array([1.0, 2.0, 3.0])) + + +def test_exact_event_time_has_a_quantile_mean_and_moments(): + # A point mass has no density and no hazard rate -- df and hf raise, + # and say why -- but its quantile, mean and moments are all exact and + # trivial. They were simply missing, so a caller reaching for the mean + # of a known event time got an AttributeError. + T = 5.0 + np.testing.assert_allclose( + np.asarray(ExactEventTime.qf([0.01, 0.5, 0.99], T), dtype=float), T + ) + assert ExactEventTime.mean(T) == T + assert ExactEventTime.moment(1, T) == T + assert ExactEventTime.moment(3, T) == T**3 + # And the ones that genuinely do not exist still refuse. + for method in ("df", "hf"): + with pytest.raises(NotImplementedError): + getattr(ExactEventTime, method)(np.array([1.0]), T) diff --git a/surpyval/tests/univariate/parametric/test_distributions_math.py b/surpyval/tests/univariate/parametric/test_distributions_math.py index 2e704e1..45d3c18 100644 --- a/surpyval/tests/univariate/parametric/test_distributions_math.py +++ b/surpyval/tests/univariate/parametric/test_distributions_math.py @@ -12,10 +12,12 @@ """ import math +from math import comb import numpy as np import pytest from scipy import integrate +from scipy.special import gamma as gamma_func from scipy.special import xlogy from surpyval import ( @@ -265,10 +267,16 @@ def test_mean_matches_numerical_integration(dist, params): ), f"{dist.name}: mean() = {computed}, integration gives {expected}" -# ExpoWeibull does not implement moment(); LogLogistic needs a larger shape -# parameter than DIST_PARAMS uses so its second moment exists with a tail -# light enough for truncated integration. +# LogLogistic needs a larger shape parameter than DIST_PARAMS uses so its +# second moment exists with a tail light enough for truncated integration. +# +# ExpoWeibull is included even though its own moment() is quadrature: the +# reference here integrates between quantiles with breakpoints, which is a +# different scheme from moment()'s plain 0-to-infinity call, so the two +# agreeing is a real check rather than a tautology. The closed-form checks +# below pin it independently. MOMENT_PARAMS = [ + (ExpoWeibull, (3.0, 1.5, 0.8)), (Gumbel, (-1.0, 2.0)), (GumbelLEV, (3.0, 1.5)), (Normal, (5.0, 2.0)), @@ -296,3 +304,131 @@ def test_moment_matches_numerical_integration(dist, params, n): assert math.isclose( computed, expected, rel_tol=1e-5 ), f"{dist.name}: moment({n}) = {computed}, integration gives {expected}" + + +# --------------------------------------------------------------------------- +# 8. ExpoWeibull.moment against closed forms that do not use quadrature +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("m", [1, 2, 3, 4]) +def test_expoweibull_moment_reduces_to_weibull_at_mu_one(m): + # mu = 1 collapses the exponentiated Weibull to the Weibull, whose + # m-th moment is alpha^m * Gamma(1 + m/beta) exactly. This is the + # strongest check available: an independent closed form, no + # integration on the reference side. + alpha, beta = 3.0, 4.0 + expected = alpha**m * gamma_func(1.0 + m / beta) + computed = ExpoWeibull.moment(m, alpha, beta, 1.0) + assert math.isclose(computed, expected, rel_tol=1e-9), ( + f"ExpoWeibull.moment({m}, {alpha}, {beta}, 1) = {computed}, " + f"Weibull closed form gives {expected}" + ) + np.testing.assert_allclose( + computed, float(np.ravel(Weibull.moment(m, alpha, beta))[0]), rtol=1e-9 + ) + + +@pytest.mark.parametrize("mu", [1, 2, 3, 5]) +@pytest.mark.parametrize("m", [1, 2]) +def test_expoweibull_moment_matches_the_terminating_series(m, mu): + # For integer mu the series solution terminates: + # + # E[X^m] = mu alpha^m Gamma(1 + m/beta) + # sum_i (-1)^i C(mu - 1, i) (i + 1)^-(1 + m/beta) + # + # It is only usable here because mu is an integer -- for other mu it + # is an infinite alternating series that loses significance, which + # is why moment() integrates instead. + alpha, beta = 3.0, 4.0 + expected = ( + mu + * alpha**m + * gamma_func(1.0 + m / beta) + * sum( + (-1) ** i * comb(mu - 1, i) / (i + 1) ** (1.0 + m / beta) + for i in range(mu) + ) + ) + computed = ExpoWeibull.moment(m, alpha, beta, float(mu)) + assert math.isclose(computed, expected, rel_tol=1e-9), ( + f"ExpoWeibull.moment({m}, {alpha}, {beta}, {mu}) = {computed}, " + f"series gives {expected}" + ) + + +def test_expoweibull_mean_is_the_first_moment(): + # mean() delegates to moment(1) rather than repeating the integral. + for params in [(3.0, 4.0, 1.2), (10.0, 2.0, 0.7), (1.0, 1.0, 3.0)]: + assert math.isclose( + ExpoWeibull.mean(*params), + ExpoWeibull.moment(1, *params), + rel_tol=1e-12, + ) + + +def test_expoweibull_exponential_case_is_the_harmonic_sum(): + # alpha = beta = 1 is the exponentiated exponential, whose mean is + # the harmonic number H_mu for integer mu. Another reference with no + # integration in it. + for mu in (1, 2, 3, 4): + expected = sum(1.0 / i for i in range(1, mu + 1)) + assert math.isclose( + ExpoWeibull.mean(1.0, 1.0, float(mu)), expected, rel_tol=1e-9 + ) + + +# --------------------------------------------------------------------------- +# 9. Logistic moments come from differentiating the MGF +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("m", [1, 2, 3]) +def test_logistic_moment_matches_the_closed_form(m): + # Logistic.moment differentiates the moment generating function m + # times with autograd rather than using a closed form, because the + # general one needs Bernoulli numbers. These are the low-order raw + # moments written out, so the path is pinned against algebra rather + # than against another numerical method. + mu, sigma = 3.0, 2.0 + var = sigma**2 * math.pi**2 / 3.0 + expected = { + 1: mu, + 2: mu**2 + var, + 3: mu**3 + 3 * mu * var, # the logistic is symmetric, so skew = 0 + }[m] + assert math.isclose( + float(Logistic.moment(m, mu, sigma)), expected, rel_tol=1e-9 + ) + + +def test_logistic_variance_from_its_moments(): + mu, sigma = 3.0, 2.0 + var = ( + float(Logistic.moment(2, mu, sigma)) + - float(Logistic.moment(1, mu, sigma)) ** 2 + ) + assert math.isclose(var, sigma**2 * math.pi**2 / 3.0, rel_tol=1e-9) + + +def test_no_distribution_exposes_a_public_mgf(): + # Logistic had the only public `mgf` in the package, which read as a + # method the others were missing rather than as its own internal + # machinery -- it exists solely for `moment` to differentiate. It is + # `_mgf` now. + import surpyval + from surpyval.univariate.parametric.parametric_fitter import ( + ParametricFitter, + ) + + with_mgf = [ + name + for name in dir(surpyval) + if isinstance(getattr(surpyval, name), ParametricFitter) + and any( + "mgf" in k.__dict__ for k in type(getattr(surpyval, name)).__mro__ + ) + ] + assert not with_mgf, f"public mgf on: {with_mgf}" + # The private one is still there and still drives moment(). + assert hasattr(Logistic, "_mgf") diff --git a/surpyval/tests/univariate/parametric/test_fit.py b/surpyval/tests/univariate/parametric/test_fit.py index b2d63f2..7733d7d 100644 --- a/surpyval/tests/univariate/parametric/test_fit.py +++ b/surpyval/tests/univariate/parametric/test_fit.py @@ -16,6 +16,7 @@ Rayleigh, Weibull, ) +from surpyval.utils.surpyval_data import SurpyvalData DISTS = [ Gumbel, @@ -361,7 +362,13 @@ def test_offset_initialiser_puts_the_offset_first(dist, dist_params): n = np.ones(x.size, dtype=np.int64) init = np.asarray( - dist._initial_guess(x, c, n, True, False, False, "Nelson-Aalen"), + dist._initial_guess( + SurpyvalData(x, c, n, group_and_sort=False), + True, + False, + False, + "Nelson-Aalen", + ), dtype=float, ) assert len(init) == len(dist_params) + 1 @@ -566,7 +573,9 @@ def test_expoweibull_offset_seeds_from_shifted_data(): np.random.seed(11) x = 100.0 + ExpoWeibull.random(500, 10.0, 2.0, 1.0) - gamma, alpha, beta, mu = ExpoWeibull._parameter_initialiser(x, offset=True) + gamma, alpha, beta, mu = ExpoWeibull._parameter_initialiser( + SurpyvalData(x), offset=True + ) # Seeded from x - gamma these sit near the truth; seeded from x they # came back as alpha = 111, beta = 23.5. assert beta == pytest.approx(2.0, rel=0.5) diff --git a/surpyval/tests/univariate/parametric/test_fit_helpers.py b/surpyval/tests/univariate/parametric/test_fit_helpers.py index 6fed0ac..2f8ac93 100644 --- a/surpyval/tests/univariate/parametric/test_fit_helpers.py +++ b/surpyval/tests/univariate/parametric/test_fit_helpers.py @@ -18,6 +18,7 @@ Uniform, Weibull, ) +from surpyval.utils.surpyval_data import SurpyvalData # --------------------------------------------------------------------------- # _clamp_truncation_to_support @@ -57,11 +58,15 @@ def test_clamp_truncation_unbounded_is_noop(): def test_initial_guess_returns_one_value_per_parameter(): np.random.seed(0) x = Weibull.random(500, 10.0, 3.0) - c = np.zeros_like(x) - n = np.ones_like(x) + c = np.zeros(x.shape[0], dtype=int) + n = np.ones(x.shape[0], dtype=int) init = Weibull._initial_guess( - x, c, n, offset=False, zi=False, lfp=False, heuristic="Nelson-Aalen" + SurpyvalData(x, c, n, group_and_sort=False), + offset=False, + zi=False, + lfp=False, + heuristic="Nelson-Aalen", ) assert len(init) == Weibull.k @@ -73,11 +78,15 @@ def test_initial_guess_returns_one_value_per_parameter(): def test_initial_guess_lfp_appends_a_bounded_p_seed(): np.random.seed(1) x = Weibull.random(500, 10.0, 3.0) - c = np.zeros_like(x) - n = np.ones_like(x) + c = np.zeros(x.shape[0], dtype=int) + n = np.ones(x.shape[0], dtype=int) init = Weibull._initial_guess( - x, c, n, offset=False, zi=False, lfp=True, heuristic="Nelson-Aalen" + SurpyvalData(x, c, n, group_and_sort=False), + offset=False, + zi=False, + lfp=True, + heuristic="Nelson-Aalen", ) assert len(init) == Weibull.k + 1 @@ -88,11 +97,15 @@ def test_initial_guess_lfp_appends_a_bounded_p_seed(): def test_initial_guess_zi_appends_zero_fraction(): np.random.seed(2) x = np.concatenate([np.zeros(50), Weibull.random(450, 10.0, 3.0)]) - c = np.zeros_like(x) - n = np.ones_like(x) + c = np.zeros(x.shape[0], dtype=int) + n = np.ones(x.shape[0], dtype=int) init = Weibull._initial_guess( - x, c, n, offset=False, zi=True, lfp=False, heuristic="Nelson-Aalen" + SurpyvalData(x, c, n, group_and_sort=False), + offset=False, + zi=True, + lfp=False, + heuristic="Nelson-Aalen", ) assert len(init) == Weibull.k + 1 @@ -103,11 +116,15 @@ def test_initial_guess_zi_appends_zero_fraction(): def test_initial_guess_offset_seeds_gamma_below_min(): np.random.seed(3) x = Weibull.random(500, 10.0, 3.0) + 7.0 - c = np.zeros_like(x) - n = np.ones_like(x) + c = np.zeros(x.shape[0], dtype=int) + n = np.ones(x.shape[0], dtype=int) init = Weibull._initial_guess( - x, c, n, offset=True, zi=False, lfp=False, heuristic="Nelson-Aalen" + SurpyvalData(x, c, n, group_and_sort=False), + offset=True, + zi=False, + lfp=False, + heuristic="Nelson-Aalen", ) # Offset distributions carry gamma as the leading parameter; the seed @@ -123,10 +140,14 @@ def test_initial_guess_interval_data_uses_midpoint(): centres = Weibull.random(300, 10.0, 3.0) x = np.vstack([centres - 0.5, centres + 0.5]).T c = np.full(centres.shape, 2) - n = np.ones(centres.shape) + n = np.ones(centres.shape, dtype=int) init = Weibull._initial_guess( - x, c, n, offset=False, zi=False, lfp=False, heuristic="Nelson-Aalen" + SurpyvalData(x, c, n, group_and_sort=False), + offset=False, + zi=False, + lfp=False, + heuristic="Nelson-Aalen", ) assert len(init) == Weibull.k diff --git a/surpyval/tests/univariate/parametric/test_regressions.py b/surpyval/tests/univariate/parametric/test_regressions.py index 2ca086f..7432c48 100644 --- a/surpyval/tests/univariate/parametric/test_regressions.py +++ b/surpyval/tests/univariate/parametric/test_regressions.py @@ -16,6 +16,7 @@ Parametric, Weibull, ) +from surpyval.utils.surpyval_data import SurpyvalData def test_mse_with_offset(): @@ -44,7 +45,7 @@ def test_loglogistic_offset_initial_guess_length(): # The offset initial guess returned one parameter too many, which the # parameter transforms silently truncated. init = LogLogistic._parameter_initialiser( - np.array([1.0, 2.0, 3.0, 4.0, 5.0]), offset=True + SurpyvalData(np.array([1.0, 2.0, 3.0, 4.0, 5.0])), offset=True ) assert len(init) == LogLogistic.k + 1 @@ -53,7 +54,7 @@ def test_lognormal_offset_initial_guess_length(): # The offset initial guess ignored the offset flag and returned one # parameter too few, so offset fits crashed before optimising. init = LogNormal._parameter_initialiser( - np.array([1.0, 2.0, 3.0, 4.0, 5.0]), offset=True + SurpyvalData(np.array([1.0, 2.0, 3.0, 4.0, 5.0])), offset=True ) assert len(init) == LogNormal.k + 1 diff --git a/surpyval/tests/univariate/parametric/test_shared_signatures.py b/surpyval/tests/univariate/parametric/test_shared_signatures.py new file mode 100644 index 0000000..6cd50e6 --- /dev/null +++ b/surpyval/tests/univariate/parametric/test_shared_signatures.py @@ -0,0 +1,326 @@ +""" +A method that every distribution implements must have one signature. + +Not a style rule. A distribution is reached through a +``ParametricFitter`` reference all over the package -- ``fit_best`` +iterates a list of them, ``Discretize`` and ``MixtureModel`` wrap one, +the regression fitters hold one as ``self.dist`` -- and code written +against that reference has to work for every member. When the same +positional slot is called ``p`` on one distribution and ``u`` on +another, a keyword call is correct for a subset and a ``TypeError`` for +the rest, with nothing to say which until it runs. That is what made +the narrow ``from_params`` overrides on ``Bernoulli``, ``Binomial`` and +``ExactEventTime`` a defect rather than a naming preference (#257), and +the same argument applies to every shared method. + +These tests read the signatures rather than asserting a list of names, +so a distribution added later is covered without touching this file. +""" + +import ast +import inspect +import pathlib +from collections import defaultdict + +import numpy as np +import pytest + +import surpyval +from surpyval.univariate.parametric.parametric_fitter import ParametricFitter + +DIST_DIR = pathlib.Path(surpyval.__file__).parent / ( + "univariate/parametric/distributions" +) + +# The order of a moment, and the probability a quantile is taken at. +# ``m`` and ``u`` rather than ``n`` and ``p`` because ``n`` is +# Binomial's trial count and ``p`` is a parameter of Bernoulli, +# Binomial, Geometric and NegativeBinomial -- the two names that would +# otherwise read as the obvious choice are already taken. +CANONICAL_FIRST_ARG = { + "moment": "m", + "qf": "u", + "mpp_x_transform": "x", +} + + +def _signatures(method): + """{module stem: [parameter names]} for every implementation.""" + out = {} + for path in sorted(DIST_DIR.glob("*.py")): + if path.stem == "__init__": + continue + tree = ast.parse(path.read_text()) + for cls in [n for n in tree.body if isinstance(n, ast.ClassDef)]: + for fn in [n for n in cls.body if isinstance(n, ast.FunctionDef)]: + if fn.name != method: + continue + args = fn.args.posonlyargs + fn.args.args + out[path.stem] = [a.arg for a in args][1:] # drop self + return out + + +@pytest.mark.parametrize("method, first", sorted(CANONICAL_FIRST_ARG.items())) +def test_shared_method_first_argument_is_uniform(method, first): + sigs = _signatures(method) + assert len(sigs) > 5, f"expected many implementations of {method}" + wrong = { + mod: params[0] + for mod, params in sigs.items() + if params and params[0] != first + } + assert not wrong, ( + f"{method}'s first argument must be {first!r} everywhere; " + f"these differ: {wrong}" + ) + + +def test_mpp_x_transform_takes_only_x(): + # It used to take a vestigial ``gamma`` on eleven distributions and + # not on the other four. No caller ever passed it -- the MPP fitter + # subtracts the offset from x before calling (fitters/mpp.py) -- so + # a caller that did pass it would have subtracted twice. + sigs = _signatures("mpp_x_transform") + extra = {mod: params for mod, params in sigs.items() if params != ["x"]} + assert not extra, f"mpp_x_transform must take x alone: {extra}" + + +def test_moment_order_is_a_scalar_everywhere(): + # The docstrings used to promise "integer or numpy array of + # integers" on nine distributions; only six delivered, and the + # other three raised. The contract is now a scalar order, which + # every implementation honours and every caller passes. + checked = 0 + for name in dir(surpyval): + dist = getattr(surpyval, name) + if not isinstance(dist, ParametricFitter): + continue + if not hasattr(dist, "moment"): + continue + params = list(inspect.signature(type(dist).moment).parameters) + assert params[1] == "m", f"{name}.moment first arg is {params[1]!r}" + checked += 1 + assert checked > 15, f"only checked {checked} distributions" + + +def test_mpp_override_and_supports_mpp_do_not_contradict(): + # ``mpp`` is an opt-in hook: fitters/mpp.py dispatches on + # hasattr(dist, "mpp") and otherwise runs the generic probability + # plotting path. Independently, fit() refuses how="MPP" outright + # when supports_mpp is False. + # + # A distribution that sets supports_mpp = False *and* defines mpp is + # carrying dead code -- the guard in fit() raises before dispatch can + # reach the method. Beta and Beta4 each had one whose whole body was + # `raise NotImplementedError`, which read as the thing doing the + # refusing when it could never run. + # + # The converse is not an error: a distribution can support MPP and + # use the generic path, which is what most of them do. + contradictory = [] + for name in dir(surpyval): + dist = getattr(surpyval, name) + if not isinstance(dist, ParametricFitter): + continue + defines = any("mpp" in k.__dict__ for k in type(dist).__mro__) + if defines and not dist.supports_mpp: + contradictory.append(name) + assert not contradictory, ( + "these declare supports_mpp = False but still define mpp, which " + f"can never run: {contradictory}" + ) + + +def test_every_distribution_refusing_mpp_raises_the_same_way(): + # One refusal, one message, one place -- rather than each + # distribution inventing its own NotImplementedError. + # + # Restricted to distributions whose fit() takes a ``how`` at all. + # Bernoulli, Binomial and ExactEventTime override fit() with a + # narrow signature that has no ``how``, so asking them for MPP is a + # TypeError from argument binding rather than this refusal. That is + # the separate fit() divergence, not this one. + refusers = [ + name + for name in dir(surpyval) + if isinstance(getattr(surpyval, name), ParametricFitter) + and not getattr(surpyval, name).supports_mpp + and "how" + in inspect.signature(type(getattr(surpyval, name)).fit).parameters + ] + assert len(refusers) > 3, f"expected several refusers, got {refusers}" + for name in refusers: + dist = getattr(surpyval, name) + with pytest.raises(ValueError, match="does not work"): + dist.fit(np.array([1.0, 2.0, 3.0, 4.0]), how="MPP") + + +def _param_names_by_module(): + """{module stem: set of that distribution's parameter names}.""" + out = defaultdict(set) + for name in dir(surpyval): + dist = getattr(surpyval, name) + if isinstance(dist, ParametricFitter): + stem = type(dist).__module__.rsplit(".", 1)[-1] + out[stem].update(getattr(dist, "param_names", []) or []) + return out + + +def test_no_shared_method_diverges_in_its_data_argument(): + # A guard for methods not yet in CANONICAL_FIRST_ARG: any method + # implemented by five or more distributions must agree on the name + # of its leading *data* argument. Catches the next occurrence of + # this bug without needing the method listed above. + # + # Parameter names are skipped, because those are the one thing that + # is legitimately per-distribution: ``Weibull.mean(alpha, beta)`` + # and ``Poisson.mean(mu)`` are not a divergence, they are what the + # distributions are. What must agree is everything else -- the x a + # function is evaluated at, the u a quantile is taken at, the m of a + # moment. + params_by_mod = _param_names_by_module() + leading = defaultdict(dict) + for path in sorted(DIST_DIR.glob("*.py")): + if path.stem == "__init__": + continue + own = params_by_mod.get(path.stem, set()) + tree = ast.parse(path.read_text()) + for cls in [n for n in tree.body if isinstance(n, ast.ClassDef)]: + for fn in [n for n in cls.body if isinstance(n, ast.FunctionDef)]: + if fn.name.startswith("__"): + # Constructors: the wrappers (Discretize, + # CustomDistribution) take a distribution rather + # than a name, which is the point of them. + continue + args = fn.args.posonlyargs + fn.args.args + names = [a.arg for a in args][1:] + data = [n for n in names if n not in own] + if data: + leading[fn.name][path.stem] = data[0] + + diverging = {} + for method, by_mod in leading.items(): + if len(by_mod) < 5: + continue + firsts = set(by_mod.values()) + if len(firsts) > 1: + diverging[method] = { + first: sorted(m for m, f in by_mod.items() if f == first) + for first in sorted(firsts) + } + + assert not diverging, ( + "these shared methods disagree on their leading data argument: " + f"{diverging}" + ) + + +# --------------------------------------------------------------------------- +# Type conventions, now that every distribution is annotated +# --------------------------------------------------------------------------- + +# ``degenerate`` holds InstantlyOccurs and NeverOccurs, which inherit +# ``Distribution`` rather than ``ParametricFitter``. Their signatures are +# dictated by that supertype, so they are not part of these conventions. +_NOT_PARAMETRIC_FITTERS = {"degenerate"} + + +def _annotations(method): + """{module stem: (arg annotations, return annotation)}.""" + out = {} + for path in sorted(DIST_DIR.glob("*.py")): + if path.stem in _NOT_PARAMETRIC_FITTERS or path.stem == "__init__": + continue + tree = ast.parse(path.read_text()) + for cls in [n for n in tree.body if isinstance(n, ast.ClassDef)]: + for fn in [n for n in cls.body if isinstance(n, ast.FunctionDef)]: + if fn.name != method: + continue + args = [ + ( + a.arg, + ast.unparse(a.annotation) if a.annotation else None, + ) + for a in fn.args.posonlyargs + fn.args.args + if a.arg not in ("self", "cls") + ] + ret = ast.unparse(fn.returns) if fn.returns else None + out[path.stem] = (args, ret) + return out + + +@pytest.mark.parametrize( + "method", ["sf", "ff", "df", "hf", "Hf", "log_sf", "log_ff", "log_df"] +) +def test_distribution_functions_take_numeric_and_return_boxable(method): + # ``x`` is what the function is evaluated at -- always real data, so + # ``Numeric``. The return may be an autograd box, because a maximum + # likelihood fit differentiates these, so ``Boxable``. The one + # exception is ExactEventTime, whose step functions are built with + # np.atleast_1d and provably return a real array; a narrower return + # is a stronger promise, not a broken one. + wrong_x, wrong_ret = {}, {} + for mod, (args, ret) in _annotations(method).items(): + if args and args[0][0] == "x" and args[0][1] != "Numeric": + wrong_x[mod] = args[0][1] + if ret not in ("Boxable", "npt.NDArray"): + wrong_ret[mod] = ret + assert not wrong_x, f"{method}'s x must be Numeric: {wrong_x}" + assert not wrong_ret, f"{method} must return Boxable: {wrong_ret}" + + +def test_distribution_parameters_are_boxable(): + # A parameter can be an autograd box while a fit differentiates the + # likelihood through it. Anything narrower is false; ``Any`` would + # make the position uncheckable, which is the whole point of naming + # the box in the first place (see the Numeric/Boxable comment in + # parametric_fitter). + params_by_mod = _param_names_by_module() + wrong = {} + for method in ("sf", "ff", "df", "hf", "Hf", "qf"): + for mod, (args, _) in _annotations(method).items(): + own = params_by_mod.get(mod, set()) + for name, ann in args: + if name in own and ann != "Boxable": + wrong[f"{mod}.{method}({name})"] = ann + assert not wrong, f"distribution parameters must be Boxable: {wrong}" + + +@pytest.mark.parametrize( + "method", ["mpp_x_transform", "mpp_y_transform", "mpp_inv_y_transform"] +) +def test_mpp_transforms_take_arrays(method): + # These act on plotting positions, which are always real arrays: + # every call site in the package passes one, and eight of the + # fifteen implementations index their argument. Probability plotting + # is a regression on those positions and is never differentiated, so + # the input is never an autograd box and never a scalar. + wrong = {} + for mod, (args, _) in _annotations(method).items(): + if args and args[0][1] != "npt.NDArray": + wrong[mod] = f"{args[0][0]}: {args[0][1]}" + assert not wrong, f"{method} must take an npt.NDArray: {wrong}" + + +def test_random_returns_an_array(): + wrong = { + mod: ret + for mod, (_, ret) in _annotations("random").items() + if ret != "npt.NDArray" + } + assert not wrong, f"random must return npt.NDArray: {wrong}" + + +def test_user_entry_points_accept_array_likes(): + # ``fit`` and ``from_params`` take whatever a user has: a scalar, a + # list or an array. ``Numeric`` and ``Boxable`` both exclude ``list`` + # -- and every one of these accepts a list, as their own docstring + # examples show (``Binomial.from_params([5, 0.3])``). + wrong = {} + for mod, (args, _) in _annotations("fit").items(): + if args and args[0][0] == "x" and args[0][1] != "npt.ArrayLike": + wrong[f"{mod}.fit(x)"] = args[0][1] + for mod, (args, _) in _annotations("from_params").items(): + if args and args[0][0] == "params" and args[0][1] != "npt.ArrayLike": + wrong[f"{mod}.from_params(params)"] = args[0][1] + assert not wrong, f"must accept an array-like: {wrong}" diff --git a/surpyval/univariate/parametric/discrete_fitter.py b/surpyval/univariate/parametric/discrete_fitter.py index 81c9657..c2adea5 100644 --- a/surpyval/univariate/parametric/discrete_fitter.py +++ b/surpyval/univariate/parametric/discrete_fitter.py @@ -1,3 +1,7 @@ +from typing import Any + +import numpy.typing as npt + """Base class for the discrete lifetime distributions. The discrete/continuous distinction used to live in scattered per-class @@ -12,6 +16,8 @@ is ``P(T > k)``. """ +from surpyval import np + from .parametric_fitter import ParametricFitter @@ -35,8 +41,31 @@ class DiscreteParametricFitter(ParametricFitter): discrete = True - def __init__(self, *args, **kwargs): + def __init__(self, *args: Any, **kwargs: Any) -> None: super().__init__(*args, **kwargs) # Kept as an instance attribute (not only the class trait) because # the shared ``_validate_fit_inputs`` reads it for every fitter. self.supports_mpp = False + + def log_df(self, x: npt.NDArray, *params: Any) -> Any: + # On the integers the mass at k is the hazard there times the + # survival to just before it: + # + # P(T = k) = h(k) R(k - 1) + # + # ``ParametricFitter.log_df`` encodes the continuous identity + # f = h R(x) instead, which is the same statement with R(k) in + # place of R(k - 1) and so is wrong by a factor R(k)/R(k - 1) -- + # not a rounding difference. Binomial reached the continuous + # version and returned values that drifted from 0.88 of the true + # mass down to 0.15 across k = 1..7. + # + # Most discrete distributions here override this with the + # closed-form log-pmf, which is better conditioned than any + # identity assembled from hf and sf. This is the fallback for + # those that do not. + x_arr = np.asarray(x, dtype=float) + # hf and log_sf come from the concrete distribution; the + # base declares them for typing on OptimisedFitMixin only. + hf = self.hf(x_arr, *params) # type: ignore[attr-defined] + return np.log(hf) + self.log_sf(x_arr - 1.0, *params) diff --git a/surpyval/univariate/parametric/distributions/__init__.py b/surpyval/univariate/parametric/distributions/__init__.py index 10184c5..c9d79d4 100644 --- a/surpyval/univariate/parametric/distributions/__init__.py +++ b/surpyval/univariate/parametric/distributions/__init__.py @@ -1,4 +1,5 @@ -from .bernoulli import Bernoulli, FixedEventProbability +from .bernoulli import Bernoulli +from .fixed_event_probability import FixedEventProbability from .beta import Beta from .beta4 import Beta4 from .beta_geometric import BetaGeometric diff --git a/surpyval/univariate/parametric/distributions/bernoulli.py b/surpyval/univariate/parametric/distributions/bernoulli.py index a11e8dc..aa636a5 100644 --- a/surpyval/univariate/parametric/distributions/bernoulli.py +++ b/surpyval/univariate/parametric/distributions/bernoulli.py @@ -15,6 +15,42 @@ class Bernoulli_(DiscreteParametricFitter): + r"""A single weighted coin flip: ``X`` is 0 or 1 with ``P(X = 1) = p``. + + ``x`` is the outcome, not a time, so 0 and 1 are the only values any + of these functions accept and anything else raises. The survival + function is :math:`R(x) = P(X \geq x)`, giving ``R(0) = 1`` and + ``R(1) = p``: read as a one-shot device, ``p`` is the probability it + works when demanded. + + .. note:: + ``p`` is the probability of the ``1`` outcome. Before 0.19.1 this + distribution had ``F(x) = p`` at every ``x`` -- a flat curve with + no time axis, where ``p`` was documented as the probability of + *failure*. The parameter has therefore changed direction: code + that coded failures as 1 now fits the survival probability, and + wants ``1 - p``. The flat model itself is unchanged and still + available as :data:`FixedEventProbability`. + + Note also that this is not ``Binomial`` with ``n = 1`` evaluated at + the same points. Binomial follows the package's discrete convention + :math:`R(k) = P(X > k)`; this one uses :math:`P(X \geq x)`, so the + two are offset by one: ``Bernoulli.sf(x, p) == Binomial.sf(x - 1, + 1, p)``. + """ + + @staticmethod + def _check_x(x: Numeric) -> npt.NDArray: + """Reject anything that is not a Bernoulli outcome.""" + x_arr = np.atleast_1d(np.asarray(x, dtype=float)) + if not np.isin(x_arr, (0.0, 1.0)).all(): + raise ValueError( + "Bernoulli is defined at x = 0 and x = 1 only; x is the " + "outcome of the flip, not a time. For a model whose event " + "probability is p at every x, use FixedEventProbability." + ) + return x_arr + def __init__(self, name: str) -> None: super().__init__( name=name, @@ -29,51 +65,59 @@ def __init__(self, name: str) -> None: def sf(self, x: Numeric, p: Boxable) -> Boxable: r""" - Survival (or reliability) function for the Bernoulli Distribution: + Survival function for the Bernoulli Distribution: .. math:: - R(x) = 1 - p + R(x) = P(X \geq x) + + which is 1 at ``x = 0`` and ``p`` at ``x = 1``. Parameters ---------- x : numpy array or scalar - The values at which the function will be calculated + The outcome(s) at which the function will be calculated. + Must be 0 or 1. p : float - The probability of failure of the thing + The probability of the ``1`` outcome Returns ------- sf : scalar or numpy array - The value(s) of the reliability function at x. Which for this - distribution is constant + The value(s) of the survival function at x. Examples -------- >>> import numpy as np >>> from surpyval import Bernoulli - >>> x = np.array([1, 2, 3, 4, 5]) - >>> Bernoulli.sf(x, 0.5) - array([0.5, 0.5, 0.5, 0.5, 0.5]) + >>> Bernoulli.sf(np.array([0, 1]), 0.3) + array([1. , 0.3]) """ - return 1.0 - self.ff(x, p) + x_arr = self._check_x(x) + return np.where(x_arr == 0.0, 1.0, p) def ff(self, x: Numeric, p: Boxable) -> Boxable: r""" - Failure (CDF or unreliability) function for the Bernoulli Distribution: + Failure (CDF) function for the Bernoulli Distribution: .. math:: - F(x) = p + F(x) = P(X < x) + + which is 0 at ``x = 0`` and ``1 - p`` at ``x = 1``. This is + ``P(X < x)`` rather than the more usual ``P(X \leq x)`` because + the package's survival and failure functions sum to one, and + ``R(x)`` here is ``P(X \geq x)``. Parameters ---------- x : numpy array or scalar - The values at which the function will be calculated + The outcome(s) at which the function will be calculated. + Must be 0 or 1. p : float - The probability of failure of the thing + The probability of the ``1`` outcome Returns ------- @@ -85,27 +129,224 @@ def ff(self, x: Numeric, p: Boxable) -> Boxable: -------- >>> import numpy as np >>> from surpyval import Bernoulli - >>> x = np.array([1, 2, 3, 4, 5]) - >>> Bernoulli.ff(x, 0.5) - array([0.5, 0.5, 0.5, 0.5, 0.5]) + >>> Bernoulli.ff(np.array([0, 1]), 0.3) + array([0. , 0.7]) + """ + return 1.0 - self.sf(x, p) + + def df(self, x: Numeric, p: Boxable) -> Boxable: + r""" + + Probability mass function for the Bernoulli Distribution: + + .. math:: + f(0) = 1 - p, \quad f(1) = p + + Parameters + ---------- + + x : numpy array or scalar + The outcome(s) at which the function will be calculated. + Must be 0 or 1. + p : float + The probability of the ``1`` outcome + + Returns + ------- + + df : scalar or numpy array + The probability of the outcome(s) in x. + + Examples + -------- + >>> import numpy as np + >>> from surpyval import Bernoulli + >>> Bernoulli.df(np.array([0, 1]), 0.3) + array([0.7, 0.3]) + """ + x_arr = self._check_x(x) + return np.where(x_arr == 0.0, 1.0 - p, p) + + def hf(self, x: Numeric, p: Boxable) -> Boxable: + r""" + + Hazard rate for the Bernoulli Distribution: + + .. math:: + h(x) = \frac{f(x)}{P(X \geq x)} + + which is ``1 - p`` at ``x = 0`` and 1 at ``x = 1``: everything + still at risk at the last outcome fails there. + + Parameters + ---------- + + x : numpy array or scalar + The outcome(s) at which the function will be calculated. + Must be 0 or 1. + p : float + The probability of the ``1`` outcome + + Returns + ------- + + hf : scalar or numpy array + The value(s) of the hazard rate at x. + + Examples + -------- + >>> import numpy as np + >>> from surpyval import Bernoulli + >>> Bernoulli.hf(np.array([0, 1]), 0.3) + array([0.7, 1. ]) + """ + x_arr = self._check_x(x) + return np.where(x_arr == 0.0, 1.0 - p, 1.0) + + def Hf(self, x: Numeric, p: Boxable) -> Boxable: + r""" + + Cumulative hazard rate for the Bernoulli Distribution: + + .. math:: + H(x) = -\ln R(x) + + which is 0 at ``x = 0`` and :math:`-\ln p` at ``x = 1``. + + Parameters + ---------- + + x : numpy array or scalar + The outcome(s) at which the function will be calculated. + Must be 0 or 1. + p : float + The probability of the ``1`` outcome + + Returns + ------- + + Hf : scalar or numpy array + The value(s) of the cumulative hazard rate at x. + + Examples + -------- + >>> import numpy as np + >>> from surpyval import Bernoulli + >>> Bernoulli.Hf(np.array([0, 1]), 0.3) + array([0. , 1.2039728]) """ - return np.ones_like(x).astype(float) * p + x_arr = self._check_x(x) + return np.where(x_arr == 0.0, 0.0, -np.log(p)) - def moment(self, n: int, p: Boxable) -> Boxable: + def qf(self, u: Numeric, p: Boxable) -> Boxable: r""" - n-th moment of the Bernoulli distribution + Quantile function for the Bernoulli Distribution: .. math:: - M(n) = p + q(u) = \begin{cases} + 0 & u \leq 1 - p \\ + 1 & u > 1 - p + \end{cases} + + This inverts :math:`P(X \leq x)`, the ordinary CDF, which is the + standard quantile and the one that makes inverse-transform + sampling work: ``qf(U)`` for uniform ``U`` is 1 with probability + ``p``. On the open interval it agrees exactly with + ``Binomial.qf(u, 1, p)`` and with ``scipy.stats.binom.ppf``. At + ``u = 0`` those return ``-1``, one below the support, where this + returns 0 -- the smallest outcome there is. + + .. note:: + It is *not* the inverse of this class's ``ff``. That is a + consequence of the survival convention rather than an + oversight: ``R(x) = P(X \geq x)`` forces ``F(x) = P(X < x)`` + if the two are to sum to one, and ``P(X < x)`` never exceeds + ``1 - p`` anywhere on ``{0, 1}`` -- so no ``x`` in the support + satisfies ``F(x) >= u`` once ``u`` passes ``1 - p``. The other + discrete distributions, whose ``R(k)`` is ``P(X > k)``, do not + have this split. Parameters ---------- - n : integer or numpy array of integers + u : numpy array or scalar + The probability or probabilities at which the quantile will + be calculated + p : float + The probability of the ``1`` outcome + + Returns + ------- + + qf : scalar or numpy array + The outcome(s) at the given probabilities. + + Examples + -------- + >>> import numpy as np + >>> from surpyval import Bernoulli + >>> Bernoulli.qf(np.array([0.1, 0.7, 0.75, 0.99]), 0.3) + array([0., 0., 1., 1.]) + """ + u_arr = np.asarray(u, dtype=float) + return np.where(u_arr <= 1.0 - p, 0.0, 1.0) + + def log_df(self, x: Numeric, p: Boxable) -> Boxable: + # Neither inherited relation fits. DiscreteParametricFitter uses + # f(k) = h(k) R(k - 1), which assumes R(k) = P(X > k); here R is + # P(X >= x), so the at-risk set at x is R(x) itself and the + # mass is f(x) = h(x) R(x) -- the continuous form. Taking the + # log of the pmf directly sidesteps the choice. + x_arr = self._check_x(x) + return np.where(x_arr == 0.0, np.log1p(-p), np.log(p)) + + def mean(self, p: Boxable) -> Boxable: + r""" + + Mean of the Bernoulli distribution: + + .. math:: + E = p + + Parameters + ---------- + + p : float + The probability of the ``1`` outcome + + Returns + ------- + + mean : scalar or numpy array + The mean of the Bernoulli distribution + + Examples + -------- + >>> from surpyval import Bernoulli + >>> Bernoulli.mean(0.3) + 0.3 + """ + return p + + def moment(self, m: int, p: Boxable) -> Boxable: + r""" + + m-th moment of the Bernoulli distribution + + .. math:: + E[X^{m}] = p + + The same for every ``m``, because ``X`` is 0 or 1 and so + ``X**m == X``. + + Parameters + ---------- + + m : integer The ordinal of the moment to calculate p : float - The probability of failure of the thing + The probability of the ``1`` outcome Returns ------- @@ -135,7 +376,7 @@ def random(self, size: int | tuple[int, ...], p: Boxable) -> npt.NDArray: size : integer or tuple of positive integers Shape or size of the random draw p : float - The probability of failure of the thing + The probability of the ``1`` outcome Returns ------- @@ -147,7 +388,9 @@ def random(self, size: int | tuple[int, ...], p: Boxable) -> npt.NDArray: U = uniform.rvs(size=size) return (U <= p).astype(int) - def fit(self, x: Numeric, n: npt.NDArray | None = None) -> Parametric: + def fit( + self, x: npt.ArrayLike, n: npt.NDArray | None = None + ) -> Parametric: x_arr = np.atleast_1d(x) # Each observation must be a 0 or a 1 — elementwise, for any length # (the previous check broadcast x against the literal [0, 1], so any @@ -174,7 +417,7 @@ def fit(self, x: Numeric, n: npt.NDArray | None = None) -> Parametric: # back, with a deprecation alias, and is tracked separately. def from_params( self, - params: Boxable, + params: npt.ArrayLike, gamma: Boxable | None = None, p: Boxable | None = None, f0: Boxable | None = None, @@ -207,5 +450,4 @@ def from_params( return model -Bernoulli = Bernoulli_("Bernoulli") -FixedEventProbability = Bernoulli_("FixedEventProbability") +Bernoulli: Bernoulli_ = Bernoulli_("Bernoulli") diff --git a/surpyval/univariate/parametric/distributions/beta.py b/surpyval/univariate/parametric/distributions/beta.py index bdd1d58..476307e 100755 --- a/surpyval/univariate/parametric/distributions/beta.py +++ b/surpyval/univariate/parametric/distributions/beta.py @@ -1,18 +1,22 @@ +import numpy.typing as npt from autograd.scipy.special import beta as abeta from autograd.scipy.special import betaln as abetaln from scipy.special import betaincinv, digamma from surpyval import np from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) from surpyval.utils.autograd_gamma_compat import betainc as abetainc from surpyval.utils.autograd_gamma_compat import betaincln as abetaincln +from surpyval.utils.surpyval_data import SurpyvalData class Beta_(OptimisedFitMixin, ParametricFitter): - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=2, @@ -30,15 +34,17 @@ def __init__(self, name): # or MOM instead (MOM is analytic for the Beta). self.supports_mpp = False - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): - if (c is not None) and (c == 0).all(): - x = np.repeat(x, n) + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + if (data.c == 0).all(): + x = np.repeat(data.x, data.n) p = self._mom(x) else: p = 1.0, 1.0 - return p + return np.asarray(p, dtype=float) - def sf(self, x, alpha, beta): + def sf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Survival (or reliability) function for the Beta Distribution: @@ -73,44 +79,7 @@ def sf(self, x, alpha, beta): """ return 1 - self.ff(x, alpha, beta) - def cs(self, x, X, alpha, beta): - r""" - - Conditional survival (or reliability) function for the Beta - Distribution: - - .. math:: - R(x, X) = \frac{R(x + X)}{R(X)} - - Parameters - ---------- - - x : numpy array or scalar - The values at which the function will be calculated - X : numpy array or scalar - The value(s) at which each value(s) in x was known to have survived - alpha : numpy array or scalar - One shape parameter for the Beta distribution - beta : numpy array or scalar - The scale parameter for the Beta distribution - - Returns - ------- - - cs : scalar or numpy array - The value(s) of the conditional survival function at x. - - Examples - -------- - >>> import numpy as np - >>> from surpyval import Beta - >>> x = np.array([.1, .2, .3, .4, .5]) - >>> Beta.cs(x, 0.4, 3, 4) - array([0.6315219 , 0.32921811, 0.12946429, 0.03115814, 0.00233319]) - """ - return self.sf(x + X, alpha, beta) / self.sf(X, alpha, beta) - - def ff(self, x, alpha, beta): + def ff(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Failure (CDF or unreliability) function for the Beta Distribution: @@ -144,7 +113,7 @@ def ff(self, x, alpha, beta): """ return abetainc(alpha, beta, x) - def df(self, x, alpha, beta): + def df(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Density function for the Beta Distribution: @@ -179,7 +148,7 @@ def df(self, x, alpha, beta): """ return (x ** (alpha - 1) * (1 - x) ** (beta - 1)) / abeta(alpha, beta) - def hf(self, x, alpha, beta): + def hf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Instantaneous hazard rate for the Beta distribution. @@ -213,7 +182,7 @@ def hf(self, x, alpha, beta): """ return self.df(x, alpha, beta) / self.sf(x, alpha, beta) - def Hf(self, x, alpha, beta): + def Hf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Cumulative hazard rate for the Beta distribution. @@ -247,7 +216,7 @@ def Hf(self, x, alpha, beta): """ return -np.log(self.sf(x, alpha, beta)) - def qf(self, p, alpha, beta): + def qf(self, u: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Quantile function for the Beta Distribution: @@ -255,7 +224,7 @@ def qf(self, p, alpha, beta): Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated alpha : numpy array or scalar One shape parameter for the Beta distribution @@ -266,19 +235,19 @@ def qf(self, p, alpha, beta): ------- q : scalar or numpy array - The quantiles for the Beta distribution at each value p. + The quantiles for the Beta distribution at each value u. Examples -------- >>> import numpy as np >>> from surpyval import Beta - >>> p = np.array([.1, .2, .3, .4, .5]) - >>> Beta.qf(p, 3, 4) + >>> u = np.array([.1, .2, .3, .4, .5]) + >>> Beta.qf(u, 3, 4) array([0.20090888, 0.26864915, 0.32332388, 0.37307973, 0.42140719]) """ - return betaincinv(alpha, beta, p) + return betaincinv(alpha, beta, u) - def mean(self, alpha, beta): + def mean(self, alpha: Boxable, beta: Boxable) -> Boxable: r""" Mean of the Beta distribution @@ -308,19 +277,19 @@ def mean(self, alpha, beta): """ return alpha / (alpha + beta) - def moment(self, n, alpha, beta): + def moment(self, m: int, alpha: Boxable, beta: Boxable) -> Boxable: r""" - n-th (non central) moment of the Beta distribution + m-th (non central) moment of the Beta distribution .. math:: - E = \frac{B \left( n + \alpha, \beta \right )}{B + E = \frac{B \left( m + \alpha, \beta \right )}{B \left ( \alpha, \beta \right )} Parameters ---------- - n : integer or numpy array of integers + m : integer The ordinal of the moment to calculate alpha : numpy array or scalar One shape parameter for the Beta distribution @@ -339,9 +308,9 @@ def moment(self, n, alpha, beta): >>> Beta.moment(2, 3, 4) np.float64(0.2142857142857143) """ - return np.exp(abetaln(n + alpha, beta) - abetaln(alpha, beta)) + return np.exp(abetaln(m + alpha, beta) - abetaln(alpha, beta)) - def entropy(self, alpha, beta): + def entropy(self, alpha: Boxable, beta: Boxable) -> Boxable: r""" Calculates the entropy of the Beta distribution. @@ -382,42 +351,28 @@ def entropy(self, alpha, beta): + (alpha + beta - 2) * digamma(alpha + beta) ) - def log_df(self, x, alpha, beta): + def log_df(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: return ( (alpha - 1) * np.log(x) + (beta - 1) * np.log1p(-x) - abetaln(alpha, beta) ) - def log_ff(self, x, alpha, beta): + def log_ff(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: return abetaincln(alpha, beta, x) - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return self.qf(y, *params) - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: # The inverse of the quantile transform is the CDF; the point must # be the *last* argument of betainc, not the first shape (#257). return abetainc(*params, y) - def mpp_x_transform(self, x, gamma=0): - return x - gamma - - def mpp( - self, - x, - c=None, - n=None, - t=None, - heuristic="Nelson-Aalen", - rr="y", - on_d_is_0=False, - offset=False, - ): - msg = "Probability Plotting Method for Beta distribution" - raise NotImplementedError(msg) - - def _mom(self, x): + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: + return x + + def _mom(self, x: npt.NDArray) -> tuple[float, float]: """ MOM: Method of Moments for the beta distribution has an analytic answer """ @@ -441,7 +396,9 @@ def _mom(self, x): return alpha, beta - def _plot_x_bounds(self, x, params): + def _plot_x_bounds( + self, x: npt.NDArray, params: npt.NDArray + ) -> tuple[float, float] | None: return 0.0, 1.0 diff --git a/surpyval/univariate/parametric/distributions/beta4.py b/surpyval/univariate/parametric/distributions/beta4.py index c1c88d4..3f89dcb 100644 --- a/surpyval/univariate/parametric/distributions/beta4.py +++ b/surpyval/univariate/parametric/distributions/beta4.py @@ -1,14 +1,18 @@ +import numpy.typing as npt from autograd.scipy.special import beta as abeta from autograd.scipy.special import betaln as abetaln from scipy.special import betaincinv, comb, digamma from surpyval import np from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) from surpyval.utils.autograd_gamma_compat import betainc as abetainc from surpyval.utils.autograd_gamma_compat import betaincln as abetaincln +from surpyval.utils.surpyval_data import SurpyvalData class Beta4_(OptimisedFitMixin, ParametricFitter): @@ -29,7 +33,7 @@ class Beta4_(OptimisedFitMixin, ParametricFitter): lower bound while keeping the upper bound pinned at 1. """ - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=4, @@ -46,10 +50,12 @@ def __init__(self, name): # ``a`` and ``b`` supply the left and right support bounds. self.support_param_index = (2, 3) - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): - x = np.asarray(x, dtype=float) - if (n is not None) and (c is not None) and (c == 0).all(): - x = np.repeat(x, n) + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + x = np.asarray(data.x, dtype=float) + if (data.c == 0).all(): + x = np.repeat(x, data.n) span = x.max() - x.min() if span <= 0: @@ -69,13 +75,15 @@ def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): alpha = max(term1 * mean, 0.5) beta = max(term1 * (1 - mean), 0.5) - return alpha, beta, a, b + return np.array([alpha, beta, a, b], dtype=float) - def _z(self, x, a, b): + def _z(self, x: Numeric, a: Boxable, b: Boxable) -> Boxable: """Standardise ``x`` onto the unit interval.""" return (x - a) / (b - a) - def sf(self, x, alpha, beta, a, b): + def sf( + self, x: Numeric, alpha: Boxable, beta: Boxable, a: Boxable, b: Boxable + ) -> Boxable: r""" Survival (or reliability) function for the four-parameter Beta @@ -115,42 +123,9 @@ def sf(self, x, alpha, beta, a, b): """ return 1 - self.ff(x, alpha, beta, a, b) - def cs(self, x, X, alpha, beta, a, b): - r""" - - Conditional survival (or reliability) function for the - four-parameter Beta distribution: - - .. math:: - R(x, X) = \frac{R(x + X)}{R(X)} - - Parameters - ---------- - - x : numpy array or scalar - The values at which the function will be calculated - X : numpy array or scalar - The value(s) at which each value(s) in x was known to have survived - alpha : numpy array or scalar - The first shape parameter for the Beta distribution - beta : numpy array or scalar - The second shape parameter for the Beta distribution - a : numpy array or scalar - The lower bound of the support - b : numpy array or scalar - The upper bound of the support - - Returns - ------- - - cs : scalar or numpy array - The value(s) of the conditional survival function at x. - """ - return self.sf(x + X, alpha, beta, a, b) / self.sf( - X, alpha, beta, a, b - ) - - def ff(self, x, alpha, beta, a, b): + def ff( + self, x: Numeric, alpha: Boxable, beta: Boxable, a: Boxable, b: Boxable + ) -> Boxable: r""" Failure (CDF or unreliability) function for the four-parameter @@ -191,7 +166,9 @@ def ff(self, x, alpha, beta, a, b): z = np.clip(self._z(x, a, b), 0.0, 1.0) return abetainc(alpha, beta, z) - def df(self, x, alpha, beta, a, b): + def df( + self, x: Numeric, alpha: Boxable, beta: Boxable, a: Boxable, b: Boxable + ) -> Boxable: r""" Density function for the four-parameter Beta distribution: @@ -240,7 +217,9 @@ def df(self, x, alpha, beta, a, b): den = abeta(alpha, beta) * (b - a) ** (alpha + beta - 1) return np.where(inside, num / den, 0.0) - def hf(self, x, alpha, beta, a, b): + def hf( + self, x: Numeric, alpha: Boxable, beta: Boxable, a: Boxable, b: Boxable + ) -> Boxable: r""" Instantaneous hazard rate for the four-parameter Beta @@ -280,7 +259,9 @@ def hf(self, x, alpha, beta, a, b): out = np.where(x_arr < a, 0.0, out) return np.where(x_arr >= b, np.inf, out) - def Hf(self, x, alpha, beta, a, b): + def Hf( + self, x: Numeric, alpha: Boxable, beta: Boxable, a: Boxable, b: Boxable + ) -> Boxable: r""" Cumulative hazard rate for the four-parameter Beta distribution. @@ -310,18 +291,20 @@ def Hf(self, x, alpha, beta, a, b): """ return -np.log(self.sf(x, alpha, beta, a, b)) - def qf(self, p, alpha, beta, a, b): + def qf( + self, u: Numeric, alpha: Boxable, beta: Boxable, a: Boxable, b: Boxable + ) -> Boxable: r""" Quantile function for the four-parameter Beta distribution: .. math:: - q(p) = a + \left(b - a\right) I^{-1}_{p}\left(\alpha, \beta\right) + q(u) = a + \left(b - a\right) I^{-1}_{u}\left(\alpha, \beta\right) Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated alpha : numpy array or scalar The first shape parameter for the Beta distribution @@ -336,19 +319,21 @@ def qf(self, p, alpha, beta, a, b): ------- q : scalar or numpy array - The quantiles for the Beta distribution at each value p. + The quantiles for the Beta distribution at each value u. Examples -------- >>> import numpy as np >>> from surpyval import Beta4 - >>> p = np.array([.1, .2, .3, .4, .5]) - >>> Beta4.qf(p, 3, 4, 2, 3) + >>> u = np.array([.1, .2, .3, .4, .5]) + >>> Beta4.qf(u, 3, 4, 2, 3) array([2.20090888, 2.26864915, 2.32332388, 2.37307973, 2.42140719]) """ - return a + (b - a) * betaincinv(alpha, beta, p) + return a + (b - a) * betaincinv(alpha, beta, u) - def mean(self, alpha, beta, a, b): + def mean( + self, alpha: Boxable, beta: Boxable, a: Boxable, b: Boxable + ) -> Boxable: r""" Mean of the four-parameter Beta distribution @@ -382,7 +367,9 @@ def mean(self, alpha, beta, a, b): """ return a + (b - a) * alpha / (alpha + beta) - def moment(self, m, alpha, beta, a, b): + def moment( + self, m: int, alpha: Boxable, beta: Boxable, a: Boxable, b: Boxable + ) -> Boxable: r""" m-th (non central) moment of the four-parameter Beta distribution. @@ -424,7 +411,9 @@ def moment(self, m, alpha, beta, a, b): total = total + comb(m, k) * a ** (m - k) * scale**k * u_moment return total - def entropy(self, alpha, beta, a, b): + def entropy( + self, alpha: Boxable, beta: Boxable, a: Boxable, b: Boxable + ) -> Boxable: r""" Differential entropy of the four-parameter Beta distribution. @@ -458,7 +447,9 @@ def entropy(self, alpha, beta, a, b): ) return standard + np.log(b - a) - def log_df(self, x, alpha, beta, a, b): + def log_df( + self, x: Numeric, alpha: Boxable, beta: Boxable, a: Boxable, b: Boxable + ) -> Boxable: return ( (alpha - 1) * np.log(x - a) + (beta - 1) * np.log(b - x) @@ -466,24 +457,24 @@ def log_df(self, x, alpha, beta, a, b): - (alpha + beta - 1) * np.log(b - a) ) - def log_ff(self, x, alpha, beta, a, b): + def log_ff( + self, x: Numeric, alpha: Boxable, beta: Boxable, a: Boxable, b: Boxable + ) -> Boxable: z = np.clip(self._z(x, a, b), 0.0, 1.0) return abetaincln(alpha, beta, z) - def mpp_x_transform(self, x, gamma=0): - return x - gamma + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: + return x - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return self.qf(y, *params) - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return self.ff(y, *params) - def mpp(self, *args, **kwargs): - msg = "Probability Plotting Method for Beta4 distribution" - raise NotImplementedError(msg) - - def _plot_x_bounds(self, x, params): + def _plot_x_bounds( + self, x: npt.NDArray, params: npt.NDArray + ) -> tuple[float, float] | None: return float(params[2]), float(params[3]) diff --git a/surpyval/univariate/parametric/distributions/beta_geometric.py b/surpyval/univariate/parametric/distributions/beta_geometric.py index 7ce6f31..6c8b8d0 100644 --- a/surpyval/univariate/parametric/distributions/beta_geometric.py +++ b/surpyval/univariate/parametric/distributions/beta_geometric.py @@ -12,6 +12,7 @@ Numeric, OptimisedFitMixin, ) +from surpyval.utils.surpyval_data import SurpyvalData class BetaGeometric_(OptimisedFitMixin, DiscreteParametricFitter): @@ -53,12 +54,7 @@ def __init__(self, name: str) -> None: ) def _parameter_initialiser( - self, - x: npt.NDArray, - c: npt.NDArray | None = None, - n: npt.NDArray | None = None, - t: npt.NDArray | None = None, - offset: bool = False, + self, data: SurpyvalData, offset: bool = False ) -> npt.NDArray: # A neutral, proper starting point; the Beta(1, 1) mixing is the # uniform prior over p, i.e. a diffuse heterogeneity. @@ -97,7 +93,12 @@ def qf(self, u: Numeric, a: Boxable, b: Boxable) -> Boxable: if ui <= 0.0: out[idx] = 1.0 continue - target = 1.0 - ui + # ``target`` is reached by cancellation -- the caller almost + # always passes u = F(k) = 1 - R(k), and 1 - (1 - R(k)) lands + # one ulp below R(k). A strict comparison then rejects the + # exact answer and returns k + 1, so F and its quantile did + # not invert each other. Compare with a relative slack. + target = (1.0 - ui) * (1.0 + 1e-12) hi = 1 while self.sf(float(hi), a, b) > target and hi < 2**40: hi *= 2 @@ -119,8 +120,19 @@ def mean(self, a: Boxable, b: Boxable) -> Boxable: return (a + b - 1.0) / (a - 1.0) def moment(self, m: int, a: Boxable, b: Boxable) -> Boxable: - # Truncated sum of the pmf; the tail can be heavy, so integrate out to - # a far survival quantile. + # The survival decays as k^-a, so E[T^m] converges only for a > m -- + # the same condition ``mean`` applies at m = 1. Without the test a + # truncated sum reports a finite value for a moment that does not + # exist: at a = 2, b = 3 the second moment is infinite and the old + # sum returned about 25. + if a <= m: + return np.inf + if m == 1: + # Exact, and the reason mean() and moment(1) now agree: the + # truncated sum lost 0.17% of a heavy tail even at the 1 - 1e-6 + # quantile. + return self.mean(a, b) + # No closed form for general m; sum out to a far quantile. upper = int(self.qf(1.0 - 1e-6, a, b)) k = np.arange(1, upper + 1, dtype=float) return np.sum(k**m * self.df(k, a, b)) @@ -135,10 +147,21 @@ def random( return geom.rvs(p).astype(float) def log_sf(self, x: Numeric, a: Boxable, b: Boxable) -> Boxable: - return self._log_beta(a, b + x) - self._log_beta(a, b) + # R(k) = 1 for every k below the first mass point. The Beta-ratio + # form does not know that -- at k = -1 it returns 2.0, a survival + # above one -- so clamp the argument at zero, where it is already 1. + safe_x = np.where(x < 0.0, 0.0, x) + return self._log_beta(a, b + safe_x) - self._log_beta(a, b) def log_df(self, x: Numeric, a: Boxable, b: Boxable) -> Boxable: - return self._log_beta(a + 1.0, b + x - 1.0) - self._log_beta(a, b) + # Zero mass below k = 1. The Beta-ratio form returns 1.0 at k = 0, + # and B(a + 1, b + k - 1) is undefined once b + k - 1 <= 0, so the + # argument is clamped before the guard chooses the branch. + safe_x = np.where(x < 1.0, 1.0, x) + log_df = self._log_beta(a + 1.0, b + safe_x - 1.0) - self._log_beta( + a, b + ) + return np.where(x < 1.0, -np.inf, log_df) BetaGeometric = BetaGeometric_("BetaGeometric") diff --git a/surpyval/univariate/parametric/distributions/binomial.py b/surpyval/univariate/parametric/distributions/binomial.py index 68cc976..9e3927b 100644 --- a/surpyval/univariate/parametric/distributions/binomial.py +++ b/surpyval/univariate/parametric/distributions/binomial.py @@ -22,6 +22,12 @@ class Binomial_(DiscreteParametricFitter): It is the recurrent (repeated-trials) counterpart of the :class:`Bernoulli` distribution, which is the special case ``n = 1``. + The two agree exactly on the probability mass there. Their survival + functions are offset by one, which is a convention rather than a + disagreement: this class follows the package's discrete rule + :math:`R(k) = P(K > k)`, while Bernoulli uses :math:`P(X \geq x)` so + that ``R(0) = 1`` and ``R(1) = p``. Hence + ``Bernoulli.sf(x, p) == Binomial.sf(x - 1, 1, p)``. The distribution is parameterised by ``n`` (the number of trials, a positive integer) and ``p`` (the per-trial event probability). Because @@ -36,7 +42,22 @@ def __init__(self, name: str) -> None: name=name, k=2, bounds=((1, None), (0, 1)), - support=(0, np.inf), + # ``support`` is a pair of *exclusive* bounds: the shared + # ``_validate_fit_inputs`` rejects data with + # ``x <= support[0]`` or ``x >= support[1]``, so a distribution + # declares the bound one step outside its first and last mass + # points. The first mass point here is k = 0 -- zero events in + # n trials is an ordinary outcome, P = 0.168 at n = 5, p = 0.3 + # -- so the lower bound is -1, as for ``Poisson``. It read 0, + # which is ``Geometric``'s value and says zero events lie + # outside the distribution. Nothing observed it because + # ``Binomial`` does not inherit ``OptimisedFitMixin``, where + # that check lives, and validates its own inputs instead. + # + # The upper bound stays infinite here because n is not known + # until the model is built; ``fit`` and ``from_params`` set it + # to n + 1 for the same reason. + support=(-1, np.inf), param_names=["n", "p"], param_map={"n": 0, "p": 1}, plot_x_scale="linear", @@ -195,16 +216,16 @@ def Hf(self, x: Numeric, n: Boxable, p: Boxable) -> Boxable: """ return -np.log(self.sf(x, n, p)) - def qf(self, q: Numeric, n: Boxable, p: Boxable) -> Boxable: + def qf(self, u: Numeric, n: Boxable, p: Boxable) -> Boxable: r""" Quantile (inverse CDF) function for the Binomial distribution; the - smallest number of events ``x`` such that :math:`F(x) \geq q`. + smallest number of events ``x`` such that :math:`F(x) \geq u`. Parameters ---------- - q : numpy array or scalar + u : numpy array or scalar The values, between 0 and 1, at which the quantile is evaluated n : integer The number of trials @@ -215,7 +236,7 @@ def qf(self, q: Numeric, n: Boxable, p: Boxable) -> Boxable: ------- qf : scalar or numpy array - The quantile(s) at q + The quantile(s) at u Examples -------- @@ -223,36 +244,7 @@ def qf(self, q: Numeric, n: Boxable, p: Boxable) -> Boxable: >>> Binomial.qf(0.5, 5, 0.3) np.float64(1.0) """ - return binom.ppf(q, n, p) - - def cs(self, x: Numeric, X: Numeric, n: Boxable, p: Boxable) -> Boxable: - r""" - - Conditional survival; the probability of surviving a further ``x`` - events having already survived ``X``: - - .. math:: - R(x \mid X) = \frac{R(x + X)}{R(X)} - - Parameters - ---------- - - x : numpy array or scalar - The further number of events - X : numpy array or scalar - The number of events already survived - n : integer - The number of trials - p : float - The per-trial probability of an event - - Returns - ------- - - cs : scalar or numpy array - The conditional survival probability - """ - return self.sf(x + X, n, p) / self.sf(X, n, p) + return binom.ppf(u, n, p) def mean(self, n: Boxable, p: Boxable) -> Boxable: r""" @@ -339,7 +331,7 @@ def random( def fit( self, - x: Numeric, + x: npt.ArrayLike, n_trials: int, c: npt.NDArray | None = None, n: npt.NDArray | None = None, @@ -402,7 +394,9 @@ def fit( model = Parametric(self, "MLE", None, False, False, False) p = (x_arr * n).sum() / (n_trials * n.sum()) model.params = np.array([float(n_trials), p]) - model.support = np.array([0, n_trials]) + # Exclusive bounds either side of the outcomes {0, ..., n_trials}; + # see the note in __init__. + model.support = np.array([-1, n_trials + 1]) return model # Narrower than ParametricFitter.from_params, which takes @@ -468,7 +462,9 @@ def from_params( model = Parametric(self, "given parameters", None, False, False, False) model.params = np.array([float(n), prob]) - model.support = np.array([0, n]) + # Exclusive bounds either side of the outcomes {0, ..., n}; see the + # note in __init__. + model.support = np.array([-1, n + 1]) return model diff --git a/surpyval/univariate/parametric/distributions/custom_distribution.py b/surpyval/univariate/parametric/distributions/custom_distribution.py index 8c07e3c..abc34db 100644 --- a/surpyval/univariate/parametric/distributions/custom_distribution.py +++ b/surpyval/univariate/parametric/distributions/custom_distribution.py @@ -1,12 +1,17 @@ import inspect +from typing import Callable +import numpy.typing as npt from autograd import elementwise_grad from surpyval import np from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData class CustomDistribution(OptimisedFitMixin, ParametricFitter): @@ -58,7 +63,16 @@ class CustomDistribution(OptimisedFitMixin, ParametricFitter): >>> model = Gompertz.fit(x) """ - def __init__(self, name, fun, param_names, bounds, support): + def __init__( + self, + name: str, + # Validated at runtime to have the signature (x, *params); + # Callable[..., Boxable] is as close as the type system gets. + fun: Callable[..., Boxable], + param_names: list[str], + bounds: tuple[tuple[int | float | None, int | float | None], ...], + support: tuple[int | float, int | float], + ) -> None: if str(inspect.signature(fun)) != "(x, *params)": detail = "Function must have the signature '(x, *params)'" raise ValueError(detail) @@ -96,31 +110,52 @@ def __init__(self, name, fun, param_names, bounds, support): plot_x_scale="linear", y_ticks=np.linspace(0, 1, 11), ) - self.Hf = fun - self.hf = lambda x, *params: elementwise_grad(self.Hf)(x, *params) - self.sf = lambda x, *params: np.exp(-self.Hf(x, *params)) - self.ff = lambda x, *params: -np.expm1(-self.Hf(x, *params)) - self.df = lambda x, *params: elementwise_grad(self.ff)(x, *params) - - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): - out = [] + # Stored, then exposed through real methods below. Assigning + # over self.Hf and friends stopped being possible once + # OptimisedFitMixin declared them for its own use: a subclass + # inherits those declarations, and assigning to an inherited + # method is an error. Delegating is equivalent -- the previous + # ``self.Hf = fun`` was an unbound instance attribute, so + # ``self.Hf(x, *params)`` called ``fun(x, *params)`` either way. + self._fun = fun + + def Hf(self, x: Numeric, *params: Boxable) -> Boxable: + return self._fun(x, *params) + + def hf(self, x: Numeric, *params: Boxable) -> Boxable: + return elementwise_grad(self.Hf)(x, *params) + + def sf(self, x: Numeric, *params: Boxable) -> Boxable: + return np.exp(-self.Hf(x, *params)) + + def ff(self, x: Numeric, *params: Boxable) -> Boxable: + return -np.expm1(-self.Hf(x, *params)) + + def df(self, x: Numeric, *params: Boxable) -> Boxable: + return elementwise_grad(self.ff)(x, *params) + + # Returns a list, where Weibull returns a tuple and the discrete + # distributions return an array. The base contract does not pin + # this down; callers coerce whichever they get. + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + out: list[float] = [] for low, high in self.bounds: - if (low is None) and (high is None): - out.append(0) + if low is None: + out.append(0.0 if high is None else float(high) - 1.0) elif high is None: - out.append(low + 1.0) - elif low is None: - out.append(high - 1.0) + out.append(float(low) + 1.0) else: - out.append((high + low) / 2.0) + out.append((float(high) + float(low)) / 2.0) - return out + return np.array(out, dtype=float) - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return y - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return y - def mpp_x_transform(self, x, gamma=0): - return x - gamma + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: + return x diff --git a/surpyval/univariate/parametric/distributions/degenerate.py b/surpyval/univariate/parametric/distributions/degenerate.py index 614f76b..c73f942 100644 --- a/surpyval/univariate/parametric/distributions/degenerate.py +++ b/surpyval/univariate/parametric/distributions/degenerate.py @@ -1,3 +1,7 @@ +from typing import Any + +import numpy.typing as npt + """The two degenerate lifetime distributions. ``InstantlyOccurs`` is the point mass at zero (every unit has already @@ -26,43 +30,43 @@ class NeverOccurs(Distribution): name = "NeverOccurs" @classmethod - def sf(cls, x): + def sf(cls, x: npt.ArrayLike, *args: Any, **kwargs: Any) -> npt.NDArray: return np.ones_like(x).astype(float) @classmethod - def ff(cls, x): + def ff(cls, x: npt.ArrayLike, *args: Any, **kwargs: Any) -> npt.NDArray: return np.zeros_like(x).astype(float) @classmethod - def df(cls, x): + def df(cls, x: npt.ArrayLike, *args: Any, **kwargs: Any) -> npt.NDArray: return np.zeros_like(x).astype(float) @classmethod - def hf(cls, x): + def hf(cls, x: npt.ArrayLike, *args: Any, **kwargs: Any) -> npt.NDArray: return np.zeros_like(x).astype(float) @classmethod - def Hf(cls, x): + def Hf(cls, x: npt.ArrayLike, *args: Any, **kwargs: Any) -> npt.NDArray: return np.zeros_like(x).astype(float) @classmethod - def qf(cls, u): + def qf(cls, u: npt.ArrayLike, *args: Any, **kwargs: Any) -> npt.NDArray: return np.full_like(np.asarray(u, dtype=float), np.inf) @classmethod - def mean(cls): + def mean(cls, *args: Any, **kwargs: Any) -> float: return np.inf @classmethod - def random(cls, size): + def random(cls, size: int, *args: Any, **kwargs: Any) -> npt.NDArray: return np.ones(size) * np.inf @classmethod - def to_dict(cls): + def to_dict(cls) -> dict[str, Any]: return stamp_schema({"model": cls.name}) @classmethod - def from_dict(cls, model_dict): + def from_dict(cls, model_dict: dict[str, Any]) -> type["Distribution"]: return cls @@ -72,43 +76,43 @@ class InstantlyOccurs(Distribution): name = "InstantlyOccurs" @classmethod - def sf(cls, x): + def sf(cls, x: npt.ArrayLike, *args: Any, **kwargs: Any) -> npt.NDArray: return np.zeros_like(x).astype(float) @classmethod - def ff(cls, x): + def ff(cls, x: npt.ArrayLike, *args: Any, **kwargs: Any) -> npt.NDArray: return np.ones_like(x).astype(float) @classmethod - def df(cls, x): + def df(cls, x: npt.ArrayLike, *args: Any, **kwargs: Any) -> npt.NDArray: # Point mass at zero: the "density" is the degenerate spike there. x = np.asarray(x, dtype=float) return np.where(x == 0, np.inf, 0.0) @classmethod - def hf(cls, x): + def hf(cls, x: npt.ArrayLike, *args: Any, **kwargs: Any) -> npt.NDArray: return np.full_like(np.asarray(x, dtype=float), np.inf) @classmethod - def Hf(cls, x): + def Hf(cls, x: npt.ArrayLike, *args: Any, **kwargs: Any) -> npt.NDArray: return np.full_like(x, np.inf, dtype=float) @classmethod - def qf(cls, u): + def qf(cls, u: npt.ArrayLike, *args: Any, **kwargs: Any) -> npt.NDArray: return np.zeros_like(np.asarray(u, dtype=float)) @classmethod - def mean(cls): + def mean(cls, *args: Any, **kwargs: Any) -> float: return 0.0 @classmethod - def random(cls, size): + def random(cls, size: int, *args: Any, **kwargs: Any) -> npt.NDArray: return np.zeros(size) @classmethod - def to_dict(cls): + def to_dict(cls) -> dict[str, Any]: return stamp_schema({"model": cls.name}) @classmethod - def from_dict(cls, model_dict): + def from_dict(cls, model_dict: dict[str, Any]) -> type["Distribution"]: return cls diff --git a/surpyval/univariate/parametric/distributions/discrete_weibull.py b/surpyval/univariate/parametric/distributions/discrete_weibull.py index 723ae9d..f942349 100644 --- a/surpyval/univariate/parametric/distributions/discrete_weibull.py +++ b/surpyval/univariate/parametric/distributions/discrete_weibull.py @@ -10,6 +10,7 @@ Numeric, OptimisedFitMixin, ) +from surpyval.utils.surpyval_data import SurpyvalData class DiscreteWeibull_(OptimisedFitMixin, DiscreteParametricFitter): @@ -55,43 +56,67 @@ def __init__(self, name: str) -> None: ) def _parameter_initialiser( - self, - x: npt.NDArray, - c: npt.NDArray | None = None, - n: npt.NDArray | None = None, - t: npt.NDArray | None = None, - offset: bool = False, + self, data: SurpyvalData, offset: bool = False ) -> npt.NDArray: # q ~ P(survive the first cycle) from the empirical fraction above 1; # start beta at 1 (the geometric special case). + x = data.x finite = x[np.isfinite(x)] q = (finite > 1).mean() if finite.size else 0.5 return np.array([min(max(q, 1e-3), 1 - 1e-3), 1.0]) def sf(self, x: Numeric, q: Boxable, beta: Boxable) -> Boxable: r"""Survival function :math:`R(k) = q^{k^{\beta}}`.""" - return q ** (x**beta) + # Below zero the base of ``x**beta`` is negative and a fractional + # power of it is complex -- sf(-1) came back as 1.035+0.547j. + # Nothing can fail before the first trial, so R = 1 there. The + # dead branch is evaluated at 1 rather than 0 because ``0**beta`` + # has a NaN gradient with respect to beta (see ``log_df``). + safe_x = np.where(x < 0.0, 1.0, x) + return np.where(x < 0.0, 1.0, q ** (safe_x**beta)) def ff(self, x: Numeric, q: Boxable, beta: Boxable) -> Boxable: r"""CDF :math:`F(k) = 1 - q^{k^{\beta}}`.""" - return 1.0 - q ** (x**beta) + return 1.0 - self.sf(x, q, beta) def df(self, x: Numeric, q: Boxable, beta: Boxable) -> Boxable: r"""PMF :math:`P(T=k) = q^{(k-1)^{\beta}} - q^{k^{\beta}}`.""" - return q ** ((x - 1.0) ** beta) - q ** (x**beta) + # Below k = 1 the exponent base (k - 1) is negative, and a negative + # base to a fractional power is complex: df(0) came back as + # 0.0355+0.5468j. Guard the base as ``log_df`` already does, then + # zero the whole thing below the support. + # ``x`` is clamped, not just the result, so the discarded branch of + # the np.where never evaluates a negative base at all -- otherwise + # it still computes the NaN and warns before throwing it away. + safe_x = np.where(x < 1.0, 1.0, x) + km1 = safe_x - 1.0 + safe_km1 = np.where(km1 > 0, km1, 1.0) + term_low = np.where(km1 > 0, q ** (safe_km1**beta), 1.0) + return np.where(x < 1.0, 0.0, term_low - q ** (safe_x**beta)) def hf(self, x: Numeric, q: Boxable, beta: Boxable) -> Boxable: r"""Discrete hazard, :math:`1 - q^{k^{\beta} - (k-1)^{\beta}}`.""" - return 1.0 - q ** (x**beta - (x - 1.0) ** beta) + # Same negative-base problem as ``df``, and no mass to condition + # on below k = 1. + safe_x = np.where(x < 1.0, 1.0, x) + km1 = safe_x - 1.0 + safe_km1 = np.where(km1 > 0, km1, 1.0) + exponent = safe_x**beta - np.where(km1 > 0, safe_km1**beta, 0.0) + return np.where(x < 1.0, 0.0, 1.0 - q**exponent) def Hf(self, x: Numeric, q: Boxable, beta: Boxable) -> Boxable: r"""Cumulative hazard :math:`H(k) = -k^{\beta}\ln q`.""" - return -(x**beta) * np.log(q) + safe_x = np.where(x < 0.0, 1.0, x) + return np.where(x < 0.0, 0.0, -(safe_x**beta) * np.log(q)) def qf(self, u: Numeric, q: Boxable, beta: Boxable) -> Boxable: r"""Quantile: the smallest integer ``k`` with :math:`F(k) \geq u`.""" u = np.asarray(u, dtype=float) k = (np.log1p(-u) / np.log(q)) ** (1.0 / beta) + # See ``Geometric.qf``: inverting a CDF built by cancellation + # lands a few ulp above the integer, and ceil() would answer + # k + 1 for a u that came straight out of ``ff``. + k = np.where(np.abs(k - np.round(k)) < 1e-9, np.round(k), k) return np.maximum(np.ceil(k), 1.0) def mean(self, q: Boxable, beta: Boxable) -> Boxable: @@ -104,23 +129,31 @@ def moment(self, m: int, q: Boxable, beta: Boxable) -> Boxable: def random( self, size: int | tuple[int, ...], q: Boxable, beta: Boxable - ) -> Boxable: + ) -> npt.NDArray: U = uniform.rvs(size=size) - return self.qf(U, q, beta) + # qf is declared Boxable because a fit differentiates it; + # sampling never does, so this is always a real array. + return np.asarray(self.qf(U, q, beta)) def log_sf(self, x: Numeric, q: Boxable, beta: Boxable) -> Boxable: - return (x**beta) * np.log(q) + safe_x = np.where(x < 0.0, 1.0, x) + return np.where(x < 0.0, 0.0, (safe_x**beta) * np.log(q)) def log_df(self, x: Numeric, q: Boxable, beta: Boxable) -> Boxable: # PMF = q^{(k-1)^beta} - q^{k^beta}. At k = 1 the first term is # q^{0^beta} = 1 with no beta dependence, but 0**beta has a NaN # gradient w.r.t. beta under autograd, so guard the base: where # k = 1 the term is the constant 1. - km1 = x - 1.0 + # Below k = 1 there is no mass, so this is -inf. Clamping ``x`` to 1 + # rather than leaving it means the discarded branch evaluates the + # k = 1 mass (1 - q, safely positive) instead of a negative base + # and a log of zero, both of which warn before being thrown away. + safe_x = np.where(x < 1.0, 1.0, x) + km1 = safe_x - 1.0 safe_km1 = np.where(km1 > 0, km1, 1.0) term_low = np.where(km1 > 0, q ** (safe_km1**beta), 1.0) - term_high = q ** (x**beta) - return np.log(term_low - term_high) + term_high = q ** (safe_x**beta) + return np.where(x < 1.0, -np.inf, np.log(term_low - term_high)) DiscreteWeibull = DiscreteWeibull_("DiscreteWeibull") diff --git a/surpyval/univariate/parametric/distributions/discretize.py b/surpyval/univariate/parametric/distributions/discretize.py index 83c21b6..b2b4a97 100644 --- a/surpyval/univariate/parametric/distributions/discretize.py +++ b/surpyval/univariate/parametric/distributions/discretize.py @@ -12,6 +12,7 @@ OptimisedFitMixin, ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData class DiscretizedFitter(OptimisedFitMixin, DiscreteParametricFitter): @@ -64,14 +65,12 @@ def __init__(self, distribution: ParametricFitter) -> None: ) def _parameter_initialiser( - self, - x: npt.NDArray, - c: npt.NDArray | None = None, - n: npt.NDArray | None = None, - t: npt.NDArray | None = None, - offset: bool = False, - ) -> npt.NDArray | tuple[float, ...]: - return self.dist._parameter_initialiser(x, c=c, n=n, t=t) + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + return np.asarray( + self.dist._parameter_initialiser(data), + dtype=float, + ) def sf(self, x: Numeric, *params: Boxable) -> Boxable: r"""Survival :math:`R_K(k) = R(k)` (the continuous survival).""" diff --git a/surpyval/univariate/parametric/distributions/exact_event_time.py b/surpyval/univariate/parametric/distributions/exact_event_time.py index b115974..59ce6ce 100644 --- a/surpyval/univariate/parametric/distributions/exact_event_time.py +++ b/surpyval/univariate/parametric/distributions/exact_event_time.py @@ -1,6 +1,10 @@ +import numpy.typing as npt + import surpyval from surpyval import np from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, ParametricFitter, reject_structural_params, ) @@ -9,7 +13,7 @@ class ExactEventTime_(ParametricFitter): - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=1, @@ -20,33 +24,115 @@ def __init__(self, name): plot_x_scale="linear", ) - def sf(self, x, T): - x = np.atleast_1d(x) - return (x < T).astype(float) + def sf(self, x: Numeric, T: Boxable) -> npt.NDArray: + x_arr = np.atleast_1d(x) + return (x_arr < T).astype(float) + + def ff(self, x: Numeric, T: Boxable) -> npt.NDArray: + x_arr = np.atleast_1d(x) + return (x_arr >= T).astype(float) + + # ``df`` and ``hf`` do not exist for a point mass, and used to be + # answered with ``inf``. + # + # All the probability sits at T, so the density is a Dirac delta: + # zero everywhere, infinite at one point, integrating to one. There + # is no function of x that represents it -- the old ``df`` returned + # ``inf`` at T and 0 elsewhere, which integrates to ``inf``, not 1. + # The hazard is the same delta divided by a survival that is zero + # from T onwards, so it was ``inf`` at T *and everywhere after*. + # Inherited ``log_df`` then computed ``log(inf) - inf`` and returned + # ``nan``. + # + # Raising stops that at the call site. An ``inf`` does not: it + # propagates into a plot, a likelihood or a mixture weight and + # surfaces somewhere with no connection to the cause. Bernoulli + # already omits all three for the same reason -- no time axis to + # carry a density. + # + # ``sf``, ``ff``, ``Hf`` and ``qf`` are all well defined here and + # are unaffected. + def df(self, x: Numeric, T: Boxable) -> npt.NDArray: + raise NotImplementedError( + "ExactEventTime has no density: all of its probability is a " + "point mass at T, so the density is a Dirac delta rather than " + "a function of x. Use sf, ff or Hf, which are step functions " + "and well defined." + ) - def ff(self, x, T): - x = np.atleast_1d(x) - return (x >= T).astype(float) + def hf(self, x: Numeric, T: Boxable) -> npt.NDArray: + raise NotImplementedError( + "ExactEventTime has no hazard rate: its density is a Dirac " + "delta at T and its survival is zero from T onwards, so the " + "ratio is undefined at and after the event. Hf is well " + "defined -- it steps from 0 to infinity at T." + ) - def df(self, x, T): - x = np.atleast_1d(x) - df = np.zeros_like(x).astype(float) - df[x == T] = np.inf - return df + def Hf(self, x: Numeric, T: Boxable) -> npt.NDArray: + # -log R(x): zero while the item survives, infinite once the + # event has certainly happened. Previously this returned hf, + # which happened to be the same two values. + x_arr = np.atleast_1d(x) + Hf = np.zeros_like(x_arr).astype(float) + Hf[x_arr >= T] = np.inf + return Hf + + def qf(self, u: Numeric, T: Boxable) -> Boxable: + r"""Quantile function: :math:`T` for every :math:`u \in (0, 1)`. + + All the probability sits at ``T``, so the smallest ``x`` with + :math:`F(x) \geq u` is ``T`` whatever ``u`` is. Unlike ``df`` and + ``hf`` there is nothing undefined here -- a point mass has a + perfectly good quantile, it is just a constant one. + + Examples + -------- + >>> from surpyval import ExactEventTime + >>> ExactEventTime.qf([0.1, 0.5, 0.9], 5.0) + array([5., 5., 5.]) + """ + return np.ones_like(np.atleast_1d(np.asarray(u, dtype=float))) * T - def hf(self, x, T): - x = np.atleast_1d(x) - hf = np.zeros_like(x).astype(float) - hf[x >= T] = np.inf - return hf + def mean(self, T: Boxable) -> Boxable: + r"""Mean of the distribution: :math:`E[X] = T`. - def Hf(self, x, T): - return self.hf(x, T) + Examples + -------- + >>> from surpyval import ExactEventTime + >>> ExactEventTime.mean(5.0) + 5.0 + """ + return T + + def moment(self, m: int, T: Boxable) -> Boxable: + r"""m-th raw moment: :math:`E[X^m] = T^m`. + + Exact, where the inherited quadrature over a density would have + had no density to integrate. + + Examples + -------- + >>> from surpyval import ExactEventTime + >>> ExactEventTime.moment(2, 5.0) + 25.0 + """ + return T**m - def random(self, size, T): + def random(self, size: int | tuple[int, ...], T: Boxable) -> npt.NDArray: return np.ones(size) * T - def fit(self, x, c=None, n=None, t=None): + # Narrower than OptimisedFitMixin.fit by design, and no longer a + # Liskov violation: ExactEventTime_ does not inherit that mixin, + # so there is no wider fit above this one. The event time is + # bracketed exactly by the censoring bounds, so there is nothing + # for how, offset, zi or lfp to do. + def fit( + self, + x: npt.ArrayLike, + c: npt.ArrayLike | None = None, + n: npt.ArrayLike | None = None, + t: npt.ArrayLike | None = None, + ) -> Parametric: x, c, n, t = surpyval.xcnt_handler(x=x, c=c, n=n, t=t) if 0 in c: @@ -80,7 +166,13 @@ def fit(self, x, c=None, n=None, t=None): model.params = np.array([T]) return model - def from_params(self, params, gamma=None, p=None, f0=None): + def from_params( + self, + params: npt.ArrayLike, + gamma: Boxable | None = None, + p: Boxable | None = None, + f0: Boxable | None = None, + ) -> Parametric: """Create an ExactEventTime model from the known event time. ``params`` is the event time, previously named ``T``. ``gamma``, @@ -93,4 +185,4 @@ def from_params(self, params, gamma=None, p=None, f0=None): return model -ExactEventTime = ExactEventTime_("ExactEventTime") +ExactEventTime: ExactEventTime_ = ExactEventTime_("ExactEventTime") diff --git a/surpyval/univariate/parametric/distributions/expo_weibull.py b/surpyval/univariate/parametric/distributions/expo_weibull.py index eca1ed3..f2458f8 100755 --- a/surpyval/univariate/parametric/distributions/expo_weibull.py +++ b/surpyval/univariate/parametric/distributions/expo_weibull.py @@ -1,16 +1,20 @@ +import numpy.typing as npt from scipy import integrate from scipy.special import xlogy from surpyval import np from surpyval.univariate import parametric as para from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData class ExpoWeibull_(OptimisedFitMixin, ParametricFitter): - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=3, @@ -26,7 +30,13 @@ def __init__(self, name): ) self.supports_mpp = False - def _gumbel_seed(self, x, c, n, refine): + def _gumbel_seed( + self, + x: npt.NDArray, + c: npt.NDArray | None, + n: npt.NDArray | None, + refine: bool, + ) -> tuple[float, float]: """ Seed alpha and beta from a Gumbel fit to log(x). @@ -46,7 +56,9 @@ def _gumbel_seed(self, x, c, n, refine): log_x = np.log(x) log_x[np.isnan(log_x)] = 0 gumb = para.Gumbel.fit(log_x, c, n, how="MLE" if refine else "MPP") - if refine and not gumb.res.success: + # ``res`` is the optimiser result, present only on an MLE + # fit -- which is the only branch that sets refine. + if refine and not gumb.res.success: # type: ignore[attr-defined] gumb = para.Gumbel.fit(log_x, c, n, how="MPP") mu, sigma = gumb.params alpha, beta = np.exp(mu), 1.0 / sigma @@ -56,7 +68,10 @@ def _gumbel_seed(self, x, c, n, refine): beta = 1.0 return alpha, beta - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + x, c, n = data.x, data.c, data.n if offset: # Estimate the offset first and seed alpha and beta from the # shifted data. Taking logs before removing the shift reads @@ -72,10 +87,15 @@ def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): # actually installed defeats the point of shifting at all. gamma = np.min(x) - 1.0 alpha, beta = self._gumbel_seed(x - gamma, c, n, refine=True) - return gamma, alpha, beta, 1.0 - return (*self._gumbel_seed(x, c, n, refine=False), 1.0) + return np.array([gamma, alpha, beta, 1.0], dtype=float) + return np.array( + [*self._gumbel_seed(x, c, n, refine=False), 1.0], + dtype=float, + ) - def sf(self, x, alpha, beta, mu): + def sf( + self, x: Numeric, alpha: Boxable, beta: Boxable, mu: Boxable + ) -> Boxable: r""" Survival (or reliability) function for the ExpoWeibull Distribution: @@ -117,7 +137,9 @@ def sf(self, x, alpha, beta, mu): # inf/-inf for representable tail probabilities (#257). return -np.expm1(mu * np.log1p(-np.exp(-((x / alpha) ** beta)))) - def ff(self, x, alpha, beta, mu): + def ff( + self, x: Numeric, alpha: Boxable, beta: Boxable, mu: Boxable + ) -> Boxable: r""" Failure (CDF or unreliability) function for the ExpoWeibull @@ -155,45 +177,9 @@ def ff(self, x, alpha, beta, mu): """ return np.power(1 - np.exp(-((x / alpha) ** beta)), mu) - def cs(self, x, X, alpha, beta, mu): - r""" - - Conditional survival (or reliability) function for the ExpoWeibull - Distribution: - - .. math:: - R(x, X) = \frac{R(x + X)}{R(X)} - - Parameters - ---------- - - x : numpy array or scalar - The values at which the function will be calculated - alpha : numpy array or scalar - scale parameter for the ExpoWeibull distribution - beta : numpy array or scalar - shape parameter for the ExpoWeibull distribution - mu : numpy array or scalar - shape parameter for the ExpoWeibull distribution - - Returns - ------- - - cs : scalar or numpy array - The value(s) of the conditional survival function at x. - - Examples - -------- - >>> import numpy as np - >>> from surpyval import ExpoWeibull - >>> x = np.array([1, 2, 3, 4, 5]) - >>> ExpoWeibull.cs(x, 1, 3, 4, 1.2) - array([8.77367129e-01, 4.25451775e-01, 5.09266354e-02, 5.37452200e-04, - 1.35732908e-07]) - """ - return self.sf(x + X, alpha, beta, mu) / self.sf(X, alpha, beta, mu) - - def df(self, x, alpha, beta, mu): + def df( + self, x: Numeric, alpha: Boxable, beta: Boxable, mu: Boxable + ) -> Boxable: r""" Density function for the ExpoWeibull Distribution: @@ -237,7 +223,9 @@ def df(self, x, alpha, beta, mu): * np.exp(-((x / alpha) ** beta)) ) - def hf(self, x, alpha, beta, mu): + def hf( + self, x: Numeric, alpha: Boxable, beta: Boxable, mu: Boxable + ) -> Boxable: r""" Instantaneous hazard rate for the ExpoWeibull Distribution: @@ -273,7 +261,9 @@ def hf(self, x, alpha, beta, mu): """ return self.df(x, alpha, beta, mu) / self.sf(x, alpha, beta, mu) - def Hf(self, x, alpha, beta, mu): + def Hf( + self, x: Numeric, alpha: Boxable, beta: Boxable, mu: Boxable + ) -> Boxable: r""" Instantaneous hazard rate for the ExpoWeibull Distribution: @@ -310,18 +300,20 @@ def Hf(self, x, alpha, beta, mu): """ return -np.log(self.sf(x, alpha, beta, mu)) - def qf(self, p, alpha, beta, mu): + def qf( + self, u: Numeric, alpha: Boxable, beta: Boxable, mu: Boxable + ) -> Boxable: r""" Instantaneous hazard rate for the ExpoWeibull Distribution: .. math:: - q(p) = + q(u) = Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated alpha : numpy array or scalar scale parameter for the ExpoWeibull distribution @@ -334,19 +326,21 @@ def qf(self, p, alpha, beta, mu): ------- Q : scalar or numpy array - The quantiles for the Weibull distribution at each value p + The quantiles for the Weibull distribution at each value u Examples -------- >>> import numpy as np >>> from surpyval import ExpoWeibull - >>> p = np.array([.1, .2, .3, .4, .5]) - >>> ExpoWeibull.qf(p, 3, 4, 1.2) + >>> u = np.array([.1, .2, .3, .4, .5]) + >>> ExpoWeibull.qf(u, 3, 4, 1.2) array([1.89361341, 2.2261045 , 2.46627621, 2.66992747, 2.85807988]) """ - return alpha * (-np.log1p(-(p ** (1.0 / mu)))) ** (1 / beta) + return alpha * (-np.log1p(-(u ** (1.0 / mu)))) ** (1 / beta) - def log_df(self, x, alpha, beta, mu): + def log_df( + self, x: Numeric, alpha: Boxable, beta: Boxable, mu: Boxable + ) -> Boxable: return ( np.log(beta) + np.log(mu) @@ -356,23 +350,72 @@ def log_df(self, x, alpha, beta, mu): - ((x / alpha) ** beta) ) - def log_ff(self, x, alpha, beta, mu): + def log_ff( + self, x: Numeric, alpha: Boxable, beta: Boxable, mu: Boxable + ) -> Boxable: return mu * np.log1p(-np.exp(-((x / alpha) ** beta))) - def log_sf(self, x, alpha, beta, mu): + def log_sf( + self, x: Numeric, alpha: Boxable, beta: Boxable, mu: Boxable + ) -> Boxable: # log of the cancellation-free sf form; the naive log1p(-(...)^mu) # returns -inf once the inner power rounds to 1 (#257). return np.log( -np.expm1(mu * np.log1p(-np.exp(-((x / alpha) ** beta)))) ) - def mean(self, alpha, beta, mu): - def func(x): - return x * self.df(x, alpha, beta, mu) + def moment( + self, m: int, alpha: Boxable, beta: Boxable, mu: Boxable + ) -> Boxable: + r""" + + m-th (non central) moment of the ExpoWeibull distribution. + + .. math:: + E = \int_{0}^{\infty} x^{m} f(x) dx + + There is a closed form -- an infinite series in + :math:`\binom{\mu - 1}{i}(-1)^{i}(i + 1)^{-(1 + m/\beta)}` -- but + it only terminates when :math:`\mu` is a positive integer, and + for other :math:`\mu` it is alternating and slow to converge, + losing significance to cancellation as :math:`\mu` grows. The + integral is quadrature either way, so this takes it directly, as + ``entropy`` does for the same reason. + + Parameters + ---------- + + m : integer + The ordinal of the moment to calculate + alpha : numpy array or scalar + scale parameter for the ExpoWeibull distribution + beta : numpy array or scalar + shape parameter for the ExpoWeibull distribution + mu : numpy array or scalar + shape parameter for the ExpoWeibull distribution + + Returns + ------- + + moment : scalar or numpy array + The moment(s) of the ExpoWeibull distribution + + Examples + -------- + >>> from surpyval import ExpoWeibull + >>> ExpoWeibull.moment(2, 3, 4, 1.2) + 8.598425613605164 + """ + + def func(x: float) -> float: + return float(x**m * self.df(x, alpha, beta, mu)) return integrate.quad(func, 0, np.inf)[0] - def entropy(self, alpha, beta, mu): + def mean(self, alpha: Boxable, beta: Boxable, mu: Boxable) -> Boxable: + return self.moment(1, alpha, beta, mu) + + def entropy(self, alpha: Boxable, beta: Boxable, mu: Boxable) -> Boxable: r""" Calculates the entropy of the ExpoWeibull distribution. @@ -406,16 +449,16 @@ def entropy(self, alpha, beta, mu): 1.8227536487527594 """ - def func(x): + def func(x: float) -> float: f = self.df(x, alpha, beta, mu) - return xlogy(f, f) + return float(xlogy(f, f)) return -integrate.quad(func, 0, np.inf)[0] - def mpp_x_transform(self, x, gamma=0): - return np.log(x - gamma) + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: + return np.log(x) - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: mu = params[-1] mask = (y == 0) | (y == 1) out = np.zeros_like(y) @@ -423,12 +466,14 @@ def mpp_y_transform(self, y, *params): out[mask] = np.nan return out - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: i = len(params) mu = params[i - 1] return (1 - np.exp(-np.exp(y))) ** mu - def unpack_rr(self, params, rr): + def unpack_rr( + self, params: npt.NDArray, rr: str + ) -> tuple[Boxable, Boxable, float]: if rr == "y": beta = params[0] alpha = np.exp(params[1] / -beta) diff --git a/surpyval/univariate/parametric/distributions/exponential.py b/surpyval/univariate/parametric/distributions/exponential.py index 840c4f4..73590ef 100755 --- a/surpyval/univariate/parametric/distributions/exponential.py +++ b/surpyval/univariate/parametric/distributions/exponential.py @@ -1,5 +1,7 @@ import warnings +from typing import Any +import numpy.typing as npt from scipy.special import factorial from surpyval import np @@ -8,9 +10,12 @@ entry_times, ) from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData class Exponential_(OptimisedFitMixin, ParametricFitter): @@ -24,7 +29,7 @@ class Exponential_(OptimisedFitMixin, ParametricFitter): """ - def __init__(self, name: str): + def __init__(self, name: str) -> None: super().__init__( name=name, k=1, @@ -47,7 +52,7 @@ def __init__(self, name: str): ], ) - def _closed_form_mle(self, data): + def _closed_form_mle(self, data: SurpyvalData) -> npt.NDArray | None: r"""Exact MLE: total events over total exposure. With only exact and right-censored observations the @@ -86,14 +91,20 @@ def _closed_form_mle(self, data): return None return np.array([events / exposure]) - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + x = data.x rate = 1.0 / x[np.isfinite(x)].mean() if offset: - return np.min(x) - (np.max(x) - np.min(x)) / 10.0, rate + return np.array( + [np.min(x) - (np.max(x) - np.min(x)) / 10.0, rate], + dtype=float, + ) else: - return np.array([rate]) + return np.array([rate], dtype=float) - def sf(self, x, failure_rate): + def sf(self, x: Numeric, failure_rate: Boxable) -> Boxable: r""" Survival (or Reliability) function for the Exponential Distribution: @@ -129,7 +140,7 @@ def sf(self, x, failure_rate): """ return np.exp(-failure_rate * x) - def cs(self, x, X, failure_rate): + def cs(self, x: Numeric, X: Numeric, failure_rate: Boxable) -> Boxable: r""" Conditional survival function for the Exponential Distribution: @@ -167,7 +178,7 @@ def cs(self, x, X, failure_rate): """ return self.sf(x, failure_rate) - def ff(self, x, failure_rate): + def ff(self, x: Numeric, failure_rate: Boxable) -> Boxable: r""" CDF (or unreliability or failure) function for the Exponential @@ -200,7 +211,7 @@ def ff(self, x, failure_rate): """ return -np.expm1(-failure_rate * x) - def df(self, x, failure_rate): + def df(self, x: Numeric, failure_rate: Boxable) -> Boxable: r""" Density function for the Exponential Distribution: @@ -233,7 +244,7 @@ def df(self, x, failure_rate): """ return failure_rate * np.exp(-failure_rate * x) - def hf(self, x, failure_rate): + def hf(self, x: Numeric, failure_rate: Boxable) -> Boxable: r""" Instantaneous hazard rate for the Exponential Distribution. @@ -269,7 +280,7 @@ def hf(self, x, failure_rate): """ return np.ones_like(x) * failure_rate - def Hf(self, x, failure_rate): + def Hf(self, x: Numeric, failure_rate: Boxable) -> Boxable: r""" Cumulative hazard rate for the Exponential Distribution. @@ -303,18 +314,18 @@ def Hf(self, x, failure_rate): x = np.array(x) return failure_rate * x - def qf(self, p, failure_rate): + def qf(self, u: Numeric, failure_rate: Boxable) -> Boxable: r""" Quantile function for the Exponential Distribution: .. math:: - q(p) = \frac{-\ln\left ( 1 - p \right )}{\lambda} + q(u) = \frac{-\ln\left ( 1 - u \right )}{\lambda} Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated failure_rate : numpy array or scalar The scale parameter for the Exponential distribution @@ -323,19 +334,19 @@ def qf(self, p, failure_rate): ------- q : scalar or numpy array - The quantiles for the Exponential distribution at each value p. + The quantiles for the Exponential distribution at each value u. Examples -------- >>> import numpy as np >>> from surpyval import Exponential - >>> p = np.array([.1, .2, .3, .4, .5]) - >>> Exponential.qf(p, 3) + >>> u = np.array([.1, .2, .3, .4, .5]) + >>> Exponential.qf(u, 3) array([0.03512017, 0.07438118, 0.11889165, 0.17027521, 0.23104906]) """ - return -np.log1p(-p) / failure_rate + return -np.log1p(-u) / failure_rate - def mean(self, failure_rate): + def mean(self, failure_rate: Boxable) -> Boxable: r""" Calculates the mean of the Exponential distribution with given @@ -364,18 +375,18 @@ def mean(self, failure_rate): """ return 1.0 / failure_rate - def moment(self, n, failure_rate): + def moment(self, m: int, failure_rate: Boxable) -> Boxable: r""" - Calculates the n-th moment of the Exponential distribution. + Calculates the m-th moment of the Exponential distribution. .. math:: - E = \frac{n!}{\lambda^{n}} + E = \frac{m!}{\lambda^{m}} Parameters ---------- - n : integer or numpy array of integers + m : integer The ordinal of the moment to calculate failure_rate : numpy array or scalar The scale parameter for the Exponential distribution @@ -392,9 +403,9 @@ def moment(self, n, failure_rate): >>> Exponential.moment(2, 3) np.float64(0.2222222222222222) """ - return factorial(n) / (failure_rate**n) + return factorial(m) / (failure_rate**m) - def entropy(self, failure_rate): + def entropy(self, failure_rate: Boxable) -> Boxable: r""" Calculates the entropy of the Exponential distribution. @@ -422,37 +433,37 @@ def entropy(self, failure_rate): """ return 1 - np.log(failure_rate) - def log_df(self, x, failure_rate): + def log_df(self, x: Numeric, failure_rate: Boxable) -> Boxable: return np.log(failure_rate) - failure_rate * x - def log_sf(self, x, failure_rate): + def log_sf(self, x: Numeric, failure_rate: Boxable) -> Boxable: return -failure_rate * x - def mpp_x_transform(self, x, gamma=0): - return x - gamma + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: + return x - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: mask = (y == 0) | (y == 1) out = np.zeros_like(y) out[~mask] = -np.log(1 - y[~mask]) out[mask] = np.nan return out - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: # Inverse of y = -log(1 - F) is F = 1 - exp(-y) (#257). return 1 - np.exp(-y) def mpp( self, - x, - c=None, - n=None, - t=None, - heuristic="Nelson-Aalen", - rr="y", - on_d_is_0=False, - offset=False, - ): + x: npt.NDArray, + c: npt.NDArray | None = None, + n: npt.NDArray | None = None, + t: npt.NDArray | None = None, + heuristic: str = "Nelson-Aalen", + rr: str = "y", + on_d_is_0: bool = False, + offset: bool = False, + ) -> dict[str, Any]: assert rr in ["x", "y"] # Forward the truncation windows (previously dropped, #280). x_pp, r, d, F = plotting_positions( @@ -464,7 +475,7 @@ def mpp( F = F[d > 0] # Linearise - y_pp = self.mpp_y_transform(F) + y_pp = np.asarray(self.mpp_y_transform(F)) mask = np.isfinite(y_pp) if not mask.all(): diff --git a/surpyval/univariate/parametric/distributions/fixed_event_probability.py b/surpyval/univariate/parametric/distributions/fixed_event_probability.py new file mode 100644 index 0000000..9cffb58 --- /dev/null +++ b/surpyval/univariate/parametric/distributions/fixed_event_probability.py @@ -0,0 +1,277 @@ +"""The fixed-event-probability model. + +``F(x) = p`` at every ``x``: a fraction ``p`` of units fail and the rest +never do, with nothing said about *when*. It is the two-point mixture of +:class:`InstantlyOccurs` (weight ``p``) and :class:`NeverOccurs` (weight +``1 - p``), which is why ``degenerate.py`` describes those two as this +model's limits at ``p = 1`` and ``p = 0``. + +This was exported as ``Bernoulli`` as well until 0.19.1, when +``Bernoulli`` became a true Bernoulli -- a coin flip over ``{0, 1}`` +whose survival steps at the outcome. The two are different models and +now different classes; this one is unchanged. + +``df``, ``hf``, ``qf`` and ``mean`` are absent by construction: ``F`` is +constant, so there is no density, no invertible quantile, and no time to +average. +""" + +import numpy.typing as npt +from scipy.stats import uniform + +from surpyval import np +from surpyval.univariate.parametric.discrete_fitter import ( + DiscreteParametricFitter, +) +from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, + reject_structural_params, +) + +from ..parametric import Parametric + + +class FixedEventProbability_(DiscreteParametricFitter): + def __init__(self, name: str) -> None: + super().__init__( + name=name, + k=1, + bounds=((0, 1),), + support=(0, 1), + param_names=["p"], + param_map={"p": 0}, + plot_x_scale="linear", + ) + + def sf(self, x: Numeric, p: Boxable) -> Boxable: + r""" + + Survival (or reliability) function for the + FixedEventProbability model: + + .. math:: + R(x) = 1 - p + + Parameters + ---------- + + x : numpy array or scalar + The values at which the function will be calculated + p : float + The probability of failure of the thing + + Returns + ------- + + sf : scalar or numpy array + The value(s) of the reliability function at x. Which for this + distribution is constant + + Examples + -------- + >>> import numpy as np + >>> from surpyval import FixedEventProbability + >>> x = np.array([1, 2, 3, 4, 5]) + >>> FixedEventProbability.sf(x, 0.5) + array([0.5, 0.5, 0.5, 0.5, 0.5]) + """ + return 1.0 - self.ff(x, p) + + def ff(self, x: Numeric, p: Boxable) -> Boxable: + r""" + + Failure (CDF or unreliability) function for the + FixedEventProbability model: + + .. math:: + F(x) = p + + Parameters + ---------- + + x : numpy array or scalar + The values at which the function will be calculated + p : float + The probability of failure of the thing + + Returns + ------- + + ff : scalar or numpy array + The value(s) of the failure function at x. + + Examples + -------- + >>> import numpy as np + >>> from surpyval import FixedEventProbability + >>> x = np.array([1, 2, 3, 4, 5]) + >>> FixedEventProbability.ff(x, 0.5) + array([0.5, 0.5, 0.5, 0.5, 0.5]) + """ + return np.ones_like(x).astype(float) * p + + def Hf(self, x: Numeric, p: Boxable) -> Boxable: + r""" + + Cumulative hazard function for the FixedEventProbability model: + + .. math:: + H(x) = -\ln R(x) = -\ln (1 - p) + + Constant in ``x``, like the survival it comes from. There is no + hazard *rate* -- ``hf`` is absent because ``F`` is flat, so the + mass is an atom rather than a density -- but the cumulative + hazard is still well defined, exactly as for + :class:`ExactEventTime`, whose ``Hf`` exists while its ``hf`` + does not. + + Without it ``log_sf`` and ``log_ff``, which the base class writes + in terms of ``Hf``, raised ``AttributeError`` rather than + returning the constants they should. + + Parameters + ---------- + + x : numpy array or scalar + The values at which the function will be calculated + p : float + The probability of failure of the thing + + Returns + ------- + + Hf : scalar or numpy array + The value(s) of the cumulative hazard function at x + + Examples + -------- + >>> import numpy as np + >>> from surpyval import FixedEventProbability + >>> x = np.array([1, 2, 3]) + >>> FixedEventProbability.Hf(x, 0.5) + array([0.69314718, 0.69314718, 0.69314718]) + """ + return -np.log(self.sf(x, p)) + + def moment(self, m: int, p: Boxable) -> Boxable: + r""" + + m-th moment of the FixedEventProbability model + + .. math:: + M(m) = p + + Parameters + ---------- + + m : integer + The ordinal of the moment to calculate + p : float + The probability of failure of the thing + + Returns + ------- + + mean : scalar or numpy array + The moment(s) of the FixedEventProbability model + + Examples + -------- + >>> from surpyval import FixedEventProbability + >>> FixedEventProbability.moment(2, 0.5) + 0.5 + """ + return p + + def entropy(self, p: Boxable) -> Boxable: + return -(1 - p) * np.log1p(-p) - p * np.log(p) + + def random(self, size: int | tuple[int, ...], p: Boxable) -> npt.NDArray: + r""" + + Draws random samples from the distribution in shape `size` + + Parameters + ---------- + + size : integer or tuple of positive integers + Shape or size of the random draw + p : float + The probability of failure of the thing + + Returns + ------- + + random : scalar or numpy array + Random values drawn from the distribution in shape `size` + + """ + U = uniform.rvs(size=size) + return (U <= p).astype(int) + + def fit( + self, x: npt.ArrayLike, n: npt.NDArray | None = None + ) -> Parametric: + x_arr = np.atleast_1d(x) + # Each observation must be a 0 or a 1 — elementwise, for any length + # (the previous check broadcast x against the literal [0, 1], so any + # input of length != 2 crashed and [1, 1] was rejected, #257). + if not np.isin(x_arr, (0, 1)).all(): + raise ValueError("'x' must be either 0 or 1") + n_arr = np.ones_like(x_arr) if n is None else np.atleast_1d(n) + if n_arr.shape[0] != x_arr.shape[0]: + raise ValueError("'n' must be the same length as 'x'") + + model = Parametric(self, "MLE", None, False, False, False) + p = (x_arr * n_arr).sum() / n_arr.sum() + model.params = np.array([p]) + return model + + # Narrower than ParametricFitter.from_params, which takes + # (params, gamma, p, f0). Unlike `fit`, this one is not resolved + # by the OptimisedFitMixin split: every distribution has a + # from_params. It is a parameter *rename* -- the base's `params` + # became `p` -- so positional calls work and keyword calls + # raise. Worse here: the base's `p` means the + # limited-failure proportion, so the same keyword means two + # unrelated things across sibling classes. Fixing it means renaming + # back, with a deprecation alias, and is tracked separately. + def from_params( + self, + params: npt.ArrayLike, + gamma: Boxable | None = None, + p: Boxable | None = None, + f0: Boxable | None = None, + ) -> Parametric: + """Create a Bernoulli model from its event probability. + + Parameters + ---------- + params : scalar + The event probability, between 0 and 1. + gamma, p, f0 : None + Accepted so the signature matches + :meth:`ParametricFitter.from_params`, and rejected: a + Bernoulli has no offset, limited failure population or zero + inflation. Note that the base's ``p`` is the *never-fails* + proportion, not this distribution's parameter -- which is why + the parameter is ``params`` and not ``p``. + """ + reject_structural_params(self.name, gamma, p, f0) + prob = float(np.squeeze(np.asarray(params))) + + if prob > 1: + raise ValueError("'params' must be less than 1") + + if prob < 0: + raise ValueError("'params' must be greater than 0") + + model = Parametric(self, "given parameters", None, False, False, False) + model.params = np.atleast_1d(prob) + return model + + +FixedEventProbability: FixedEventProbability_ = FixedEventProbability_( + "FixedEventProbability" +) diff --git a/surpyval/univariate/parametric/distributions/gamma.py b/surpyval/univariate/parametric/distributions/gamma.py index 73db0f9..537506e 100755 --- a/surpyval/univariate/parametric/distributions/gamma.py +++ b/surpyval/univariate/parametric/distributions/gamma.py @@ -1,15 +1,19 @@ +import numpy.typing as npt from autograd.scipy.special import gamma as agamma from autograd.scipy.special import gammaln as agammaln from scipy.special import digamma, gammaincinv from surpyval import np from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) from surpyval.utils.autograd_gamma_compat import gammainc as agammainc from surpyval.utils.autograd_gamma_compat import gammainccln as agammainccln from surpyval.utils.autograd_gamma_compat import gammaincln as agammaincln +from surpyval.utils.surpyval_data import SurpyvalData class Gamma_(OptimisedFitMixin, ParametricFitter): @@ -23,7 +27,7 @@ class Gamma_(OptimisedFitMixin, ParametricFitter): """ - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=2, @@ -55,7 +59,7 @@ def __init__(self, name): self.supports_mpp = False @staticmethod - def _moment_estimate(x): + def _moment_estimate(x: npt.NDArray) -> tuple[float, float]: """Closed-form approximation to the Gamma MLE. The shape solves ``log(alpha) - digamma(alpha) = s`` with @@ -77,7 +81,10 @@ def _moment_estimate(x): beta = x.sum() / (len(x) * alpha) return alpha, 1.0 / beta - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + x = data.x if offset: # ``gamma`` leads the vector, as it does for every other # offset-capable distribution. Returning it last put the @@ -93,10 +100,10 @@ def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): # silent nonsense. gamma_init = np.min(x) - 1.0 alpha, beta = self._moment_estimate(x - gamma_init) - return gamma_init, alpha, beta - return self._moment_estimate(x) + return np.array([gamma_init, alpha, beta], dtype=float) + return np.asarray(self._moment_estimate(x), dtype=float) - def sf(self, x, alpha, beta): + def sf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Survival (or Reliability) function for the Gamma Distribution: @@ -132,44 +139,7 @@ def sf(self, x, alpha, beta): """ return 1 - self.ff(x, alpha, beta) - def cs(self, x, X, alpha, beta): - r""" - - Conditional survival function for the Gamma Distribution: - - .. math:: - R(x) = e^{-\lambda x} - - Parameters - ---------- - - x : numpy array or scalar - The value(s) at which the function will be calculated - X : numpy array or scalar - The value(s) at which each value(s) in x was known to have survived - alpha : numpy array or scalar - The shape parameter for the Gamma distribution - beta : numpy array or scalar - The scale parameter for the Gamma distribution - - Returns - ------- - - cs : scalar or numpy array - the conditional survival probability. - - Examples - -------- - >>> import numpy as np - >>> from surpyval import Gamma - >>> x = np.array([1, 2, 3, 4, 5]) - >>> Gamma.cs(x, 5, 3, 4) - array([2.59402488e-02, 6.39048747e-04, 1.51519143e-05, 3.48776510e-07, - 7.79933496e-09]) - """ - return self.sf(x + X, alpha, beta) / self.sf(X, alpha, beta) - - def ff(self, x, alpha, beta): + def ff(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" CDF (or unreliability or failure) function for the Gamma Distribution: @@ -206,7 +176,7 @@ def ff(self, x, alpha, beta): x = np.array(x) return agammainc(alpha, beta * x) - def df(self, x, alpha, beta): + def df(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Density function for the Gamma Distribution: @@ -247,7 +217,7 @@ def df(self, x, alpha, beta): / (agamma(alpha)) ) - def hf(self, x, alpha, beta): + def hf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Instantaneous hazard rate for the Gamma Distribution: @@ -284,7 +254,7 @@ def hf(self, x, alpha, beta): """ return self.df(x, alpha, beta) / self.sf(x, alpha, beta) - def Hf(self, x, alpha, beta): + def Hf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Cumulative hazard rate for the Gamma Distribution: @@ -320,18 +290,18 @@ def Hf(self, x, alpha, beta): """ return -np.log(self.sf(x, alpha, beta)) - def qf(self, p, alpha, beta): + def qf(self, u: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Quantile function for the Gamma Distribution: .. math:: - q(p) = \frac{-\ln\left ( p \right )}{\lambda} + q(u) = \frac{-\ln\left ( u \right )}{\lambda} Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated alpha : numpy array or scalar The shape parameter for the Gamma distribution @@ -342,19 +312,19 @@ def qf(self, p, alpha, beta): ------- q : scalar or numpy array - The quantiles for the Gamma distribution at each value p. + The quantiles for the Gamma distribution at each value u. Examples -------- >>> import numpy as np >>> from surpyval import Gamma - >>> p = np.array([.1, .2, .3, .4, .5]) - >>> Gamma.qf(p, 3, 4) + >>> u = np.array([.1, .2, .3, .4, .5]) + >>> Gamma.qf(u, 3, 4) array([0.27551633, 0.38376105, 0.47844395, 0.57126923, 0.66851508]) """ - return gammaincinv(alpha, p) / beta + return gammaincinv(alpha, u) / beta - def mean(self, alpha, beta): + def mean(self, alpha: Boxable, beta: Boxable) -> Boxable: r""" Calculates the mean of the Gamma distribution with given parameters. @@ -384,20 +354,20 @@ def mean(self, alpha, beta): """ return alpha / beta - def moment(self, n, alpha, beta): + def moment(self, m: int, alpha: Boxable, beta: Boxable) -> Boxable: r""" - Calculates the n-th moment of the Gamma distribution with + Calculates the m-th moment of the Gamma distribution with given parameters. .. math:: - E = \frac{\Gamma \left ( n + \alpha \right )}{\beta^{n}\Gamma + E = \frac{\Gamma \left ( m + \alpha \right )}{\beta^{m}\Gamma \left ( \alpha \right )} Parameters ---------- - n : integer or numpy array of integers + m : integer The ordinal of the moment to calculate alpha : numpy array or scalar The shape parameter for the Gamma distribution @@ -416,9 +386,9 @@ def moment(self, n, alpha, beta): >>> Gamma.moment(3, 3, 4) np.float64(0.9375) """ - return agamma(n + alpha) / (beta**n * agamma(alpha)) + return agamma(m + alpha) / (beta**m * agamma(alpha)) - def entropy(self, alpha, beta): + def entropy(self, alpha: Boxable, beta: Boxable) -> Boxable: r""" Calculates the entropy of the Gamma distribution. @@ -457,7 +427,7 @@ def entropy(self, alpha, beta): + (1 - alpha) * digamma(alpha) ) - def log_df(self, x, alpha, beta): + def log_df(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Calculates the log of the density function of the Gamma distribution @@ -491,22 +461,22 @@ def log_df(self, x, alpha, beta): - agammaln(alpha) ) - def log_ff(self, x, alpha, beta): + def log_ff(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: return agammaincln(alpha, beta * x) - def log_sf(self, x, alpha, beta): + def log_sf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: return agammainccln(alpha, beta * x) - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: alpha = params[0] return gammaincinv(alpha, y) - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: alpha = params[0] return agammainc(alpha, y) - def mpp_x_transform(self, x, gamma=0): - return x - gamma + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: + return x Gamma: Gamma_ = Gamma_("Gamma") diff --git a/surpyval/univariate/parametric/distributions/geometric.py b/surpyval/univariate/parametric/distributions/geometric.py index ad70e19..029979b 100644 --- a/surpyval/univariate/parametric/distributions/geometric.py +++ b/surpyval/univariate/parametric/distributions/geometric.py @@ -10,6 +10,7 @@ Numeric, OptimisedFitMixin, ) +from surpyval.utils.surpyval_data import SurpyvalData class Geometric_(OptimisedFitMixin, DiscreteParametricFitter): @@ -46,15 +47,11 @@ def __init__(self, name: str) -> None: ) def _parameter_initialiser( - self, - x: npt.NDArray, - c: npt.NDArray | None = None, - n: npt.NDArray | None = None, - t: npt.NDArray | None = None, - offset: bool = False, + self, data: SurpyvalData, offset: bool = False ) -> npt.NDArray: # Method-of-moments seed: the mean of a geometric on {1, 2, ...} is # 1 / p, so p ~ 1 / mean(x). Kept inside (0, 1). + x = data.x finite = x[np.isfinite(x)] mean = finite.mean() if finite.size else 2.0 p = 1.0 / max(mean, 1.0 + 1e-8) @@ -62,29 +59,45 @@ def _parameter_initialiser( def sf(self, x: Numeric, p: Boxable) -> Boxable: r"""Survival function :math:`R(k) = (1 - p)^{k}`.""" - return (1.0 - p) ** x + # Nothing can fail before the first trial, so R = 1 below zero. + # The algebraic form returns 1/(1 - p) there -- a survival above + # one, which ``hf`` used to divide by. + return np.where(x < 0.0, 1.0, (1.0 - p) ** x) def ff(self, x: Numeric, p: Boxable) -> Boxable: r"""CDF :math:`F(k) = 1 - (1 - p)^{k}`.""" - return 1.0 - (1.0 - p) ** x + return 1.0 - self.sf(x, p) def df(self, x: Numeric, p: Boxable) -> Boxable: - r"""PMF :math:`P(T = k) = (1 - p)^{k - 1}\,p`.""" - return (1.0 - p) ** (x - 1.0) * p + r"""PMF :math:`P(T = k) = (1 - p)^{k - 1}\,p`, zero below ``k = 1``.""" + # The algebraic form does not know where the support starts: at + # k = 0 it evaluates to p/(1 - p), a positive "probability" below + # the first mass point (0.43 at p = 0.3), and it grows without + # bound as k decreases. The fitter's interior check keeps such a + # value out of a likelihood, but df is public and a caller + # plotting a pmf from zero would get it. + return np.where(x < 1.0, 0.0, (1.0 - p) ** (x - 1.0) * p) def hf(self, x: Numeric, p: Boxable) -> Boxable: r"""Discrete hazard :math:`h(k) = p` (constant, memoryless).""" - return np.ones_like(x, dtype=float) * p + # Constant on the support, but zero below it: h(k) = P(T = k)/R(k - 1) + # and there is no mass to condition on before k = 1. + return np.where(x < 1.0, 0.0, np.ones_like(x, dtype=float) * p) def Hf(self, x: Numeric, p: Boxable) -> Boxable: r"""Cumulative hazard :math:`H(k) = -\ln R(k) = -k\ln(1 - p)`.""" - return -x * np.log(1.0 - p) + return np.where(x < 0.0, 0.0, -x * np.log(1.0 - p)) def qf(self, u: Numeric, p: Boxable) -> Boxable: r"""Quantile: the smallest integer ``k`` with :math:`F(k) \geq u`.""" u = np.asarray(u, dtype=float) - q = np.ceil(np.log1p(-u) / np.log(1.0 - p)) - return np.maximum(q, 1.0) + k = np.log1p(-u) / np.log(1.0 - p) + # A caller inverting the CDF passes u = F(k), which was formed as + # 1 - (1 - p)^k. Recovering k from it lands a few ulp above the + # integer, and a bare ceil() then answers k + 1 -- so F and its + # quantile did not invert each other. Snap first. + k = np.where(np.abs(k - np.round(k)) < 1e-9, np.round(k), k) + return np.maximum(np.ceil(k), 1.0) def mean(self, p: Boxable) -> Boxable: return 1.0 / p @@ -96,15 +109,20 @@ def moment(self, m: int, p: Boxable) -> Boxable: k = np.arange(1, upper + 1, dtype=float) return np.sum(k**m * self.df(k, p)) - def random(self, size: int | tuple[int, ...], p: Boxable) -> Boxable: + def random(self, size: int | tuple[int, ...], p: Boxable) -> npt.NDArray: U = uniform.rvs(size=size) - return self.qf(U, p) + # qf is declared Boxable because a fit differentiates it; + # sampling never does, so this is always a real array. + return np.asarray(self.qf(U, p)) def log_df(self, x: Numeric, p: Boxable) -> Boxable: - return (x - 1.0) * np.log(1.0 - p) + np.log(p) + # -inf below the support, matching ``df``'s zero. + return np.where( + x < 1.0, -np.inf, (x - 1.0) * np.log(1.0 - p) + np.log(p) + ) def log_sf(self, x: Numeric, p: Boxable) -> Boxable: - return x * np.log(1.0 - p) + return np.where(x < 0.0, 0.0, x * np.log(1.0 - p)) Geometric = Geometric_("Geometric") diff --git a/surpyval/univariate/parametric/distributions/gumbel.py b/surpyval/univariate/parametric/distributions/gumbel.py index a054d36..7583188 100755 --- a/surpyval/univariate/parametric/distributions/gumbel.py +++ b/surpyval/univariate/parametric/distributions/gumbel.py @@ -1,15 +1,19 @@ +import numpy.typing as npt from numpy import euler_gamma from scipy.stats import gumbel_l from surpyval import np from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData class Gumbel_(OptimisedFitMixin, ParametricFitter): - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=2, @@ -20,14 +24,21 @@ def __init__(self, name): plot_x_scale="linear", ) - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): - if (2 in c) or (-1 in c): + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + if (2 in data.c) or (-1 in data.c): heuristic = "Turnbull" else: heuristic = "Nelson-Aalen" - return self.fit(x, c, n, how="MPP", heuristic=heuristic).params + return np.asarray( + self.fit_from_surpyval_data( + data, how="MPP", heuristic=heuristic + ).params, + dtype=float, + ) - def sf(self, x, mu, sigma): + def sf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Survival (or Reliability) function for the Gumbel Distribution: @@ -66,7 +77,7 @@ def sf(self, x, mu, sigma): """ return np.exp(-np.exp((x - mu) / sigma)) - def ff(self, x, mu, sigma): + def ff(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" CDF (or Failure) function for the Gumbel Distribution: @@ -105,7 +116,7 @@ def ff(self, x, mu, sigma): """ return -np.expm1(-self.Hf(x, mu, sigma)) - def df(self, x, mu, sigma): + def df(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Density function (pdf) for the Gumbel Distribution: @@ -145,7 +156,7 @@ def df(self, x, mu, sigma): z = (x - mu) / sigma return (1 / sigma) * np.exp(z - np.exp(z)) - def hf(self, x, mu, sigma): + def hf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Instantaneous hazard rate for the Gumbel Distribution: @@ -182,7 +193,7 @@ def hf(self, x, mu, sigma): z = (x - mu) / sigma return (1 / sigma) * np.exp(z) - def Hf(self, x, mu, sigma): + def Hf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Cumulative hazard rate for the Gumbel Distribution: @@ -218,18 +229,18 @@ def Hf(self, x, mu, sigma): """ return np.exp((x - mu) / sigma) - def qf(self, p, mu, sigma): + def qf(self, u: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Quantile function for the Gumbel Distribution: .. math:: - q(p) = \mu + \sigma\ln\left ( -\ln\left ( 1 - p \right ) \right ) + q(u) = \mu + \sigma\ln\left ( -\ln\left ( 1 - u \right ) \right ) Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated mu : numpy array like or scalar The location parameter(s) of the distribution @@ -240,19 +251,19 @@ def qf(self, p, mu, sigma): ------- q : scalar or numpy array - The quantiles for the Gumbel distribution at each value p. + The quantiles for the Gumbel distribution at each value u. Examples -------- >>> import numpy as np >>> from surpyval import Gumbel - >>> p = np.array([0.1, 0.3, 0.5]) - >>> Gumbel.qf(p, 3, 2) + >>> u = np.array([0.1, 0.3, 0.5]) + >>> Gumbel.qf(u, 3, 2) array([-1.50073465, 0.93813913, 2.26697416]) """ - return mu + sigma * (np.log(-np.log1p(-p))) + return mu + sigma * (np.log(-np.log1p(-u))) - def mean(self, mu, sigma): + def mean(self, mu: Boxable, sigma: Boxable) -> Boxable: r""" Calculates the mean of the Gumbel distribution with given parameters. @@ -286,20 +297,20 @@ def mean(self, mu, sigma): """ return mu - sigma * euler_gamma - def log_df(self, x, mu, sigma): + def log_df(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: z = (x - mu) / sigma return z - np.exp(z) - np.log(sigma) - def log_sf(self, x, mu, sigma): + def log_sf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: return -self.Hf(x, mu, sigma) - def log_ff(self, x, mu, sigma): + def log_ff(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: return np.log(-np.expm1(-self.Hf(x, mu, sigma))) - def moment(self, n, mu, sigma): - return gumbel_l.moment(n, loc=mu, scale=sigma) + def moment(self, m: int, mu: Boxable, sigma: Boxable) -> Boxable: + return gumbel_l.moment(m, loc=mu, scale=sigma) - def entropy(self, mu, sigma): + def entropy(self, mu: Boxable, sigma: Boxable) -> Boxable: r""" Calculates the entropy of the Gumbel distribution. @@ -331,20 +342,22 @@ def entropy(self, mu, sigma): """ return np.log(sigma) + euler_gamma + 1 - def mpp_x_transform(self, x, gamma=0): - return x - gamma + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: + return x - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: mask = (y == 0) | (y == 1) out = np.zeros_like(y) out[~mask] = np.log(-np.log(1 - y[~mask])) out[mask] = np.nan return out - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return 1 - np.exp(-np.exp(y)) - def unpack_rr(self, params, rr): + def unpack_rr( + self, params: npt.NDArray, rr: str + ) -> tuple[Boxable, Boxable]: if rr == "y": sigma = 1.0 / params[0] mu = -sigma * params[1] diff --git a/surpyval/univariate/parametric/distributions/gumbel_lev.py b/surpyval/univariate/parametric/distributions/gumbel_lev.py index e1c1765..29aac88 100644 --- a/surpyval/univariate/parametric/distributions/gumbel_lev.py +++ b/surpyval/univariate/parametric/distributions/gumbel_lev.py @@ -1,15 +1,19 @@ +import numpy.typing as npt from numpy import euler_gamma from scipy.stats import gumbel_r from surpyval import np from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData class GumbelLEV_(OptimisedFitMixin, ParametricFitter): - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=2, @@ -20,11 +24,18 @@ def __init__(self, name): plot_x_scale="linear", ) - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: heuristic = "Fleming-Harrington" - return self.fit(x, c, n, how="MPP", heuristic=heuristic).params + return np.asarray( + self.fit_from_surpyval_data( + data, how="MPP", heuristic=heuristic + ).params, + dtype=float, + ) - def sf(self, x, mu, sigma): + def sf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Survival (or reliability) function for the Gumbel LEV Distribution: @@ -63,7 +74,7 @@ def sf(self, x, mu, sigma): """ return 1 - self.ff(x, mu, sigma) - def ff(self, x, mu, sigma): + def ff(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" CDF (or Failure) function for the Gumbel LEV Distribution: @@ -103,7 +114,7 @@ def ff(self, x, mu, sigma): z = (x - mu) / sigma return np.exp(-np.exp(-z)) - def df(self, x, mu, sigma): + def df(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Density function (pdf) for the Gumbel LEV Distribution: @@ -143,7 +154,7 @@ def df(self, x, mu, sigma): z = (x - mu) / sigma return (1.0 / sigma) * np.exp(-(z + np.exp(-z))) - def hf(self, x, mu, sigma): + def hf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Instantaneous hazard rate for the Gumbel LEV Distribution: @@ -179,7 +190,7 @@ def hf(self, x, mu, sigma): """ return self.df(x, mu, sigma) / self.sf(x, mu, sigma) - def Hf(self, x, mu, sigma): + def Hf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Cumulative hazard rate for the Gumbel LEV Distribution: @@ -215,18 +226,18 @@ def Hf(self, x, mu, sigma): """ return -np.log(self.sf(x, mu, sigma)) - def qf(self, p, mu, sigma): + def qf(self, u: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Quantile function for the Gumbel LEV Distribution: .. math:: - q(p) = \mu - \sigma\ln\left ( -\ln\left ( p \right ) \right ) + q(u) = \mu - \sigma\ln\left ( -\ln\left ( u \right ) \right ) Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated mu : numpy array like or scalar The location parameter(s) of the distribution @@ -237,19 +248,19 @@ def qf(self, p, mu, sigma): ------- q : scalar or numpy array - The quantiles for the GumbelLEV distribution at each value p. + The quantiles for the GumbelLEV distribution at each value u. Examples -------- >>> import numpy as np >>> from surpyval import GumbelLEV - >>> p = np.array([.1, .2, .3, .4, .5]) - >>> GumbelLEV.qf(p, 3, 2) + >>> u = np.array([.1, .2, .3, .4, .5]) + >>> GumbelLEV.qf(u, 3, 2) array([1.33193511, 2.04823001, 2.62874648, 3.17484314, 3.73302584]) """ - return mu - sigma * np.log(-np.log(p)) + return mu - sigma * np.log(-np.log(u)) - def mean(self, mu, sigma): + def mean(self, mu: Boxable, sigma: Boxable) -> Boxable: r""" Calculates the mean of the Gumbel LEV distribution with given @@ -282,35 +293,35 @@ def mean(self, mu, sigma): """ return mu + sigma * euler_gamma - def log_sf(self, x, mu, sigma): + def log_sf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: return -self.Hf(x, mu, sigma) - def log_ff(self, x, mu, sigma): + def log_ff(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: return -np.exp(-(x - mu) / sigma) - def log_df(self, x, mu, sigma): + def log_df(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: z = (x - mu) / sigma return -np.log(sigma) - (z + np.exp(-z)) - def mpp_x_transform(self, x, gamma=0): - return x - gamma + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: + return x - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: mask = (y == 0) | (y == 1) out = np.zeros_like(y) out[~mask] = -np.log(-np.log(y[~mask])) out[mask] = np.nan return out - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: # Inverse of the LEV transform y = -log(-log(F)) is # F = exp(-exp(-y)); the previous form was the SEV inverse (#257). return np.exp(-np.exp(-y)) - def moment(self, n, mu, sigma): - return gumbel_r.moment(n, loc=mu, scale=sigma) + def moment(self, m: int, mu: Boxable, sigma: Boxable) -> Boxable: + return gumbel_r.moment(m, loc=mu, scale=sigma) - def entropy(self, mu, sigma): + def entropy(self, mu: Boxable, sigma: Boxable) -> Boxable: r""" Calculates the entropy of the Gumbel LEV distribution. @@ -342,7 +353,9 @@ def entropy(self, mu, sigma): """ return np.log(sigma) + euler_gamma + 1 - def unpack_rr(self, params, rr): + def unpack_rr( + self, params: npt.NDArray, rr: str + ) -> tuple[Boxable, Boxable]: if rr == "y": sigma = 1.0 / params[0] mu = -sigma * params[1] diff --git a/surpyval/univariate/parametric/distributions/logistic.py b/surpyval/univariate/parametric/distributions/logistic.py index 22a21d9..c6ea302 100755 --- a/surpyval/univariate/parametric/distributions/logistic.py +++ b/surpyval/univariate/parametric/distributions/logistic.py @@ -1,15 +1,19 @@ +import numpy.typing as npt from autograd import grad from autograd.scipy.special import beta as abeta from surpyval import np from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData class Logistic_(OptimisedFitMixin, ParametricFitter): - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=2, @@ -23,10 +27,14 @@ def __init__(self, name): plot_x_scale="linear", ) - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): - return self.fit(x, c, n, how="MPP").params + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + return np.asarray( + self.fit_from_surpyval_data(data, how="MPP").params, dtype=float + ) - def sf(self, x, mu, sigma): + def sf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Survival (or reliability) function for the Logistic Distribution: @@ -62,7 +70,7 @@ def sf(self, x, mu, sigma): exp_term = np.exp(-(x - mu) / sigma) return exp_term / (1 + exp_term) - def ff(self, x, mu, sigma): + def ff(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Failure (CDF or unreliability) function for the Logistic Distribution: @@ -97,7 +105,7 @@ def ff(self, x, mu, sigma): z = (x - mu) / sigma return 1.0 / (1 + np.exp(-z)) - def df(self, x, mu, sigma): + def df(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Failure (CDF or unreliability) function for the Logistic Distribution: @@ -134,7 +142,7 @@ def df(self, x, mu, sigma): z = (x - mu) / sigma return np.exp(-z) / (sigma * (1 + np.exp(-z)) ** 2) - def hf(self, x, mu, sigma): + def hf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Instantaneous hazard rate for the Logistic Distribution: @@ -168,7 +176,7 @@ def hf(self, x, mu, sigma): """ return self.df(x, mu, sigma) / self.sf(x, mu, sigma) - def Hf(self, x, mu, sigma): + def Hf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Cumulative hazard rate for the Logistic distribution: @@ -202,18 +210,18 @@ def Hf(self, x, mu, sigma): """ return -np.log(self.sf(x, mu, sigma)) - def qf(self, p, mu, sigma): + def qf(self, u: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Quantile function for the Logistic distribution: .. math:: - q(p) = \\mu + \\sigma \\ln \\left ( \\frac{p}{1 - p} \\right) + q(u) = \\mu + \\sigma \\ln \\left ( \\frac{u}{1 - u} \\right) Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated mu : numpy array or scalar The location parameter for the Logistic distribution @@ -224,19 +232,19 @@ def qf(self, p, mu, sigma): ------- q : scalar or numpy array - The quantiles for the Logistic distribution at each value p + The quantiles for the Logistic distribution at each value u Examples -------- >>> import numpy as np >>> from surpyval import Logistic - >>> p = np.array([0.1, 0.2, 0.3, 0.4]) - >>> Logistic.qf(p, 3, 4) + >>> u = np.array([0.1, 0.2, 0.3, 0.4]) + >>> Logistic.qf(u, 3, 4) array([-5.78889831, -2.54517744, -0.38919144, 1.37813957]) """ - return mu + sigma * (np.log(p) - np.log1p(-p)) + return mu + sigma * (np.log(u) - np.log1p(-u)) - def mean(self, mu, sigma): + def mean(self, mu: Boxable, sigma: Boxable) -> Boxable: r""" Mean of the Logistic distribution @@ -266,30 +274,69 @@ def mean(self, mu, sigma): """ return mu - def log_df(self, x, mu, sigma): + def log_df(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: # logaddexp(0, -z) = log(1 + e^-z) without overflowing exp for # z < -709 (#257). z = (x - mu) / sigma return -(z + np.log(sigma) + 2 * np.logaddexp(0.0, -z)) - def log_sf(self, x, mu, sigma): + def log_sf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: z = (x - mu) / sigma return -(z + np.logaddexp(0.0, -z)) - def log_ff(self, x, mu, sigma): + def log_ff(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: z = (x - mu) / sigma return -np.logaddexp(0.0, -z) - def mgf(self, t, mu, sigma): + # Private: the only reason this exists is `moment` below, which + # differentiates it. It was the one public `mgf` on any distribution + # in the package, which read as a surface other distributions were + # missing rather than as this one's internal machinery. + def _mgf(self, t: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: return np.exp(mu * t) * abeta(1 + sigma * t, 1 - sigma * t) - def moment(self, n, mu, sigma): - d = self.mgf - for i in range(n): + def moment(self, m: int, mu: Boxable, sigma: Boxable) -> Boxable: + r""" + + m-th (non central) moment of the Logistic distribution. + + Obtained by differentiating the moment generating function + + .. math:: + M(t) = e^{\mu t} B(1 + \sigma t, 1 - \sigma t) + + ``m`` times at :math:`t = 0`, which is the definition of the raw + moment. The general closed form needs Bernoulli numbers, so + autograd differentiating the MGF is both shorter and exact. + + Parameters + ---------- + + m : integer + The ordinal of the moment to calculate + mu : numpy array or scalar + The location parameter of the distribution + sigma : numpy array or scalar + The scale parameter of the distribution + + Returns + ------- + + moment : scalar or numpy array + The moment(s) of the Logistic distribution + + Examples + -------- + >>> from surpyval import Logistic + >>> Logistic.moment(2, 3, 4) + np.float64(61.63789013914325) + """ + d = self._mgf + for _ in range(m): d = grad(d) return d(0.0, mu, sigma) - def entropy(self, mu, sigma): + def entropy(self, mu: Boxable, sigma: Boxable) -> Boxable: r""" Calculates the entropy of the Logistic distribution. @@ -319,20 +366,22 @@ def entropy(self, mu, sigma): """ return np.log(sigma) + 2 - def mpp_x_transform(self, x, gamma=0): - return x - gamma + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: + return x - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: mask = (y == 0) | (y == 1) out = np.zeros_like(y) out[~mask] = -np.log(1.0 / y[~mask] - 1) out[mask] = np.nan return out - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return 1.0 / (np.exp(-y) + 1) - def unpack_rr(self, params, rr): + def unpack_rr( + self, params: npt.NDArray, rr: str + ) -> tuple[Boxable, Boxable]: if rr == "y": sigma = 1.0 / params[0] mu = -sigma * params[1] diff --git a/surpyval/univariate/parametric/distributions/loglogistic.py b/surpyval/univariate/parametric/distributions/loglogistic.py index 131e8c3..20f36ba 100755 --- a/surpyval/univariate/parametric/distributions/loglogistic.py +++ b/surpyval/univariate/parametric/distributions/loglogistic.py @@ -1,15 +1,18 @@ +import numpy.typing as npt from scipy.stats import fisk from surpyval import np from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) -from surpyval.utils import xcnt_handler +from surpyval.utils.surpyval_data import SurpyvalData class LogLogistic_(OptimisedFitMixin, ParametricFitter): - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=2, @@ -20,17 +23,29 @@ def __init__(self, name): plot_x_scale="log", ) - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: if offset: - x, c, n, _ = xcnt_handler(x, c, n, t) + # The data arrives already validated in xcnt form, so the + # ``xcnt_handler`` round trip that used to open this branch + # is gone. It never had anything to re-derive: no caller has + # ever passed ``t`` down to an initialiser. + x, c, n = data.x, data.c, data.n flag = (c == 0).astype(int) value_range = np.max(x) - np.min(x) gamma_init = np.min(x) - value_range / 10 - return gamma_init, x.sum() / (n * flag).sum(), 2.0 + return np.array( + [gamma_init, x.sum() / (n * flag).sum(), 2.0], + dtype=float, + ) else: - return self.fit(x, c, n, how="MPP").params + return np.asarray( + self.fit_from_surpyval_data(data, how="MPP").params, + dtype=float, + ) - def sf(self, x, alpha, beta): + def sf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Survival (or reliability) function for the LogLogistic Distribution: @@ -67,43 +82,7 @@ def sf(self, x, alpha, beta): # of raising/NaN-ing on the negative power (#280). return 1.0 / (1.0 + (x / alpha) ** beta) - def cs(self, x, X, alpha, beta): - r""" - - Conditional survival function for the LogLogistic Distribution: - - .. math:: - R(x, X) = \frac{R(x + X)}{R(X)} - - Parameters - ---------- - - x : numpy array or scalar - The values at which the function will be calculated - X : numpy array or scalar - The value(s) at which each value(s) in x was known to have survived - alpha : numpy array or scalar - scale parameter for the LogLogistic distribution - beta : numpy array or scalar - shape parameter for the LogLogistic distribution - - Returns - ------- - - cs : scalar or numpy array - The value(s) of the conditional survival function at x. - - Examples - -------- - >>> import numpy as np - >>> from surpyval import LogLogistic - >>> x = np.array([1, 2, 3, 4, 5]) - >>> LogLogistic.cs(x, 5, 3, 4) - array([0.51270879, 0.28444803, 0.16902083, 0.10629329, 0.07003273]) - """ - return self.sf(x + X, alpha, beta) / self.sf(X, alpha, beta) - - def ff(self, x, alpha, beta): + def ff(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Failure (CDF or unreliability) function for the LogLogistic @@ -141,7 +120,7 @@ def ff(self, x, alpha, beta): z = (x / alpha) ** beta return z / (1.0 + z) - def df(self, x, alpha, beta): + def df(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Density function for the LogLogistic Distribution: @@ -179,7 +158,7 @@ def df(self, x, alpha, beta): (1.0 + (x / alpha) ** beta) ** 2.0 ) - def hf(self, x, alpha, beta): + def hf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Instantaneous hazard rate for the LogLogistic Distribution: @@ -213,7 +192,7 @@ def hf(self, x, alpha, beta): """ return self.df(x, alpha, beta) / self.sf(x, alpha, beta) - def Hf(self, x, alpha, beta): + def Hf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Cumulative hazard rate for the LogLogistic Distribution: @@ -247,18 +226,18 @@ def Hf(self, x, alpha, beta): """ return -np.log(self.sf(x, alpha, beta)) - def qf(self, p, alpha, beta): + def qf(self, u: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Quantile function for the LogLogistic distribution: .. math:: - q(p) = \alpha \left ( \frac{p}{1 - p} \right )^{\frac{1}{\beta}} + q(u) = \alpha \left ( \frac{u}{1 - u} \right )^{\frac{1}{\beta}} Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated alpha : numpy array or scalar scale parameter for the LogLogistic distribution @@ -269,19 +248,19 @@ def qf(self, p, alpha, beta): ------- q : scalar or numpy array - The quantiles for the LogLogistic distribution at each value p + The quantiles for the LogLogistic distribution at each value u Examples -------- >>> import numpy as np >>> from surpyval import LogLogistic - >>> p = np.array([.1, .2, .3, .4, .5]) - >>> LogLogistic.qf(p, 3, 4) + >>> u = np.array([.1, .2, .3, .4, .5]) + >>> LogLogistic.qf(u, 3, 4) array([1.73205081, 2.12132034, 2.42732013, 2.71080601, 3. ]) """ - return alpha * (p / (1 - p)) ** (1.0 / beta) + return alpha * (u / (1 - u)) ** (1.0 / beta) - def mean(self, alpha, beta): + def mean(self, alpha: Boxable, beta: Boxable) -> Boxable: r""" Mean of the LogLogistic distribution @@ -314,14 +293,14 @@ def mean(self, alpha, beta): else: return np.nan - def log_df(self, x, alpha, beta): + def log_df(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: return ( np.log(beta / alpha) + (beta - 1) * np.log(x / alpha) - 2 * np.log(1 + (x / alpha) ** beta) ) - def log_sf(self, x, alpha, beta): + def log_sf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: # logaddexp form: log(alpha^beta + x^beta) overflows for # beta*log(alpha) or beta*log(x) beyond ~709 even when the log # probability itself is modest (#280). @@ -329,25 +308,27 @@ def log_sf(self, x, alpha, beta): lx = beta * np.log(x) return la - np.logaddexp(la, lx) - def log_ff(self, x, alpha, beta): + def log_ff(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: la = beta * np.log(alpha) lx = beta * np.log(x) return lx - np.logaddexp(la, lx) - def mpp_x_transform(self, x, gamma=0): - return np.log(x - gamma) + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: + return np.log(x) - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: mask = (y == 0) | (y == 1) out = np.zeros_like(y) out[~mask] = -np.log(1.0 / y[~mask] - 1) out[mask] = np.nan return out - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return 1.0 / (np.exp(-y) + 1) - def unpack_rr(self, params, rr): + def unpack_rr( + self, params: npt.NDArray, rr: str + ) -> tuple[Boxable, Boxable]: if rr == "y": beta = params[0] alpha = np.exp(params[1] / -beta) @@ -356,10 +337,10 @@ def unpack_rr(self, params, rr): alpha = np.exp(params[1] / (beta * params[0])) return alpha, beta - def moment(self, n, alpha, beta): - return fisk.moment(n, beta, scale=alpha) + def moment(self, m: int, alpha: Boxable, beta: Boxable) -> Boxable: + return fisk.moment(m, beta, scale=alpha) - def entropy(self, alpha, beta): + def entropy(self, alpha: Boxable, beta: Boxable) -> Boxable: r""" Calculates the entropy of the LogLogistic distribution. diff --git a/surpyval/univariate/parametric/distributions/lognormal.py b/surpyval/univariate/parametric/distributions/lognormal.py index f262105..4ca99c4 100755 --- a/surpyval/univariate/parametric/distributions/lognormal.py +++ b/surpyval/univariate/parametric/distributions/lognormal.py @@ -1,3 +1,4 @@ +import numpy.typing as npt from autograd.scipy.stats import norm from scipy.stats import norm as scipy_norm @@ -8,13 +9,16 @@ weighted_mean_and_std, ) from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData class LogNormal_(OptimisedFitMixin, ParametricFitter): - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=2, @@ -42,7 +46,10 @@ def __init__(self, name): ], ) - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + x, c, n = data.x, data.c, data.n if offset: # Shift the data so the log transform is defined, then # initialise mu and sigma from the shifted data @@ -51,12 +58,12 @@ def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): np.log(x - gamma_init), c=c, n=n, how="MLE" ) mu, sigma = norm_mod.params - return gamma_init, mu, sigma + return np.array([gamma_init, mu, sigma], dtype=float) norm_mod = para.Normal.fit(np.log(x), c=c, n=n, how="MLE") mu, sigma = norm_mod.params - return mu, sigma + return np.array([mu, sigma], dtype=float) - def _closed_form_mle(self, data): + def _closed_form_mle(self, data: SurpyvalData) -> npt.NDArray | None: r"""Exact MLE on complete data: the Normal closed form applied to :math:`\log x`, since the parameters are those of the underlying normal. Censoring or truncation fall back to the optimiser for @@ -69,7 +76,7 @@ def _closed_form_mle(self, data): return None return weighted_mean_and_std(np.log(x), data.n) - def sf(self, x, mu, sigma): + def sf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Survival (or Reliability) function for the LogNormal Distribution: @@ -103,43 +110,7 @@ def sf(self, x, mu, sigma): """ return 1 - self.ff(x, mu, sigma) - def cs(self, x, X, mu, sigma): - r""" - - Conditional survival function for the LogNormal Distribution: - - .. math:: - R(x, X) = \frac{R(x + X)}{R(X)} - - Parameters - ---------- - - x : numpy array or scalar - The value(s) at which the function will be calculated - X : numpy array or scalar - The value(s) at which each value(s) in x was known to have survived - mu : numpy array or scalar - The location parameter for the LogNormal distribution - sigma : numpy array or scalar - The scale parameter for the LogNormal distribution - - Returns - ------- - - cs : scalar or numpy array - the conditional survival probability at x - - Examples - -------- - >>> import numpy as np - >>> from surpyval import LogNormal - >>> x = np.array([1, 2, 3, 4, 5]) - >>> LogNormal.cs(x, 5, 3, 4) - array([0.97287811, 0.9496515 , 0.92933892, 0.91129122, 0.89505592]) - """ - return self.sf(x + X, mu, sigma) / self.sf(X, mu, sigma) - - def ff(self, x, mu, sigma): + def ff(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Failure (CDF or unreliability) function for the LogNormal Distribution: @@ -173,7 +144,7 @@ def ff(self, x, mu, sigma): """ return norm.cdf(np.log(x), mu, sigma) - def df(self, x, mu, sigma): + def df(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Density function for the LogNormal Distribution: @@ -208,7 +179,7 @@ def df(self, x, mu, sigma): """ return 1.0 / x * norm.pdf(np.log(x), mu, sigma) - def hf(self, x, mu, sigma): + def hf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Instantaneous hazard rate for the LogNormal Distribution: @@ -242,7 +213,7 @@ def hf(self, x, mu, sigma): """ return self.df(x, mu, sigma) / self.sf(x, mu, sigma) - def Hf(self, x, mu, sigma): + def Hf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Cumulative hazard rate for the LogNormal Distribution: @@ -276,18 +247,18 @@ def Hf(self, x, mu, sigma): """ return -np.log(self.sf(x, mu, sigma)) - def qf(self, p, mu, sigma): + def qf(self, u: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Quantile function for the LogNormal Distribution: .. math:: - q(p) = e^{\mu + \sigma \Phi^{-1} \left( p \right )} + q(u) = e^{\mu + \sigma \Phi^{-1} \left( u \right )} Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated mu : numpy array or scalar The location parameter for the LogNormal distribution @@ -298,19 +269,19 @@ def qf(self, p, mu, sigma): ------- q : scalar or numpy array - The quantiles for the LogNormal distribution at each value p. + The quantiles for the LogNormal distribution at each value u. Examples -------- >>> import numpy as np >>> from surpyval import LogNormal - >>> p = np.array([0.1, 0.2, 0.3, 0.4]) - >>> LogNormal.qf(p, 3, 4) + >>> u = np.array([0.1, 0.2, 0.3, 0.4]) + >>> LogNormal.qf(u, 3, 4) array([0.11928899, 0.69316658, 2.46550819, 7.29078766]) """ - return np.exp(scipy_norm.ppf(p, mu, sigma)) + return np.exp(scipy_norm.ppf(u, mu, sigma)) - def mean(self, mu, sigma): + def mean(self, mu: Boxable, sigma: Boxable) -> Boxable: r""" Mean of the LogNormal Distribution: @@ -340,10 +311,10 @@ def mean(self, mu, sigma): """ return np.exp(mu + (sigma**2) / 2) - def moment(self, n, mu, sigma): + def moment(self, m: int, mu: Boxable, sigma: Boxable) -> Boxable: r""" - n-th (non central) moment of the LogNormal distribution + m-th (non central) moment of the LogNormal distribution .. math:: E = ... complicated. @@ -351,7 +322,7 @@ def moment(self, n, mu, sigma): Parameters ---------- - n : integer or numpy array of integers + m : integer The ordinal of the moment to calculate mu : numpy array or scalar The location parameter for the LogNormal distribution @@ -370,9 +341,9 @@ def moment(self, n, mu, sigma): >>> LogNormal.moment(2, 3, 4) np.float64(3.1855931757113756e+16) """ - return np.exp(n * mu + (n**2 * sigma**2) / 2) + return np.exp(m * mu + (m**2 * sigma**2) / 2) - def entropy(self, mu, sigma): + def entropy(self, mu: Boxable, sigma: Boxable) -> Boxable: r""" Calculates the entropy of the LogNormal distribution. @@ -402,25 +373,27 @@ def entropy(self, mu, sigma): """ return mu + 0.5 * np.log(2 * np.pi * np.e * sigma**2) - def log_df(self, x, mu, sigma): + def log_df(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: return -np.log(x) + norm.logpdf(np.log(x), mu, sigma) - def log_ff(self, x, mu, sigma): + def log_ff(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: return norm.logcdf(np.log(x), mu, sigma) - def log_sf(self, x, mu, sigma): + def log_sf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: return norm.logsf(np.log(x), mu, sigma) - def mpp_x_transform(self, x, gamma=0): - return np.log(x - gamma) + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: + return np.log(x) - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return para.Normal.qf(y, 0, 1) - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return para.Normal.ff(y, 0, 1) - def unpack_rr(self, params, rr): + def unpack_rr( + self, params: npt.NDArray, rr: str + ) -> tuple[Boxable, Boxable]: if rr == "y": sigma, mu = params mu = -mu / sigma @@ -429,7 +402,7 @@ def unpack_rr(self, params, rr): sigma, mu = params return mu, sigma - def _mom(self, x): + def _mom(self, x: npt.NDArray) -> tuple[float, float]: norm_mod = para.Normal.fit(np.log(x), how="MOM") mu, sigma = norm_mod.params return mu, sigma diff --git a/surpyval/univariate/parametric/distributions/negative_binomial.py b/surpyval/univariate/parametric/distributions/negative_binomial.py index 4d1fc00..d51d076 100644 --- a/surpyval/univariate/parametric/distributions/negative_binomial.py +++ b/surpyval/univariate/parametric/distributions/negative_binomial.py @@ -12,6 +12,7 @@ OptimisedFitMixin, ) from surpyval.utils.autograd_gamma_compat import betainc, betaincln +from surpyval.utils.surpyval_data import SurpyvalData class NegativeBinomial_(OptimisedFitMixin, DiscreteParametricFitter): @@ -52,17 +53,13 @@ def __init__(self, name: str) -> None: ) def _parameter_initialiser( - self, - x: npt.NDArray, - c: npt.NDArray | None = None, - n: npt.NDArray | None = None, - t: npt.NDArray | None = None, - offset: bool = False, + self, data: SurpyvalData, offset: bool = False ) -> npt.NDArray: # Method-of-moments seed from the shifted counts Y = T - 1: for the # negative binomial mean_Y = r(1-p)/p and var_Y = mean_Y / p, so # p = mean_Y / var_Y and r = mean_Y p / (1 - p). Falls back to a # neutral guess when the data are not overdispersed. + x = data.x finite = x[np.isfinite(x)] y = finite - 1.0 if finite.size else np.array([1.0]) mean_y = max(y.mean(), 1e-3) @@ -76,11 +73,16 @@ def _parameter_initialiser( def sf(self, x: Numeric, r: Boxable, p: Boxable) -> Boxable: r"""Survival function :math:`R(k) = I_{1-p}(k, r)`.""" - return betainc(x, r, 1.0 - p) + # R = 1 below the first mass point at k = 1. The incomplete beta's + # first argument must be positive, so it returns NaN for k < 0 + # rather than the 1 it happens to give at k = 0. + safe_x = np.where(x < 0.0, 1.0, x) + return np.where(x < 0.0, 1.0, betainc(safe_x, r, 1.0 - p)) def ff(self, x: Numeric, r: Boxable, p: Boxable) -> Boxable: r"""CDF :math:`F(k) = I_{p}(r, k)`.""" - return betainc(r, x, p) + safe_x = np.where(x < 0.0, 1.0, x) + return np.where(x < 0.0, 0.0, betainc(r, safe_x, p)) def df(self, x: Numeric, r: Boxable, p: Boxable) -> Boxable: r"""PMF :math:`P(T = k)`.""" @@ -112,16 +114,20 @@ def random( return nbinom.rvs(r, p, size=size) + 1.0 def log_df(self, x: Numeric, r: Boxable, p: Boxable) -> Boxable: - return ( - gammaln(x - 1.0 + r) + safe_x = np.where(x < 1.0, 1.0, x) + return np.where( + x < 1.0, + -np.inf, + gammaln(safe_x - 1.0 + r) - gammaln(r) - - gammaln(x) + - gammaln(safe_x) + r * np.log(p) - + (x - 1.0) * np.log(1.0 - p) + + (safe_x - 1.0) * np.log(1.0 - p), ) def log_sf(self, x: Numeric, r: Boxable, p: Boxable) -> Boxable: - return betaincln(x, r, 1.0 - p) + safe_x = np.where(x < 0.0, 1.0, x) + return np.where(x < 0.0, 0.0, betaincln(safe_x, r, 1.0 - p)) NegativeBinomial = NegativeBinomial_("NegativeBinomial") diff --git a/surpyval/univariate/parametric/distributions/normal.py b/surpyval/univariate/parametric/distributions/normal.py index 00d82a3..98a01d0 100755 --- a/surpyval/univariate/parametric/distributions/normal.py +++ b/surpyval/univariate/parametric/distributions/normal.py @@ -1,3 +1,4 @@ +import numpy.typing as npt from autograd.scipy.stats import norm from scipy.stats import norm as scipy_norm @@ -8,9 +9,12 @@ weighted_mean_and_std, ) from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData class Normal_(OptimisedFitMixin, ParametricFitter): @@ -24,7 +28,7 @@ class Normal_(OptimisedFitMixin, ParametricFitter): """ - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=2, @@ -50,14 +54,20 @@ def __init__(self, name): ], ) - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + x, c, n = data.x, data.c, data.n if 2 in c: raise ValueError(c) - return para.Normal.fit( - x[c != -1], c[c != -1], n[c != -1], how="MPP" - ).params + return np.asarray( + para.Normal.fit( + x[c != -1], c[c != -1], n[c != -1], how="MPP" + ).params, + dtype=float, + ) - def _closed_form_mle(self, data): + def _closed_form_mle(self, data: SurpyvalData) -> npt.NDArray | None: r"""Exact MLE on complete data: the sample mean and (MLE) standard deviation, dividing by the total weight rather than ``total - 1``. @@ -74,7 +84,7 @@ def _closed_form_mle(self, data): return None return weighted_mean_and_std(x, data.n) - def sf(self, x, mu, sigma): + def sf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Survival (or Reliability) function for the Normal Distribution: @@ -108,43 +118,7 @@ def sf(self, x, mu, sigma): """ return norm.sf(x, mu, sigma) - def cs(self, x, X, mu, sigma): - r""" - - Conditional survival function for the Normal Distribution: - - .. math:: - R(x, X) = \frac{R(x + X)}{R(X)} - - Parameters - ---------- - - x : numpy array or scalar - The value(s) at which the function will be calculated - X : numpy array or scalar - The value(s) at which each value(s) in x was known to have survived - mu : numpy array or scalar - The location parameter for the Normal distribution - sigma : numpy array or scalar - The scale parameter for the Normal distribution - - Returns - ------- - - cs : scalar or numpy array - the conditional survival probability at x - - Examples - -------- - >>> import numpy as np - >>> from surpyval import Normal - >>> x = np.array([1, 2, 3, 4, 5]) - >>> Normal.cs(x, 5, 3, 4) - array([0.73452116, 0.51421702, 0.34242113, 0.2165286 , 0.1298356 ]) - """ - return self.sf(x + X, mu, sigma) / self.sf(X, mu, sigma) - - def ff(self, x, mu, sigma): + def ff(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" CDF (or unreliability or failure) function for the Normal Distribution: @@ -178,7 +152,7 @@ def ff(self, x, mu, sigma): """ return norm.cdf(x, mu, sigma) - def df(self, x, mu, sigma): + def df(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Density function for the Normal Distribution: @@ -213,7 +187,7 @@ def df(self, x, mu, sigma): """ return norm.pdf(x, mu, sigma) - def hf(self, x, mu, sigma): + def hf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Instantaneous hazard rate for the Normal Distribution: @@ -249,7 +223,7 @@ def hf(self, x, mu, sigma): """ return norm.pdf(x, mu, sigma) / self.sf(x, mu, sigma) - def Hf(self, x, mu, sigma): + def Hf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Cumulative hazard rate for the Normal Distribution: @@ -284,18 +258,18 @@ def Hf(self, x, mu, sigma): """ return -np.log(norm.sf(x, mu, sigma)) - def qf(self, p, mu, sigma): + def qf(self, u: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: r""" Quantile function for the Normal Distribution: .. math:: - q(p) = \Phi^{-1} \left( p \right ) + q(u) = \Phi^{-1} \left( u \right ) Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated mu : numpy array or scalar The location parameter for the Normal distribution @@ -306,19 +280,19 @@ def qf(self, p, mu, sigma): ------- q : scalar or numpy array - The quantiles for the Normal distribution at each value p. + The quantiles for the Normal distribution at each value u. Examples -------- >>> import numpy as np >>> from surpyval import Normal - >>> p = np.array([0.1, 0.2, 0.3, 0.4]) - >>> Normal.qf(p, 3, 4) + >>> u = np.array([0.1, 0.2, 0.3, 0.4]) + >>> Normal.qf(u, 3, 4) array([-2.12620626, -0.36648493, 0.90239795, 1.98661159]) """ - return scipy_norm.ppf(p, mu, sigma) + return scipy_norm.ppf(u, mu, sigma) - def mean(self, mu, sigma): + def mean(self, mu: Boxable, sigma: Boxable) -> Boxable: r""" Mean of the Normal distribution @@ -348,10 +322,10 @@ def mean(self, mu, sigma): """ return mu - def moment(self, n, mu, sigma): + def moment(self, m: int, mu: Boxable, sigma: Boxable) -> Boxable: r""" - n-th (non central) moment of the Normal distribution + m-th (non central) moment of the Normal distribution .. math:: E = ... complicated. @@ -359,7 +333,7 @@ def moment(self, n, mu, sigma): Parameters ---------- - n : integer or numpy array of integers + m : integer The ordinal of the moment to calculate mu : numpy array or scalar The location parameter for the Normal distribution @@ -378,9 +352,9 @@ def moment(self, n, mu, sigma): >>> Normal.moment(2, 3, 4) np.float64(25.0) """ - return scipy_norm.moment(n, mu, sigma) + return scipy_norm.moment(m, mu, sigma) - def entropy(self, mu, sigma): + def entropy(self, mu: Boxable, sigma: Boxable) -> Boxable: r""" Calculates the entropy of the Normal distribution. @@ -410,25 +384,27 @@ def entropy(self, mu, sigma): """ return 0.5 * np.log(2 * np.pi * np.e * sigma**2) - def log_df(self, x, mu, sigma): + def log_df(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: return norm.logpdf(x, mu, sigma) - def log_sf(self, x, mu, sigma): + def log_sf(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: return norm.logsf(x, mu, sigma) - def log_ff(self, x, mu, sigma): + def log_ff(self, x: Numeric, mu: Boxable, sigma: Boxable) -> Boxable: return norm.logcdf(x, mu, sigma) - def mpp_x_transform(self, x): + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: return x - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return self.qf(y, 0, 1) - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return self.ff(y, 0, 1) - def unpack_rr(self, params, rr): + def unpack_rr( + self, params: npt.NDArray, rr: str + ) -> tuple[Boxable, Boxable]: if rr == "y": sigma, mu = params mu = -mu / sigma diff --git a/surpyval/univariate/parametric/distributions/poisson.py b/surpyval/univariate/parametric/distributions/poisson.py index 349e70c..8abda79 100644 --- a/surpyval/univariate/parametric/distributions/poisson.py +++ b/surpyval/univariate/parametric/distributions/poisson.py @@ -11,6 +11,7 @@ Numeric, OptimisedFitMixin, ) +from surpyval.utils.surpyval_data import SurpyvalData class Poisson_(OptimisedFitMixin, DiscreteParametricFitter): @@ -50,14 +51,10 @@ def __init__(self, name: str) -> None: ) def _parameter_initialiser( - self, - x: npt.NDArray, - c: npt.NDArray | None = None, - n: npt.NDArray | None = None, - t: npt.NDArray | None = None, - offset: bool = False, + self, data: SurpyvalData, offset: bool = False ) -> npt.NDArray: # The Poisson mean is mu, so the sample mean is the moment seed. + x = data.x finite = x[np.isfinite(x)] mu = finite.mean() if finite.size else 1.0 return np.array([max(float(mu), 1e-3)]) @@ -66,11 +63,18 @@ def sf(self, x: Numeric, mu: Boxable) -> Boxable: r"""Survival function :math:`R(k) = P(T > k)`.""" # P(X > k) = P(X >= k + 1) is the regularised lower incomplete gamma # ``gammainc(k + 1, mu)``. - return gammainc(np.floor(x) + 1.0, mu) + # + # The first mass point is k = 0, so R = 1 below it. The gamma form + # only reaches that far by accident: its first argument is + # floor(k) + 1, which is 0 at k = -1 (where gammainc returns 1) but + # negative below that, where it returns NaN. + safe_a = np.where(x < 0.0, 1.0, np.floor(x) + 1.0) + return np.where(x < 0.0, 1.0, gammainc(safe_a, mu)) def ff(self, x: Numeric, mu: Boxable) -> Boxable: r"""CDF :math:`F(k) = P(T \le k)`.""" - return gammaincc(np.floor(x) + 1.0, mu) + safe_a = np.where(x < 0.0, 1.0, np.floor(x) + 1.0) + return np.where(x < 0.0, 0.0, gammaincc(safe_a, mu)) def df(self, x: Numeric, mu: Boxable) -> Boxable: r"""PMF :math:`P(T = k) = \mu^{k} e^{-\mu} / k!`.""" @@ -102,10 +106,13 @@ def random(self, size: int | tuple[int, ...], mu: Boxable) -> npt.NDArray: return poisson.rvs(mu, size=size).astype(float) def log_df(self, x: Numeric, mu: Boxable) -> Boxable: - return x * np.log(mu) - mu - gammaln(x + 1.0) + return np.where( + x < 0.0, -np.inf, x * np.log(mu) - mu - gammaln(x + 1.0) + ) def log_sf(self, x: Numeric, mu: Boxable) -> Boxable: - return np.log(gammainc(np.floor(x) + 1.0, mu)) + safe_a = np.where(x < 0.0, 1.0, np.floor(x) + 1.0) + return np.where(x < 0.0, 0.0, np.log(gammainc(safe_a, mu))) Poisson = Poisson_("Poisson") diff --git a/surpyval/univariate/parametric/distributions/rayleigh.py b/surpyval/univariate/parametric/distributions/rayleigh.py index ea03d2e..e5cd183 100644 --- a/surpyval/univariate/parametric/distributions/rayleigh.py +++ b/surpyval/univariate/parametric/distributions/rayleigh.py @@ -1,16 +1,22 @@ +from typing import Any + +import numpy.typing as npt from numpy import euler_gamma from scipy.special import gamma as gamma_func from surpyval import np from surpyval.univariate.nonparametric import plotting_positions from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData class Rayleigh_(OptimisedFitMixin, ParametricFitter): - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=1, @@ -44,15 +50,30 @@ def __init__(self, name): ], ) - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + x = data.x # sqrt(E[x^2] / 2) is the closed-form uncensored MLE for sigma if offset: gamma_init = np.min(x) - 1.0 sigma_init = np.sqrt(np.mean((x - gamma_init) ** 2) / 2) - return gamma_init, sigma_init - return np.sqrt(np.mean(x**2) / 2) + return np.array([gamma_init, sigma_init], dtype=float) + # A one-tuple, not the bare scalar this used to return. Rayleigh + # is the only single-parameter distribution here, and the scalar + # made `np.array(init)` in _initial_guess 0-dimensional rather + # than length-1. The lfp and zi paths then concatenate the p and + # f0 seeds onto it, and a 0-d array cannot be concatenated, so + # `Rayleigh.fit(x, lfp=True)` and `zi=True` both raised + # "zero-dimensional arrays cannot be concatenated". + return np.array( + [ + np.sqrt(np.mean(x**2) / 2), + ], + dtype=float, + ) - def sf(self, x, sigma): + def sf(self, x: Numeric, sigma: Boxable) -> Boxable: r""" Survival (or reliability) function for the Rayleigh Distribution: @@ -84,7 +105,7 @@ def sf(self, x, sigma): """ return np.exp(-(x**2) / (2 * sigma**2)) - def ff(self, x, sigma): + def ff(self, x: Numeric, sigma: Boxable) -> Boxable: r""" Failure (CDF or unreliability) function for the Rayleigh @@ -117,41 +138,7 @@ def ff(self, x, sigma): """ return -np.expm1(-(x**2) / (2 * sigma**2)) - def cs(self, x, X, sigma): - r""" - - Conditional survival function for the Rayleigh Distribution: - - .. math:: - R(x, X) = \frac{R(x + X)}{R(X)} - - Parameters - ---------- - - x : numpy array or scalar - The values at which the function will be calculated - X : numpy array or scalar - The values at which the item is known to have survived - sigma : numpy array or scalar - scale parameter for the Rayleigh distribution - - Returns - ------- - - cs : scalar or numpy array - The value(s) of the conditional survival function at x. - - Examples - -------- - >>> import numpy as np - >>> from surpyval import Rayleigh - >>> x = np.array([1, 2, 3, 4, 5]) - >>> Rayleigh.cs(x, 5, 3) - array([0.54274748, 0.26359714, 0.11455884, 0.04455143, 0.01550385]) - """ - return self.sf(x + X, sigma) / self.sf(X, sigma) - - def df(self, x, sigma): + def df(self, x: Numeric, sigma: Boxable) -> Boxable: r""" Density function for the Rayleigh Distribution: @@ -183,7 +170,7 @@ def df(self, x, sigma): """ return (x / (sigma**2)) * self.sf(x, sigma) - def hf(self, x, sigma): + def hf(self, x: Numeric, sigma: Boxable) -> Boxable: r""" Instantaneous hazard rate for the Rayleigh Distribution: @@ -215,7 +202,7 @@ def hf(self, x, sigma): """ return x / (sigma**2) - def Hf(self, x, sigma): + def Hf(self, x: Numeric, sigma: Boxable) -> Boxable: r""" Cumulative hazard rate for the Rayleigh Distribution: @@ -247,18 +234,18 @@ def Hf(self, x, sigma): """ return x**2 / (2 * sigma**2) - def qf(self, p, sigma): + def qf(self, u: Numeric, sigma: Boxable) -> Boxable: r""" Quantile function for the Rayleigh distribution: .. math:: - q(p) = \sigma \sqrt{-2 \ln \left ( 1 - p \right )} + q(u) = \sigma \sqrt{-2 \ln \left ( 1 - u \right )} Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated sigma : numpy array or scalar scale parameter for the Rayleigh distribution @@ -267,19 +254,19 @@ def qf(self, p, sigma): ------- q : scalar or numpy array - The quantiles for the Rayleigh distribution at each value p + The quantiles for the Rayleigh distribution at each value u Examples -------- >>> import numpy as np >>> from surpyval import Rayleigh - >>> p = np.array([.1, .2, .3, .4, .5]) - >>> Rayleigh.qf(p, 3) + >>> u = np.array([.1, .2, .3, .4, .5]) + >>> Rayleigh.qf(u, 3) array([1.37713082, 2.00414169, 2.53380129, 3.03230296, 3.53223007]) """ - return sigma * np.sqrt(2 * np.log(1 / (1 - p))) + return sigma * np.sqrt(2 * np.log(1 / (1 - u))) - def mean(self, sigma): + def mean(self, sigma: Boxable) -> Boxable: r""" Mean of the Rayleigh distribution @@ -307,18 +294,18 @@ def mean(self, sigma): """ return sigma * np.sqrt(np.pi / 2) - def moment(self, n, sigma): + def moment(self, m: int, sigma: Boxable) -> Boxable: r""" - n-th moment of the Rayleigh distribution + m-th moment of the Rayleigh distribution .. math:: - M(n) = \sigma^n 2^{n/2} \Gamma \left ( 1 + \frac{n}{2} \right ) + M(m) = \sigma^m 2^{m/2} \Gamma \left ( 1 + \frac{m}{2} \right ) Parameters ---------- - n : integer or numpy array of integers + m : integer The ordinal of the moment to calculate sigma : numpy array or scalar scale parameter for the Rayleigh distribution @@ -335,28 +322,28 @@ def moment(self, n, sigma): >>> Rayleigh.moment(2, 3) np.float64(18.0) """ - return (sigma**n) * (2 ** (n / 2)) * gamma_func(1 + n / 2) + return (sigma**m) * (2 ** (m / 2)) * gamma_func(1 + m / 2) - def entropy(self, sigma): + def entropy(self, sigma: Boxable) -> Boxable: return euler_gamma / 2 + 1 + np.log(sigma / (np.sqrt(2))) - def log_df(self, x, sigma): + def log_df(self, x: Numeric, sigma: Boxable) -> Boxable: return np.log(x) - 2 * np.log(sigma) - 0.5 * (x / sigma) ** 2 - def log_sf(self, x, sigma): + def log_sf(self, x: Numeric, sigma: Boxable) -> Boxable: return -0.5 * (x / sigma) ** 2 def mpp( self, - x, - c=None, - n=None, - t=None, - heuristic="Nelson-Aalen", - rr="y", - on_d_is_0=False, - offset=False, - ): + x: npt.NDArray, + c: npt.NDArray | None = None, + n: npt.NDArray | None = None, + t: npt.NDArray | None = None, + heuristic: str = "Nelson-Aalen", + rr: str = "y", + on_d_is_0: bool = False, + offset: bool = False, + ) -> dict[str, Any]: assert rr in ["x", "y"] # Forward the truncation windows: the custom Rayleigh path used to # drop them silently, making fits with and without tl/tr @@ -376,8 +363,8 @@ def mpp( F = F[valid] # Linearise - y_pp = self.mpp_y_transform(F) - x_pp = self.mpp_x_transform(x_pp) + y_pp = np.asarray(self.mpp_y_transform(F)) + x_pp = np.asarray(self.mpp_x_transform(x_pp)) if offset: if rr == "y": @@ -407,17 +394,17 @@ def mpp( return {"params": params} - def mpp_x_transform(self, x): + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: return x - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: mask = y == 0 out = np.zeros_like(y) out[~mask] = np.sqrt(-np.log(1 - y[~mask])) out[mask] = np.nan return out - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return 1 - np.exp(-(y**2)) diff --git a/surpyval/univariate/parametric/distributions/uniform.py b/surpyval/univariate/parametric/distributions/uniform.py index 38cffd8..d6725dd 100755 --- a/surpyval/univariate/parametric/distributions/uniform.py +++ b/surpyval/univariate/parametric/distributions/uniform.py @@ -1,12 +1,17 @@ +import numpy.typing as npt + from surpyval import np from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, OptimisedFitMixin, ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData class Uniform_(OptimisedFitMixin, ParametricFitter): - def __init__(self, name): + def __init__(self, name: str) -> None: super().__init__( name=name, k=2, @@ -22,10 +27,13 @@ def __init__(self, name): y_ticks=np.linspace(0, 1, 21)[1:-1], ) - def _parameter_initialiser(self, x, c=None, n=None, t=None, offset=False): - return np.min(x) - 1.0, np.max(x) + 1.0 + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + x = data.x + return np.array([np.min(x) - 1.0, np.max(x) + 1.0], dtype=float) - def sf(self, x, a, b): + def sf(self, x: Numeric, a: Boxable, b: Boxable) -> Boxable: r""" Survival (or Reliability) function for the Uniform Distribution: @@ -59,41 +67,7 @@ def sf(self, x, a, b): """ return 1 - self.ff(x, a, b) - def cs(self, x, X, a, b): - r""" - - Survival (or Reliability) function for the Uniform Distribution: - - .. math:: - R(x, X) = \frac{R(x + X)}{R(X)} - - Parameters - ---------- - - x : numpy array or scalar - The values at which the function will be calculated - a : numpy array or scalar - The lower parameter for the Uniform distribution - b : numpy array or scalar - The upper parameter for the Uniform distribution - - Returns - ------- - - cs : scalar or numpy array - The value(s) of the conditional survival function at x. - - Examples - -------- - >>> import numpy as np - >>> from surpyval import Uniform - >>> x = np.array([1, 2, 3, 4, 5]) - >>> Uniform.cs(x, 4, 0, 10) - array([0.83333333, 0.66666667, 0.5 , 0.33333333, 0.16666667]) - """ - return self.sf(x + X, a, b) / self.sf(X, a, b) - - def ff(self, x, a, b): + def ff(self, x: Numeric, a: Boxable, b: Boxable) -> Boxable: r""" Failure (CDF or unreliability) function for the Uniform Distribution: @@ -131,7 +105,7 @@ def ff(self, x, a, b): f = np.where(((x <= b) & (x >= a)), (x - a) / (b - a), f) return f - def df(self, x, a, b): + def df(self, x: Numeric, a: Boxable, b: Boxable) -> Boxable: r""" Failure (CDF or unreliability) function for the Uniform Distribution: @@ -169,7 +143,7 @@ def df(self, x, a, b): d = np.where(((x <= b) & (x >= a)), 1.0 / (b - a), d) return d - def hf(self, x, a, b): + def hf(self, x: Numeric, a: Boxable, b: Boxable) -> Boxable: r""" Instantaneous hazard rate for the Uniform Distribution: @@ -203,7 +177,7 @@ def hf(self, x, a, b): """ return self.df(x, a, b) / self.sf(x, a, b) - def log_df(self, x, a, b): + def log_df(self, x: Numeric, a: Boxable, b: Boxable) -> Boxable: r"""Log density, :math:`-\ln(b - a)` on the support. Defined directly rather than through the generic @@ -218,7 +192,7 @@ def log_df(self, x, a, b): inside = (x >= a) & (x <= b) return np.where(inside, -np.log(b - a), -np.inf) - def Hf(self, x, a, b): + def Hf(self, x: Numeric, a: Boxable, b: Boxable) -> Boxable: r""" Instantaneous hazard rate for the Uniform Distribution: @@ -252,18 +226,18 @@ def Hf(self, x, a, b): """ return -np.log(self.sf(x, a, b)) - def qf(self, p, a, b): + def qf(self, u: Numeric, a: Boxable, b: Boxable) -> Boxable: r""" Quantile function for the Uniform Distribution: .. math:: - q(p) = a + p(b - a) + q(u) = a + u(b - a) Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated a : numpy array or scalar The lower parameter for the Uniform distribution @@ -274,19 +248,19 @@ def qf(self, p, a, b): ------- q : scalar or numpy array - The quantiles for the Uniform distribution at each value p. + The quantiles for the Uniform distribution at each value u. Examples -------- >>> import numpy as np >>> from surpyval import Uniform - >>> p = np.array([.1, .2, .3, .4, .5]) - >>> Uniform.qf(p, 0, 6) + >>> u = np.array([.1, .2, .3, .4, .5]) + >>> Uniform.qf(u, 0, 6) array([0.6, 1.2, 1.8, 2.4, 3. ]) """ - return a + p * (b - a) + return a + u * (b - a) - def mean(self, a, b): + def mean(self, a: Boxable, b: Boxable) -> Boxable: r""" Mean of the Uniform distribution @@ -316,18 +290,18 @@ def mean(self, a, b): """ return 0.5 * (a + b) - def moment(self, n, a, b): + def moment(self, m: int, a: Boxable, b: Boxable) -> Boxable: r""" - n-th (non central) moment of the Uniform distribution + m-th (non central) moment of the Uniform distribution .. math:: - M(n) = \frac{1}{n +1} \sum_{i=0}^{n}a^ib^{n-i} + M(m) = \frac{1}{m +1} \sum_{i=0}^{m}a^ib^{m-i} Parameters ---------- - n : integer or numpy array of integers + m : integer The ordinal of the moment to calculate a : numpy array or scalar The lower parameter for the Uniform distribution @@ -346,15 +320,15 @@ def moment(self, n, a, b): >>> Uniform.moment(2, 0, 6) np.float64(12.0) """ - if n == 0: + if m == 0: return 1 else: - out = np.zeros(n + 1) - for i in range(n + 1): - out[i] = a**i * b ** (n - i) - return np.sum(out) / (n + 1) + out = np.zeros(m + 1) + for i in range(m + 1): + out[i] = a**i * b ** (m - i) + return np.sum(out) / (m + 1) - def entropy(self, a, b): + def entropy(self, a: Boxable, b: Boxable) -> Boxable: r""" Calculates the entropy of the Uniform distribution. @@ -384,7 +358,7 @@ def entropy(self, a, b): """ return np.log(b - a) - def _closed_form_mle(self, data): + def _closed_form_mle(self, data: SurpyvalData) -> npt.NDArray | None: if np.asarray(data.x).ndim == 2 or (data.c == 2).any(): # The closed-form min/max estimator is not the MLE with # interval-censored rows (an interval term favours shrinking @@ -424,16 +398,18 @@ def _closed_form_mle(self, data): return np.array([np.min(data.x), np.max(data.x)]) - def mpp_x_transform(self, x): + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: return x - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return y - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return y - def unpack_rr(self, params, rr): + def unpack_rr( + self, params: npt.NDArray, rr: str + ) -> tuple[Boxable, Boxable]: if rr == "y": a = -params[1] / params[0] b = (1 - params[1]) / params[0] @@ -443,7 +419,7 @@ def unpack_rr(self, params, rr): return a, b - def _mom(self, x): + def _mom(self, x: npt.NDArray) -> tuple[float, float]: mu_1 = np.mean(x) mu_2 = np.mean(x**2) @@ -452,7 +428,9 @@ def _mom(self, x): b = mu_1 + d return a, b - def _plot_x_bounds(self, x, params): + def _plot_x_bounds( + self, x: npt.NDArray, params: npt.NDArray + ) -> tuple[float, float] | None: return float(np.min(params)), float(np.max(params)) diff --git a/surpyval/univariate/parametric/distributions/weibull.py b/surpyval/univariate/parametric/distributions/weibull.py index c568c6e..9b40e52 100755 --- a/surpyval/univariate/parametric/distributions/weibull.py +++ b/surpyval/univariate/parametric/distributions/weibull.py @@ -9,6 +9,7 @@ OptimisedFitMixin, ParametricFitter, ) +from surpyval.utils.surpyval_data import SurpyvalData class Weibull_(OptimisedFitMixin, ParametricFitter): @@ -24,20 +25,15 @@ def __init__(self, name: str) -> None: ) def _parameter_initialiser( - self, - x: Numeric, - c: npt.NDArray | None = None, - n: npt.NDArray | None = None, - t: npt.NDArray | None = None, - offset: bool = False, - ) -> tuple[float, ...]: - mpp_model = self.fit( - x, c, n, offset=offset, how="MPP", heuristic="Nelson-Aalen" + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: + mpp_model = self.fit_from_surpyval_data( + data, offset=offset, how="MPP", heuristic="Nelson-Aalen" ) if offset: - return (mpp_model.gamma, *mpp_model.params) + return np.array([mpp_model.gamma, *mpp_model.params], dtype=float) else: - return tuple(mpp_model.params) + return np.asarray(mpp_model.params, dtype=float) def sf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" @@ -110,45 +106,6 @@ def ff(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: # same as np.exp for large values return -np.expm1(-((x / alpha) ** beta)) - def cs( - self, x: Numeric, X: Numeric, alpha: Boxable, beta: Boxable - ) -> Boxable: - r""" - - Conditional survival function for the Weibull Distribution: - - .. math:: - R(x, X) = \frac{R(x + X)}{R(X)} - - Parameters - ---------- - - x : numpy array or scalar - The values at which the function will be calculated - X : numpy array or scalar - The values at which the item is known to have survived - alpha : numpy array or scalar - scale parameter for the Weibull distribution - beta : numpy array or scalar - shape parameter for the Weibull distribution - - Returns - ------- - - cs : scalar or numpy array - The value(s) of the conditional survival function at x. - - Examples - -------- - >>> import numpy as np - >>> from surpyval import Weibull - >>> x = np.array([1, 2, 3, 4, 5]) - >>> Weibull.cs(x, 5, 3, 4) - array([2.52537548e-04, 3.00394073e-10, 2.45288508e-19, 1.48999440e-32, - 5.42544000e-51]) - """ - return self.sf(x + X, alpha, beta) / self.sf(X, alpha, beta) - def df(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" @@ -257,19 +214,19 @@ def Hf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: """ return (x / alpha) ** beta - def qf(self, p: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: + def qf(self, u: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: r""" Quantile function for the Weibull distribution: .. math:: - q(p) = \alpha \left ( -\ln \left ( 1 - p \right ) \right )^{1/ + q(u) = \alpha \left ( -\ln \left ( 1 - u \right ) \right )^{1/ \beta} Parameters ---------- - p : numpy array or scalar + u : numpy array or scalar The percentiles at which the quantile will be calculated alpha : numpy array or scalar scale parameter for the Weibull distribution @@ -280,17 +237,17 @@ def qf(self, p: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: ------- q : scalar or numpy array - The quantiles for the Weibull distribution at each value p + The quantiles for the Weibull distribution at each value u Examples -------- >>> import numpy as np >>> from surpyval import Weibull - >>> p = np.array([.1, .2, .3, .4, .5]) - >>> Weibull.qf(p, 3, 4) + >>> u = np.array([.1, .2, .3, .4, .5]) + >>> Weibull.qf(u, 3, 4) array([1.70919151, 2.06189877, 2.31840554, 2.5362346 , 2.73733292]) """ - return alpha * (-np.log1p(-p)) ** (1 / beta) + return alpha * (-np.log1p(-u)) ** (1 / beta) def mean(self, alpha: Boxable, beta: Boxable) -> Boxable: r""" @@ -322,18 +279,18 @@ def mean(self, alpha: Boxable, beta: Boxable) -> Boxable: """ return alpha * gamma_func(1 + 1.0 / beta) - def moment(self, n: int, alpha: Boxable, beta: Boxable) -> Boxable: + def moment(self, m: int, alpha: Boxable, beta: Boxable) -> Boxable: r""" - n-th moment of the Weibull distribution + m-th moment of the Weibull distribution .. math:: - M(n) = \alpha^n \Gamma \left ( 1 + \frac{n}{\beta} \right ) + M(m) = \alpha^m \Gamma \left ( 1 + \frac{m}{\beta} \right ) Parameters ---------- - n : integer or numpy array of integers + m : integer The ordinal of the moment to calculate alpha : numpy array or scalar scale parameter for the Weibull distribution @@ -352,7 +309,7 @@ def moment(self, n: int, alpha: Boxable, beta: Boxable) -> Boxable: >>> Weibull.moment(2, 3, 4) np.float64(7.976042329074821) """ - return alpha**n * gamma_func(1 + n / beta) + return alpha**m * gamma_func(1 + m / beta) def entropy(self, alpha: Boxable, beta: Boxable) -> Boxable: return euler_gamma * (1 - 1 / beta) + np.log(alpha) - np.log(beta) + 1 @@ -369,17 +326,17 @@ def log_df(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: def log_sf(self, x: Numeric, alpha: Boxable, beta: Boxable) -> Boxable: return -((x / alpha) ** beta) - def mpp_x_transform(self, x: Numeric) -> npt.NDArray: + def mpp_x_transform(self, x: npt.NDArray) -> Boxable: return np.log(x) - def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> npt.NDArray: + def mpp_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: mask = (y == 0) | (y == 1) out = np.zeros_like(y) out[~mask] = np.log(-np.log(1 - y[~mask])) out[mask] = np.nan return out - def mpp_inv_y_transform(self, y: Numeric, *params: Boxable) -> Boxable: + def mpp_inv_y_transform(self, y: npt.NDArray, *params: Boxable) -> Boxable: return 1 - np.exp(-np.exp(y)) def unpack_rr( diff --git a/surpyval/univariate/parametric/fitters/__init__.py b/surpyval/univariate/parametric/fitters/__init__.py index f759dd2..d505f7e 100755 --- a/surpyval/univariate/parametric/fitters/__init__.py +++ b/surpyval/univariate/parametric/fitters/__init__.py @@ -1,9 +1,18 @@ +from typing import Any, Callable, Sequence +import numpy.typing as npt from scipy.optimize import minimize from surpyval import np -def fallback_minimize(fun, init, args, jac, hess, newton_tol=None): +def fallback_minimize( + fun: Callable[..., Any], + init: npt.NDArray, + args: tuple[Any, ...], + jac: Callable[..., Any] | None, + hess: Callable[..., Any] | None, + newton_tol: float | None = None, +) -> Any: """ Minimise ``fun`` with BFGS and the supplied jacobian, escalating to Newton-CG with the hessian and then to Nelder-Mead whenever a method @@ -29,6 +38,7 @@ def fallback_minimize(fun, init, args, jac, hess, newton_tol=None): while reporting success, so there is nothing to escalate to and Nelder-Mead should take over instead. """ + assert jac is not None and hess is not None with np.errstate(all="ignore"): res = minimize(fun, init, method="BFGS", jac=jac, args=args) @@ -56,7 +66,13 @@ def fallback_minimize(fun, init, args, jac, hess, newton_tol=None): return res -def preconditioned_bfgs(fun, x0, args=(), jac=None, options=None): +def preconditioned_bfgs( + fun: Callable[..., Any], + x0: npt.NDArray, + args: tuple[Any, ...] = (), + jac: Callable[..., Any] | None = None, + options: dict[str, Any] | None = None, +) -> Any: """BFGS on a diagonally rescaled copy of the search vector. scipy stops BFGS when ``max|grad| < gtol``, an absolute threshold on @@ -104,10 +120,12 @@ def preconditioned_bfgs(fun, x0, args=(), jac=None, options=None): f0 = float(fun(x0, *args)) obj_scale = max(abs(f0), 1.0) if np.isfinite(f0) else 1.0 - def scaled_fun(v, *inner): + def scaled_fun(v: npt.NDArray, *inner: Any) -> Any: return fun(scale * v, *inner) / obj_scale - def scaled_jac(v, *inner): + assert jac is not None + + def scaled_jac(v: npt.NDArray, *inner: Any) -> Any: return ( scale * np.asarray(jac(scale * v, *inner), dtype=float) ) / obj_scale @@ -128,7 +146,7 @@ def scaled_jac(v, *inner): return res -def _dead_branch_safe_exp(x): +def _dead_branch_safe_exp(x: npt.NDArray) -> Any: """``exp(x)`` for the ``x < 0`` half, with the other half clamped. ``np.where`` picks the right value but autograd evaluates *both* @@ -144,25 +162,31 @@ def _dead_branch_safe_exp(x): return np.exp(np.minimum(x, 0.0)) -def adj_relu(x): +def adj_relu(x: npt.NDArray) -> Any: return np.where(x >= 0, x + 1, _dead_branch_safe_exp(x)) -def inv_adj_relu(x): +def inv_adj_relu(x: npt.NDArray) -> Any: # No clamp needed here, unlike ``adj_relu``: this dead branch is # ``x >= 1``, which is precisely where ``log`` is best behaved. return np.where(x >= 1, x - 1, np.log(x)) -def rev_adj_relu(x): +def rev_adj_relu(x: npt.NDArray) -> Any: return -np.where(x >= 0, x + 1, _dead_branch_safe_exp(x)) -def inv_rev_adj_relu(x): +def inv_rev_adj_relu(x: npt.NDArray) -> Any: return np.where(x < -1, -x - 1, np.log(-x)) -def add_to_funcs(low, upp, i, funcs, inv_f): +def add_to_funcs( + low: float | None, + upp: float | None, + i: int, + funcs: list[Callable[..., Any]], + inv_f: list[Callable[..., Any]], +) -> None: if (low is None) and (upp is None): funcs.append(lambda x: x) inv_f.append(lambda x: x) @@ -181,15 +205,20 @@ def add_to_funcs(low, upp, i, funcs, inv_f): inv_f.append(lambda x: x) -def bounds_convert(x, bounds, fixed, param_map): +def bounds_convert( + x: npt.ArrayLike, + bounds: Sequence[tuple[float | None, float | None]], + fixed: dict[str, float] | None, + param_map: dict[str, int], +) -> tuple[Any, ...]: """ This function is used to transform the parameters from the bounded parameter space to the unbounded parameter space. This is an improvement over using the scipy.optimize.minimize function's bounds parameter as it allows us to avoid the use of the constrained optimization methods. """ - bounded_to_unbounded_transforms = [] - unbounded_to_bounded_transforms = [] + bounded_to_unbounded_transforms: list[Callable[..., Any]] = [] + unbounded_to_bounded_transforms: list[Callable[..., Any]] = [] for i, (lower, upper) in enumerate(bounds): add_to_funcs( @@ -200,7 +229,7 @@ def bounds_convert(x, bounds, fixed, param_map): unbounded_to_bounded_transforms, ) - def transform_params_to_unbounded(params): + def transform_params_to_unbounded(params: npt.NDArray) -> Any: return np.array( [ f(p) @@ -210,7 +239,7 @@ def transform_params_to_unbounded(params): ] ) - def transform_unbounded_value_to_params(params): + def transform_unbounded_value_to_params(params: npt.NDArray) -> Any: return np.array( [ f(p) @@ -227,7 +256,7 @@ def transform_unbounded_value_to_params(params): not_fixed = [x for x in range(n_params) if x not in fixed_idx] not_fixed = np.array(not_fixed, dtype=int) - def constraints(p): + def constraints(p: npt.NDArray) -> Any: params = [0] * (n_params) for k, v in fixed.items(): params[param_map[k]] = bounded_to_unbounded_transforms[ @@ -237,10 +266,10 @@ def constraints(p): params[i] = v return np.array(params) - const = constraints + const: Callable[..., Any] = constraints else: - def const(x): + def const(x: npt.NDArray) -> Any: return x fixed_idx = [] diff --git a/surpyval/univariate/parametric/fitters/closed_form.py b/surpyval/univariate/parametric/fitters/closed_form.py index aaa7741..f0d9140 100644 --- a/surpyval/univariate/parametric/fitters/closed_form.py +++ b/surpyval/univariate/parametric/fitters/closed_form.py @@ -1,3 +1,9 @@ +from typing import Any + +import numpy.typing as npt + +from surpyval.utils.surpyval_data import SurpyvalData + """Closed-form maximum likelihood estimation. A few distributions have an exact analytic MLE for particular data @@ -28,14 +34,16 @@ from surpyval import np -def _neg_ll_at(dist, data, params): +def _neg_ll_at(dist: Any, data: SurpyvalData, params: npt.NDArray) -> float: """The model's negative log-likelihood at ``params`` (no offset, zero-inflation or limited-failure component -- the closed-form gate excludes all three).""" return dist._neg_ll_func(data, *params, 0.0, 0.0, 1.0) -def parameter_covariance(dist, data, params): +def parameter_covariance( + dist: Any, data: SurpyvalData, params: npt.NDArray +) -> npt.NDArray | None: """Asymptotic parameter covariance, the inverse observed information. Computed directly in the natural parameter space: the closed-form @@ -46,7 +54,7 @@ def parameter_covariance(dist, data, params): """ params = onp.asarray(params, dtype=float) - def fun(p): + def fun(p: npt.NDArray) -> Any: return _neg_ll_at(dist, data, p) with onp.errstate(all="ignore"): @@ -75,7 +83,9 @@ def fun(p): return cov -def closed_form_results(dist, data, params): +def closed_form_results( + dist: Any, data: SurpyvalData, params: npt.NDArray +) -> Any: """Complete a closed-form parameter vector into a full results dict. Mirrors what the optimiser path returns, so a closed-form fit @@ -116,7 +126,7 @@ def closed_form_results(dist, data, params): } -def entry_times(data): +def entry_times(data: SurpyvalData) -> npt.NDArray: """Left-truncation times with non-finite entries replaced by 0. ``-inf`` marks "not truncated"; for a lifetime distribution supported @@ -127,7 +137,7 @@ def entry_times(data): return onp.where(onp.isfinite(tl), tl, 0.0) -def is_uncensored_and_untruncated(data): +def is_uncensored_and_untruncated(data: SurpyvalData) -> bool: """Every observation exact, with no truncation of either kind.""" return bool( (onp.asarray(data.c) == 0).all() @@ -136,7 +146,9 @@ def is_uncensored_and_untruncated(data): ) -def weighted_mean_and_std(values, n): +def weighted_mean_and_std( + values: npt.NDArray, n: npt.NDArray +) -> npt.NDArray | None: """MLE mean and standard deviation (dividing by the total weight, not by ``total - 1``), or ``None`` if the spread is degenerate.""" values = onp.asarray(values, dtype=float) diff --git a/surpyval/univariate/parametric/fitters/mle.py b/surpyval/univariate/parametric/fitters/mle.py index 64918f6..4fb2172 100755 --- a/surpyval/univariate/parametric/fitters/mle.py +++ b/surpyval/univariate/parametric/fitters/mle.py @@ -1,5 +1,10 @@ import warnings +from typing import TYPE_CHECKING, Any +if TYPE_CHECKING: + from ..parametric import Parametric + +import numpy.typing as npt from autograd import hessian, jacobian from autograd.numpy.linalg import inv from numdifftools import Hessian # type: ignore @@ -9,7 +14,7 @@ from surpyval.univariate.parametric.fitters import preconditioned_bfgs -def mle(model): +def mle(model: "Parametric") -> Any: """ Maximum Likelihood Estimation (MLE) @@ -33,15 +38,15 @@ def mle(model): """ def fun( - params, - offset=False, - lfp=False, - zi=False, - transform=True, - gamma=0, - f0=0, - p=1, - ): + params: Any, + offset: bool = False, + lfp: bool = False, + zi: bool = False, + transform: bool = True, + gamma: float = 0, + f0: float = 0, + p: float = 1, + ) -> Any: # Transform parameters from (-Inf, Inf) range to parameter # to correct bounded values if transform: @@ -224,7 +229,7 @@ def fun( embed[var_idx, np.arange(len(var_idx))] = 1.0 u_held = np.where(embed.sum(axis=1) == 0, u_full, 0.0) - def transformed_fun(u): + def transformed_fun(u: npt.NDArray) -> Any: theta = inv_trans(embed @ u + u_held)[n_head:] if zi: *theta, f0_i = theta @@ -238,7 +243,7 @@ def transformed_fun(u): model.surv_data, *theta, gamma, f0_i, p_i ) - def u_to_phi(u): + def u_to_phi(u: npt.NDArray) -> Any: return inv_trans(embed @ u + u_held)[n_head:] try: diff --git a/surpyval/univariate/parametric/fitters/mom.py b/surpyval/univariate/parametric/fitters/mom.py index a4eee05..8b28908 100755 --- a/surpyval/univariate/parametric/fitters/mom.py +++ b/surpyval/univariate/parametric/fitters/mom.py @@ -1,12 +1,17 @@ import warnings from math import comb +from typing import TYPE_CHECKING, Any, Callable +if TYPE_CHECKING: + from ..parametric import Parametric + +import numpy.typing as npt from scipy.optimize import minimize from surpyval import np -def raw_to_central(moments): +def raw_to_central(moments: npt.NDArray) -> npt.NDArray: """``(mean, var, mu3, ...)`` from raw moments ``E[X], E[X^2], ...``. ``mu_k = sum_j C(k, j) (-1)^(k-j) E[X^j] mean^(k-j)``, with the mean @@ -29,7 +34,14 @@ def raw_to_central(moments): return np.array(central) -def mom_fun(params, dist, inv_trans, const, offset, moments): +def mom_fun( + params: npt.NDArray, + dist: Any, + inv_trans: Callable[..., Any], + const: Callable[..., Any], + offset: bool, + moments: npt.NDArray, +) -> Any: """Squared mismatch between the sample and model moments. Compared as *central* moments scaled by the sample's own standard @@ -68,7 +80,7 @@ def mom_fun(params, dist, inv_trans, const, offset, moments): return (((sample - model) / scale) ** 2).sum() -def mom(model): +def mom(model: "Parametric") -> Any: """ MOM: Method of Moments. diff --git a/surpyval/univariate/parametric/fitters/mpp.py b/surpyval/univariate/parametric/fitters/mpp.py index 31db90b..a82b3ff 100755 --- a/surpyval/univariate/parametric/fitters/mpp.py +++ b/surpyval/univariate/parametric/fitters/mpp.py @@ -1,5 +1,10 @@ from copy import copy +from typing import TYPE_CHECKING, Any +if TYPE_CHECKING: + from ..parametric import Parametric + +import numpy.typing as npt from scipy.optimize import minimize from scipy.stats import pearsonr @@ -7,7 +12,7 @@ from surpyval.univariate.nonparametric import plotting_positions -def _rr_fit(a, b): +def _rr_fit(a: npt.NDArray, b: npt.NDArray) -> Any: """ Least-squares line of ``b`` on ``a``, guarding the degenerate case. @@ -43,9 +48,11 @@ def _rr_fit(a, b): return np.array([1.0, intercept]) -def mpp_from_ecfd(dist, x, F): - x_pp = copy(x) - y_pp = copy(F) +def mpp_from_ecfd( + dist: Any, x: npt.ArrayLike, F: npt.ArrayLike +) -> dict[str, Any]: + x_pp = np.asarray(copy(x)) + y_pp = np.asarray(copy(F)) mask = (y_pp != 0) & (y_pp != 1) y_pp = y_pp[mask] @@ -63,7 +70,7 @@ def mpp_from_ecfd(dist, x, F): return results -def mpp(model): +def mpp(model: "Parametric") -> dict[str, Any]: """ MPP: Method of Probability Plotting @@ -132,7 +139,7 @@ def mpp(model): if offset: x_min = np.min(x_pp) - def fun(gamma): + def fun(gamma: float) -> Any: g = x_min - np.exp(-gamma) out = -pearsonr(dist.mpp_x_transform(x_pp - g), y_pp)[0] return out diff --git a/surpyval/univariate/parametric/fitters/mps.py b/surpyval/univariate/parametric/fitters/mps.py index 84392af..3e8f125 100755 --- a/surpyval/univariate/parametric/fitters/mps.py +++ b/surpyval/univariate/parametric/fitters/mps.py @@ -1,5 +1,10 @@ import warnings +from typing import TYPE_CHECKING, Any, Callable +if TYPE_CHECKING: + from ..parametric import Parametric + +import numpy.typing as npt from autograd import hessian, jacobian from surpyval import np @@ -7,7 +12,18 @@ from . import fallback_minimize -def mps_fun(params, dist, x, inv_trans, const, c, n, tl, tr, offset): +def mps_fun( + params: npt.NDArray, + dist: Any, + x: npt.NDArray, + inv_trans: Callable[..., Any], + const: Callable[..., Any], + c: npt.NDArray | None, + n: npt.NDArray, + tl: npt.NDArray | None, + tr: npt.NDArray | None, + offset: bool, +) -> Any: if offset: gamma = inv_trans(const(params))[0] x_new = x - gamma @@ -37,7 +53,7 @@ def mps_fun(params, dist, x, inv_trans, const, c, n, tl, tr, offset): return D -def mps(model): +def mps(model: "Parametric") -> Any: """ MPS: Maximum Product Spacing diff --git a/surpyval/univariate/parametric/fitters/mse.py b/surpyval/univariate/parametric/fitters/mse.py index fdc0ef6..40ab1f4 100755 --- a/surpyval/univariate/parametric/fitters/mse.py +++ b/surpyval/univariate/parametric/fitters/mse.py @@ -1,3 +1,9 @@ +from typing import TYPE_CHECKING, Any, Callable + +if TYPE_CHECKING: + from ..parametric import Parametric + +import numpy.typing as npt from autograd import hessian, jacobian from surpyval import np @@ -7,7 +13,15 @@ from . import fallback_minimize -def mse_fun(params, dist, x, F, inv_trans, const, offset): +def mse_fun( + params: npt.NDArray, + dist: Any, + x: npt.NDArray, + F: npt.NDArray, + inv_trans: Callable[..., Any], + const: Callable[..., Any], + offset: bool, +) -> Any: params = inv_trans(const(params)) if offset: x = x - params[0] @@ -15,7 +29,7 @@ def mse_fun(params, dist, x, F, inv_trans, const, offset): return np.sum((dist.ff(x, *params) - F) ** 2) -def mse(model): +def mse(model: "Parametric") -> Any: """ MSE: Mean Square Error This is simply fitting the curve to the best estimate from a non-parametric diff --git a/surpyval/univariate/parametric/mixture_model.py b/surpyval/univariate/parametric/mixture_model.py index f43918c..2029c02 100755 --- a/surpyval/univariate/parametric/mixture_model.py +++ b/surpyval/univariate/parametric/mixture_model.py @@ -1,5 +1,7 @@ import warnings +from typing import Any +import numpy.typing as npt from matplotlib import pyplot as plt from scipy.optimize import minimize @@ -37,14 +39,21 @@ class MixtureModel(SerialisableMixin, Distribution): Defaults to 2. """ - def __init__(self, dist, m=2): + def __init__(self, dist: Any, m: int = 2) -> None: self.m = m self.dist = dist - self.data = None - self.params = None - self.w = None - self.p = None - self.loglike = None + # These are None until ``fit`` runs and arrays afterwards, so the + # honest annotation is the union -- and every use is downstream of + # a fit. Narrowing them to the fitted type would be a lie before + # the fit; narrowing to Optional would need an assert at each of + # the thirty-odd uses without making anything safer, because + # calling a predict method on an unfitted model is a contract + # error the AttributeError already reports. + self.data: Any = None + self.params: Any = None + self.w: Any = None + self.p: Any = None + self.loglike: Any = None # -- serialisation ----------------------------------------------------- @@ -89,7 +98,7 @@ def from_dict(cls, model_dict: dict) -> "MixtureModel": out.w = np.array(model_dict["w"], dtype=float) return out - def __repr__(self): + def __repr__(self) -> str: if self.params is not None: param_string = "\n".join( [ @@ -112,7 +121,7 @@ def __repr__(self): else: return "Unable to fit values" - def likelihood(self, params): + def likelihood(self, params: Any) -> Any: """Per-observation likelihood of one component (no count powers: counts ``n`` enter the log-likelihood as multipliers -- raising the per-component likelihood to ``n`` *before* mixing is wrong, since @@ -131,7 +140,7 @@ def likelihood(self, params): like[data.c == 2] = like_i return like - def _window_prob(self, params_i): + def _window_prob(self, params_i: npt.NDArray) -> Any: """One component's probability of landing in each observation's truncation window ``(tl, tr]`` -- the per-component piece of the truncation correction.""" @@ -146,7 +155,7 @@ def _window_prob(self, params_i): hi[fin] = self.dist.ff(tr[fin], *params_i) return hi - lo - def neg_ll_of(self, w, params): + def neg_ll_of(self, w: npt.NDArray, params: Any) -> Any: """Observed negative log-likelihood of the mixture: counts multiply in the log domain, and truncated observations are conditioned on their window through the mixture probability of the window.""" @@ -162,7 +171,7 @@ def neg_ll_of(self, w, params): ll -= np.sum(self.data.n * np.log(win)) return -ll - def Q(self, params): + def Q(self, params: Any) -> Any: """EM M-step objective: the (negative) expected complete-data log-likelihood over the component labels -- counts times responsibilities times each component's log-likelihood.""" @@ -180,7 +189,7 @@ def Q(self, params): total -= contrib.sum() return total - def expectation(self): + def expectation(self) -> Any: for i in range(self.m): like = self.likelihood(self.params[i]) like = np.multiply(self.w[i], like) @@ -189,19 +198,19 @@ def expectation(self): # Mixing weights are count-weighted responsibility totals. self.w = (self.p * self.data.n).sum(axis=1) / self.data.n.sum() - def maximisation(self): + def maximisation(self) -> Any: bounds = self.dist.bounds * self.m res = minimize(self.Q, self.params.ravel(), bounds=bounds) self.params = res.x.reshape(self.m, self.dist.k) - def EM(self): + def EM(self) -> Any: self.expectation() self.maximisation() # Convergence is tracked on the observed likelihood, not the # M-step objective. self.loglike = self.neg_ll_of(self.w, self.params) - def _em(self, tol=1e-10, max_iter=1000): + def _em(self, tol: float = 1e-10, max_iter: int = 1000) -> Any: i = 0 self.EM() f0 = self.loglike @@ -217,7 +226,7 @@ def _em(self, tol=1e-10, max_iter=1000): "EM algorithm reached max iterations before converging" ) - def initialise_params(self): + def initialise_params(self) -> Any: splits_x = np.array_split(self.data.x, self.m) splits_c = np.array_split(self.data.c, self.m) splits_n = np.array_split(self.data.n, self.m) @@ -232,15 +241,15 @@ def initialise_params(self): def fit( self, - x=None, - c=None, - n=None, - t=None, - tl=None, - tr=None, - xl=None, - xr=None, - ): + x: npt.ArrayLike | None = None, + c: npt.ArrayLike | None = None, + n: npt.ArrayLike | None = None, + t: npt.ArrayLike | None = None, + tl: npt.ArrayLike | None = None, + tr: npt.ArrayLike | None = None, + xl: npt.ArrayLike | None = None, + xr: npt.ArrayLike | None = None, + ) -> Any: """ Parameters ---------- @@ -326,13 +335,13 @@ def fit( else: self._em() - def _direct_mle(self): + def _direct_mle(self) -> Any: """Directly maximise the observed (truncation-corrected) negative log-likelihood over the mixing weights (via softmax logits) and the component parameters.""" k = self.dist.k - def unpack(theta): + def unpack(theta: npt.NDArray) -> Any: logits = np.append(theta[: self.m - 1], 0.0) logits = logits - logits.max() w = np.exp(logits) @@ -340,11 +349,13 @@ def unpack(theta): params = theta[self.m - 1 :].reshape(self.m, k) return w, params - def obj(theta): + def obj(theta: npt.NDArray) -> Any: w, params = unpack(theta) return self.neg_ll_of(w, params) - bounds = [(None, None)] * (self.m - 1) + bounds: list[tuple[float | None, float | None]] = [(None, None)] * ( + self.m - 1 + ) for _ in range(self.m): for lo, hi in self.dist.bounds: lo_s = None if lo is None else (1e-10 if lo == 0 else lo) @@ -356,13 +367,13 @@ def obj(theta): self.w, self.params = unpack(res.x) self.loglike = float(res.fun) - def mean(self): + def mean(self, *args: Any, **kwargs: Any) -> Any: mean = 0 for i in range(self.m): mean += self.w[i] * self.dist.mean(*self.params[i]) return mean - def random(self, size): + def random(self, size: int, *args: Any, **kwargs: Any) -> Any: sizes = np.random.multinomial(size, self.w) rvs = np.zeros(size) s_last = 0 @@ -373,7 +384,7 @@ def random(self, size): np.random.shuffle(rvs) return rvs - def df(self, x): + def df(self, x: Any, *args: Any, **kwargs: Any) -> Any: """ The probability density function of the fitted model. @@ -396,7 +407,7 @@ def df(self, x): df += self.w[i] * self.dist.df(x, *self.params[i]) return df - def ff(self, x): + def ff(self, x: Any, *args: Any, **kwargs: Any) -> Any: """ The cumulative density function of the fitted model. @@ -418,7 +429,7 @@ def ff(self, x): F = F + self.w[i] * self.dist.ff(x, *self.params[i]) return F - def sf(self, x): + def sf(self, x: Any, *args: Any, **kwargs: Any) -> Any: """ The survival function of the fitted model. @@ -436,7 +447,7 @@ def sf(self, x): """ return 1 - self.ff(x) - def cs(self, x, X): + def cs(self, x: Any, X: Any, *args: Any, **kwargs: Any) -> Any: """ The conditional survival function of the fitted model. @@ -458,7 +469,7 @@ def cs(self, x, X): """ return self.sf(x + X) / self.sf(X) - def get_plot_data(self, heuristic="Nelson-Aalen"): + def get_plot_data(self, heuristic: str = "Nelson-Aalen") -> Any: return probability_plot_data( dist=self.dist, ff=self.ff, @@ -470,11 +481,7 @@ def get_plot_data(self, heuristic="Nelson-Aalen"): params=self.params, ) - def plot( - self, - heuristic="Nelson-Aalen", - ax=None, - ): + def plot(self, heuristic: str = "Nelson-Aalen", ax: Any = None) -> Any: """ A method to do a probability plot diff --git a/surpyval/univariate/parametric/parametric.py b/surpyval/univariate/parametric/parametric.py index a0b1dcb..356bcc9 100755 --- a/surpyval/univariate/parametric/parametric.py +++ b/surpyval/univariate/parametric/parametric.py @@ -1,26 +1,22 @@ from collections import namedtuple from copy import copy, deepcopy from math import comb -from typing import TYPE_CHECKING, Any, Callable +from typing import Any, Callable import matplotlib.pyplot as plt import numpy.typing as npt from autograd import jacobian +from matplotlib.axes import Axes from scipy.optimize import NonlinearConstraint, brentq, minimize from scipy.special import ndtri as z from scipy.stats import uniform import surpyval as surv from surpyval import ParametricDistribution, np -from surpyval.utils import fsli_to_xcnt - -if TYPE_CHECKING: - from matplotlib.axes import Axes - - from surpyval.utils.surpyval_data import SurpyvalData - from surpyval.serialisation import SerialisableMixin, stamp_schema, to_native from surpyval.univariate.information_criteria import InformationCriteriaMixin +from surpyval.utils import fsli_to_xcnt +from surpyval.utils.surpyval_data import SurpyvalData from .probability_plotting import ( adjust_heuristic, @@ -360,7 +356,7 @@ def _user_fixed_idx(self) -> set: info = getattr(self, "fitting_info", None) or {} return set(info.get("fixed_idx", []) or []) - def _profile_neg_ll(self, idx: int, value: float) -> float: + def _profile_neg_ll(self, idx: int, value: Any) -> float: """Profile negative log-likelihood with core parameter ``idx`` fixed. Holds the ``idx``-th distribution parameter at ``value`` and minimises @@ -380,7 +376,7 @@ def _profile_neg_ll(self, idx: int, value: float) -> float: j for j in range(len(fixed)) if j != idx and j not in user_fixed ] - def neg_ll(theta): + def neg_ll(theta: npt.NDArray) -> Any: return float( self.dist._neg_ll_func( self.surv_data, *theta, self.gamma, self.f0, self.p @@ -391,7 +387,7 @@ def neg_ll(theta): # Single-parameter distribution: nothing left to profile over. return neg_ll(fixed) - def obj(free_vals): + def obj(free_vals: npt.NDArray) -> Any: theta = fixed.copy() theta[free_idx] = free_vals return neg_ll(theta) @@ -482,10 +478,10 @@ def _param_cb_lr( else: se = 0.5 * abs(theta_hat) if theta_hat != 0 else 1.0 - def deviance(v): + def deviance(v: npt.NDArray) -> Any: return 2.0 * (self._profile_neg_ll(idx, v) - nll_hat) - def solve_side(direction): + def solve_side(direction: Any) -> Any: limit = hi_b if direction > 0 else lo_b below = theta_hat # deviance(below) ~ 0 < crit step = se @@ -1195,7 +1191,7 @@ def cb( return cb - def _cb_lr_on_func(self, on): + def _cb_lr_on_func(self, on: str) -> Any: """Return ``g(t, theta)`` for the requested ``on`` function. Evaluates the chosen distribution function at a single time for a @@ -1206,7 +1202,7 @@ def _cb_lr_on_func(self, on): if on not in valid: raise ValueError(f"'on' must be one of {valid}") - def g(t, theta): + def g(t: Any, theta: npt.NDArray) -> Any: xt = np.atleast_1d(t) - self.gamma if on in ("sf", "R"): return self.dist.sf(xt, *theta)[0] @@ -1220,7 +1216,7 @@ def g(t, theta): return g - def _cb_lr(self, t, on, alpha_ci, bound): + def _cb_lr(self, t: Any, on: str, alpha_ci: float, bound: str) -> Any: """Profile-likelihood (likelihood-ratio) band on a model function. At each time the bound is the extreme value of the ``on`` function over @@ -1253,7 +1249,7 @@ def _cb_lr(self, t, on, alpha_ci, bound): ) ) - def deviance(theta): + def deviance(theta: npt.NDArray) -> Any: return 2.0 * ( float( self.dist._neg_ll_func( @@ -1286,7 +1282,7 @@ def deviance(theta): order = np.argsort(t) t_sorted = t[order] - def extreme(time, sign, warm): + def extreme(time: Any, sign: Any, warm: Any) -> Any: # sign = +1 minimises g (lower bound); -1 maximises g (upper). res = minimize( lambda th: sign * g(time, th), @@ -1323,7 +1319,7 @@ def extreme(time, sign, warm): else: return hi_vals[inv] - def _cb_context(self): + def _cb_context(self) -> Any: """Assemble the parameter vector and covariance used by ``cb``. The variance is computed over the extended parameter vector @@ -1355,7 +1351,7 @@ def _cb_context(self): return _CBContext(phi_hat=phi_hat, cov=cov, n_core=n_core) - def _cb_unpack(self, phi, ctx): + def _cb_unpack(self, phi: npt.NDArray, ctx: Any) -> Any: """Split an extended parameter vector into ``(core, p, f0)``.""" core = phi[: ctx.n_core] i = ctx.n_core @@ -1367,7 +1363,7 @@ def _cb_unpack(self, phi, ctx): f0 = phi[i] if self.zi else 0.0 return core, p, f0 - def _cb_full_sf(self, x, phi, ctx): + def _cb_full_sf(self, x: Any, phi: npt.NDArray, ctx: Any) -> Any: """Survival function including the LFP and zero-inflation mass. Points below the (offset) support are clamped *before* the base sf is @@ -1386,12 +1382,14 @@ def _cb_full_sf(self, x, phi, ctx): base_sf = np.where(below, 1.0, self.dist.sf(xg, *core)) return 1 - p + (p - f0) * base_sf - def _cb_delta_var(self, func, ctx): + def _cb_delta_var(self, func: Callable[..., Any], ctx: Any) -> Any: """First-order delta-method variance: ``Var(g) = J Sigma J^T``.""" jac = np.atleast_2d(jacobian(func)(ctx.phi_hat)) return np.einsum("ij,jk,ik->i", jac, ctx.cov, jac) - def _cb_sf_bound(self, x, ctx, alpha_ci, bound): + def _cb_sf_bound( + self, x: npt.ArrayLike, ctx: Any, alpha_ci: float, bound: str + ) -> Any: """Confidence bound on the survival function via a logit transform. Working on the logit of R keeps the bound within ``(0, 1)``. The @@ -1399,7 +1397,7 @@ def _cb_sf_bound(self, x, ctx, alpha_ci, bound): layout the public ``cb`` method expects. """ - def sf_func(phi): + def sf_func(phi: npt.NDArray) -> Any: return self._cb_full_sf(x, phi, ctx) var_R = self._cb_delta_var(sf_func, ctx) @@ -1424,7 +1422,9 @@ def sf_func(phi): R_cb = np.where(np.broadcast_to(R_hat == 0.0, R_cb.shape), 0.0, R_cb) return R_cb.T - def _cb_rate_bound(self, t, ctx, alpha_ci, bound, on): + def _cb_rate_bound( + self, t: Any, ctx: Any, alpha_ci: float, bound: str, on: str + ) -> Any: """Confidence bound on the hazard (``hf``) or density (``df``). Both are non-negative, so the bound is computed on the log scale to @@ -1432,13 +1432,13 @@ def _cb_rate_bound(self, t, ctx, alpha_ci, bound, on): rate function rather than differentiating the ``Hf`` bound curve. """ - def density(phi): + def density(phi: npt.NDArray) -> Any: core, p, f0 = self._cb_unpack(phi, ctx) return (p - f0) * self.dist.df(t - self.gamma, *core) if on == "hf": - def func(phi): + def func(phi: npt.NDArray) -> Any: return density(phi) / self._cb_full_sf(t, phi, ctx) else: @@ -1462,7 +1462,7 @@ def func(phi): # neg_ll/aic/bic/aic_c come from InformationCriteriaMixin. The aic_c # correction uses the same parameter count as the aic() penalty it # corrects — including gamma / p / f0 when fitted (#256). - def _ic_counts(self): + def _ic_counts(self) -> Any: n, c = self.data["n"], self.data["c"] return n[c == 0].sum(), n.sum() @@ -1507,7 +1507,7 @@ def get_plot_data( and (self.hess_inv is not None) ): - def _cb_func(x_model): + def _cb_func(x_model: npt.NDArray) -> Any: return self.cb(x_model, on="ff", alpha_ci=alpha_ci) cb_func = _cb_func diff --git a/surpyval/univariate/parametric/parametric_fitter.py b/surpyval/univariate/parametric/parametric_fitter.py index 1fc709a..6c3d7b1 100755 --- a/surpyval/univariate/parametric/parametric_fitter.py +++ b/surpyval/univariate/parametric/parametric_fitter.py @@ -92,6 +92,27 @@ def reject_structural_params( ) +def _imputed_data( + x: npt.NDArray, c: npt.NDArray, n: npt.NDArray +) -> SurpyvalData: + """Wrap ``_initial_guess``'s working copy as a ``SurpyvalData``. + + ``group_and_sort=False`` because this is not user input. The rows + have already been validated once, and merging duplicates or + reordering them would change what the initialisers see for no gain. + + The truncation bounds are deliberately left at their defaults rather + than carried over from the data being seeded. The imputation moves + interval- and left-censored points to a midpoint, which can put an + observation at or before its own left-truncation time -- a + contradiction ``xcnt_handler`` rejects outright (#260). Seeding is + not inference, so the untruncated copy is the right one: it is what + every initialiser has always been given, since no caller ever passed + ``t`` down. + """ + return SurpyvalData(x=x, c=c, n=n, group_and_sort=False) + + PARA_METHODS = ["MPP", "MLE", "MPS", "MSE", "MOM"] METHOD_FUNC_DICT = {"MPP": mpp, "MOM": mom, "MLE": mle, "MPS": mps, "MSE": mse} @@ -129,7 +150,7 @@ class ParametricFitter: A distribution needs only ``hf`` and ``Hf`` (or ``sf``, ``ff`` and ``df``) plus a ``_parameter_initialiser`` with the signature - ``(self, x, c=None, n=None, t=None, offset=False)`` for fitting to + ``(self, data: SurpyvalData, offset: bool = False)`` for fitting to work; ``log_df``, ``log_sf``, ``log_ff`` and ``random`` have generic implementations here that subclasses can override with closed forms. Probability plotting (the MPP fit method and ``Parametric.plot``) @@ -159,6 +180,35 @@ class ParametricFitter: # validation and callers branch on the trait. discrete = False + if TYPE_CHECKING: + # The distribution functions every subclass supplies and this + # base calls -- ``cs`` divides two ``sf``s, ``log_sf`` negates + # ``Hf``, ``random`` inverts ``qf``, and the four ``ll_*`` + # methods are written in terms of ``hf``, ``Hf`` and the log + # densities. The docstring above already states the contract + # ("a distribution needs only hf and Hf, or sf, ff and df"); + # this is the same statement in a form the checker reads. + # + # Declared, not defined: a body here would give every + # distribution a silently wrong inherited implementation + # instead of the AttributeError that correctly reports a + # distribution which forgot one. ``OptimisedFitMixin`` carries + # the mirror image of this block for the estimation machinery. + def sf(self, x: Any, *params: Any) -> Any: ... + def ff(self, x: Any, *params: Any) -> Any: ... + def df(self, x: Any, *params: Any) -> Any: ... + def hf(self, x: Any, *params: Any) -> Any: ... + def Hf(self, x: Any, *params: Any) -> Any: ... + def qf(self, u: Any, *params: Any) -> Any: ... + def moment(self, m: Any, *params: Any) -> Any: ... + def mpp_x_transform(self, x: Any) -> Any: ... + def mpp_y_transform(self, y: Any, *params: Any) -> Any: ... + def mpp_inv_y_transform(self, y: Any, *params: Any) -> Any: ... + + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: ... + def __init__( self, name: str, @@ -169,7 +219,7 @@ def __init__( param_map: dict[str, int], plot_x_scale: str, y_ticks: list[float] | None = None, - ): + ) -> None: self.name: str = name self.k = k self.bounds = bounds @@ -186,7 +236,7 @@ def __init__( # behaviour used by ``Uniform``. self.support_param_index = (0, 1) - def random(self, size, *params): + def random(self, size: int | tuple[int, ...], *params: Any) -> Any: r""" Draws random samples from the distribution in shape `size`, using @@ -218,23 +268,59 @@ def random(self, size, *params): U = uniform.rvs(size=size) return self.qf(U, *params) - def log_df(self, x, *params): + def log_df(self, x: npt.NDArray, *params: Any) -> Any: return np.log(self.hf(x, *params)) - self.Hf(x, *params) - def log_sf(self, x, *params): + def log_sf(self, x: Numeric, *params: Any) -> Any: return -self.Hf(x, *params) - def log_ff(self, x, *params): + def log_ff(self, x: Numeric, *params: Any) -> Any: return np.log(-np.expm1(-self.Hf(x, *params))) - def cs(self, x, X, *params): - # Conditional survival R(x + X) / R(X); distributions override - # this only to carry a docstring or a simplified closed form. - # The default also gives discrete distributions a working - # ``Parametric.cs`` (previously AttributeError). + def cs(self, x: Numeric, X: Numeric, *params: Any) -> Any: + r""" + + Conditional survival function: the probability of surviving a + further ``x`` given survival to ``X`` already. + + .. math:: + R(x, X) = \frac{R(x + X)}{R(X)} + + This is the definition for every distribution, so it lives here + rather than being restated on each one. ``Exponential`` + overrides it because the exponential is memoryless and + :math:`R(x, X) = R(x)`, which is both cheaper and free of the + cancellation the ratio suffers in the far tail. + + Parameters + ---------- + + x : numpy array or scalar + The additional time to survive, measured from ``X`` + X : numpy array or scalar + The time already survived + *params : numpy array like or scalar + The parameters of the distribution, in the order given by + its ``param_names`` + + Returns + ------- + + cs : scalar or numpy array + The value(s) of the conditional survival function. + + Examples + -------- + >>> import numpy as np + >>> from surpyval import Weibull + >>> x = np.array([1, 2, 3, 4, 5]) + >>> Weibull.cs(x, 5, 3, 4) + array([2.52537548e-04, 3.00394073e-10, 2.45288508e-19, 1.48999440e-32, + 5.42544000e-51]) + """ return self.sf(x + X, *params) / self.sf(X, *params) - def _plot_x_bounds(self, x, params): + def _plot_x_bounds(self, x: npt.NDArray, params: Any) -> Any: """Return (x_scale_min, x_scale_max) for probability plots. Returns None to auto-compute the bounds from the data. @@ -242,8 +328,8 @@ def _plot_x_bounds(self, x, params): return None @_check_x_not_empty - def ll_observed(self, x, n, *params): - *params, gamma, f0, p = params + def ll_observed(self, x: npt.NDArray, n: npt.NDArray, *params: Any) -> Any: + *dist_params, gamma, f0, p = params if f0 == 0: # Not zero-inflated; x == 0 is an ordinary observation. zero_weight = 0 @@ -257,35 +343,49 @@ def ll_observed(self, x, n, *params): x = x - gamma N = np.sum(n[non_zero_mask]) return ( - (n[non_zero_mask] * self.log_df(x[non_zero_mask], *params)).sum() + ( + n[non_zero_mask] * self.log_df(x[non_zero_mask], *dist_params) + ).sum() + zero_weight + N * np.log(p - f0) ) @_check_x_not_empty - def ll_right_censored(self, x, n, *params): - *params, gamma, f0, p = params + def ll_right_censored( + self, x: npt.NDArray, n: npt.NDArray, *params: Any + ) -> Any: + *dist_params, gamma, f0, p = params x = x - gamma if p == 1: - return np.sum(n * (np.log1p(-f0) + self.log_sf(x, *params))) + return np.sum(n * (np.log1p(-f0) + self.log_sf(x, *dist_params))) else: - F = self.ff(x, *params) + F = self.ff(x, *dist_params) return np.sum(n * np.log(1 - f0 - (p - f0) * F)) @_check_x_not_empty - def ll_left_censored(self, x, n, *params): - *params, gamma, f0, p = params + def ll_left_censored( + self, x: npt.NDArray, n: npt.NDArray, *params: Any + ) -> Any: + *dist_params, gamma, f0, p = params x = x - gamma if f0 == 0: # No zero-inflation: F_mix = p * F, so the numerically stable # log_ff path applies (the branch was inverted as ``f0 == 1``, # which never occurs, #256). - return np.sum(n * self.log_ff(x, *params)) + n.sum() * np.log(p) + return np.sum(n * self.log_ff(x, *dist_params)) + n.sum() * np.log( + p + ) else: - return np.sum(n * np.log(f0 + (p - f0) * self.ff(x, *params))) + return np.sum(n * np.log(f0 + (p - f0) * self.ff(x, *dist_params))) @_check_x_not_empty - def ll_interval_or_truncated(self, xl, xr, n, *params): + def ll_interval_or_truncated( + self, + xl: npt.NDArray, + xr: npt.NDArray, + n: npt.NDArray, + *params: Any, + ) -> Any: """ Log probability of falling inside each window ``(xl, xr]``. @@ -327,7 +427,7 @@ def ll_interval_or_truncated(self, xl, xr, n, *params): left-truncation likelihood unbounded (#269). For finite-bound intervals the ``f0`` terms cancel, so plain fits are unchanged. """ - *params, gamma, f0, p = params + *dist_params, gamma, f0, p = params if len(n) == 0: return 0.0 @@ -343,14 +443,18 @@ def ll_interval_or_truncated(self, xl, xr, n, *params): xr_safe = np.where(hi_finite, xr, stand_in) upper = np.where( - hi_finite, f0 + (p - f0) * self.ff(xr_safe - gamma, *params), 1.0 + hi_finite, + f0 + (p - f0) * self.ff(xr_safe - gamma, *dist_params), + 1.0, ) lower = np.where( - lo_finite, f0 + (p - f0) * self.ff(xl_safe - gamma, *params), 0.0 + lo_finite, + f0 + (p - f0) * self.ff(xl_safe - gamma, *dist_params), + 0.0, ) return np.sum(n * np.log(np.maximum(upper - lower, 0.0))) - def _log_likelihood(self, data, *params): + def _log_likelihood(self, data: SurpyvalData, *params: Any) -> Any: return ( self.ll_observed(data.x_o, data.n_o, *params) + self.ll_right_censored(data.x_r, data.n_r, *params) @@ -363,15 +467,15 @@ def _log_likelihood(self, data, *params): ) ) - def _neg_ll_func(self, data, *params): + def _neg_ll_func(self, data: SurpyvalData, *params: Any) -> Any: return -self._log_likelihood(data, *params) - def _moment(self, n, *params, offset=False): + def _moment(self, n: Any, *params: Any, offset: bool = False) -> Any: if offset: gamma = params[0] params = params[1::] - def fun(x): + def fun(x: Numeric) -> Any: return x**n * self.df((x - gamma), *params) m = quad(fun, gamma, np.inf)[0] @@ -380,13 +484,13 @@ def fun(x): m = self.moment(n, *params) else: - def fun(x): + def fun(x: Numeric) -> Any: return x**n * self.df(x, *params) m = quad(fun, *self.support)[0] return m - def _set_support(self, model, offset): + def _set_support(self, model: Any, offset: bool) -> Any: """Resolve and assign the fitted model's support interval. For an offset model the left edge is the fitted ``gamma``; @@ -414,7 +518,9 @@ def _set_support(self, model, offset): model.support = np.array([left, right]) - def from_params(self, params, gamma=None, p=None, f0=None): + def from_params( + self, params: Any, gamma: Any = None, p: Any = None, f0: Any = None + ) -> Any: r""" Creating a SurPyval Parametric class with provided parameters. @@ -566,7 +672,14 @@ class OptimisedFitMixin: supports_mpp: bool support_param_index: tuple[int, int] - def _parameter_initialiser(self, *args: Any, **kwargs: Any) -> Any: ... + # Every implementation returns a 1-D float array. It used to + # be a tuple in nine, an array in six, a list in one and a + # fitted model's .params in five -- and a bare scalar in + # Rayleigh, which made the seed 0-dimensional and broke the + # lfp and zi paths outright. + def _parameter_initialiser( + self, data: SurpyvalData, offset: bool = False + ) -> npt.NDArray: ... def _neg_ll_func(self, data: Any, *params: Any) -> Any: ... def _log_likelihood(self, data: Any, *params: Any) -> Any: ... def _moment(self, n: Any, *p: Any, offset: bool = False) -> Any: ... @@ -574,13 +687,16 @@ def _set_support(self, model: Any, offset: Any) -> Any: ... def sf(self, x: Any, *params: Any) -> Any: ... def ff(self, x: Any, *params: Any) -> Any: ... def df(self, x: Any, *params: Any) -> Any: ... + def hf(self, x: Any, *params: Any) -> Any: ... def Hf(self, x: Any, *params: Any) -> Any: ... def qf(self, u: Any, *params: Any) -> Any: ... def mpp_x_transform(self, x: Any, *args: Any) -> Any: ... def mpp_y_transform(self, y: Any, *params: Any) -> Any: ... def mpp_inv_y_transform(self, y: Any, *params: Any) -> Any: ... - def neg_mean_D(self, x, c, n, tl, tr, *params): + def neg_mean_D( + self, x: npt.NDArray, c: Any, n: Any, tl: Any, tr: Any, *params: Any + ) -> Any: mask = c == 0 x_obs = x[mask] n_obs = n[mask] @@ -636,7 +752,7 @@ def neg_mean_D(self, x, c, n, tl, tr, *params): obj = obj + np.sum(n[c == -1] * np.log(Dl)) return -obj / n.sum() - def mom_moment_gen(self, *params, offset=False): + def mom_moment_gen(self, *params: Any, offset: bool = False) -> Any: if offset: k = self.k + 1 else: @@ -647,7 +763,14 @@ def mom_moment_gen(self, *params, offset=False): moments[i] = self._moment(n, *params, offset=offset) return moments - def _check_identifiable(self, surv_data, offset, lfp, zi, fixed): + def _check_identifiable( + self, + surv_data: SurpyvalData, + offset: bool, + lfp: bool, + zi: bool, + fixed: dict[str, float] | None, + ) -> Any: """ Reject data that cannot pin down the free parameters. @@ -701,15 +824,15 @@ def _check_identifiable(self, surv_data, offset, lfp, zi, fixed): def _validate_fit_inputs( self, - surv_data, - how, - offset, - lfp, - zi, - fixed, - heuristic, - turnbull_estimator, - ): + surv_data: SurpyvalData, + how: str, + offset: bool, + lfp: bool, + zi: bool, + fixed: dict[str, float] | None, + heuristic: str, + turnbull_estimator: str, + ) -> Any: # Offsetting (a free location/threshold ``gamma``) only makes sense # for distributions supported on a half-line ``[0, inf)``. A # distribution with a finite upper bound (e.g. Beta on ``[0, 1]``) @@ -1089,7 +1212,7 @@ def fit_from_df( xr: str | None = None, tl: str | float | None = None, tr: str | float | None = None, - **fit_options, + **fit_options: Any, ) -> Parametric: r""" The central feature to SurPyval's capability. This function aimed to @@ -1218,11 +1341,11 @@ def fit_from_ecdf(self, x: npt.ArrayLike, F: npt.ArrayLike) -> Parametric: return model - def fit_from_non_parametric(self, non_parametric_model) -> Parametric: + def fit_from_non_parametric(self, non_parametric_model: Any) -> Parametric: x, F = non_parametric_model.x, 1 - non_parametric_model.R return self.fit_from_ecdf(x, F) - def _clamp_truncation_to_support(self, t): + def _clamp_truncation_to_support(self, t: Any) -> Any: """Clamp the truncation bounds to the distribution's support. Returns the left and right truncation arrays with any value that @@ -1240,7 +1363,14 @@ def _clamp_truncation_to_support(self, t): return tl, tr - def _initial_guess(self, x, c, n, offset, zi, lfp, heuristic): + def _initial_guess( + self, + data: SurpyvalData, + offset: bool, + zi: bool, + lfp: bool, + heuristic: str, + ) -> npt.NDArray: """Derive an initial parameter vector for the iterative fitters. Builds a working copy of the data with interval- and @@ -1249,7 +1379,13 @@ def _initial_guess(self, x, c, n, offset, zi, lfp, heuristic): the limited-failure (``p``) and zero-inflation (``f0``) seeds when those models are requested. The returned vector is in the natural (untransformed) parameter space. + + The working copy is rewrapped as a ``SurpyvalData`` before it is + handed on, rather than the caller's own object being forwarded: + the imputation rewrites ``x`` and ``c``, and the masks below drop + rows, so the caller's object no longer describes it. """ + x, c, n = data.x, data.c, data.n if x.ndim == 2: # If x has 2 dims, then there is intervally # censored data. Simply take the midpoint to @@ -1280,7 +1416,9 @@ def _initial_guess(self, x, c, n, offset, zi, lfp, heuristic): ): with np.errstate(all="ignore"): init = np.array( - self._parameter_initialiser(x_init, c_init, n_init) + self._parameter_initialiser( + _imputed_data(x_init, c_init, n_init) + ) ) else: with np.errstate(all="ignore"): @@ -1307,7 +1445,7 @@ def _initial_guess(self, x, c, n, offset, zi, lfp, heuristic): # Create an initial estimate with the new points init = self._parameter_initialiser( - x_init, c_init, n_init, offset=offset + _imputed_data(x_init, c_init, n_init), offset=offset ) init = np.array(init) @@ -1409,9 +1547,7 @@ def fit_from_surpyval_data( results = self._fit_numerically( model, fitting_info, - x, - c, - n, + surv_data, tl, tr, how, @@ -1492,7 +1628,13 @@ def fit_from_surpyval_data( return model def _try_closed_form_mle( - self, surv_data, how, offset, lfp, zi, fixed + self, + surv_data: SurpyvalData, + how: str, + offset: bool, + lfp: bool, + zi: bool, + fixed: dict[str, float] | None, ) -> "dict | None": """An exact analytic MLE, or ``None`` to use the optimiser. @@ -1528,23 +1670,21 @@ def _try_closed_form_mle( def _fit_numerically( self, - model, - fitting_info, - x, - c, - n, - tl, - tr, - how, - offset, - zi, - lfp, - fixed, - heuristic, - init, - rr, - on_d_is_0, - turnbull_estimator, + model: Any, + fitting_info: Any, + surv_data: SurpyvalData, + tl: Any, + tr: Any, + how: str, + offset: bool, + zi: bool, + lfp: bool, + fixed: dict[str, float] | None, + heuristic: str, + init: Any, + rr: str, + on_d_is_0: bool, + turnbull_estimator: str, ) -> dict: """Seed an initial guess, convert bounds and run the estimator.""" if how == "MPS": @@ -1557,7 +1697,7 @@ def _fit_numerically( if how != "MPP": transform, inv_trans, const, fixed_idx, not_fixed = bounds_convert( - x, model.bounds, fixed, model.param_map + surv_data.x, model.bounds, fixed, model.param_map ) fitting_info["inv_trans"] = inv_trans fitting_info["const"] = const @@ -1566,7 +1706,9 @@ def _fit_numerically( # ``len``-based check: comparing an ndarray to ``[]`` raises a # broadcast error (#261). if init is None or len(np.atleast_1d(init)) == 0: - init = self._initial_guess(x, c, n, offset, zi, lfp, heuristic) + init = self._initial_guess( + surv_data, offset, zi, lfp, heuristic + ) init = np.atleast_1d(init) if fixed and len(init) == len(not_fixed): # type: ignore[arg-type] diff --git a/surpyval/univariate/parametric/probability_plotting.py b/surpyval/univariate/parametric/probability_plotting.py index 673a45a..1dc6327 100644 --- a/surpyval/univariate/parametric/probability_plotting.py +++ b/surpyval/univariate/parametric/probability_plotting.py @@ -1,3 +1,7 @@ +from typing import Any, Callable + +import numpy.typing as npt + """ Shared probability plot construction for parametric models. @@ -17,7 +21,9 @@ CB_COLOUR = "#e94c54" -def adjust_heuristic(c, t, heuristic): +def adjust_heuristic( + c: npt.NDArray, t: npt.NDArray | None, heuristic: str +) -> str: """ Force the Turnbull heuristic when the data is interval censored or truncated, warning that the requested heuristic was changed. @@ -42,17 +48,17 @@ def adjust_heuristic(c, t, heuristic): def probability_plot_data( - dist, - ff, - x, - c, - n, - t, - heuristic="Nelson-Aalen", - gamma=0.0, - params=None, - cb_func=None, -): + dist: Any, + ff: Callable[..., Any], + x: npt.NDArray, + c: npt.NDArray | None, + n: npt.NDArray, + t: npt.NDArray, + heuristic: str = "Nelson-Aalen", + gamma: float = 0.0, + params: npt.NDArray | None = None, + cb_func: Callable[..., Any] | None = None, +) -> Any: """ Compute everything needed to draw a probability plot of the data against the fitted CDF ``ff``. @@ -178,13 +184,13 @@ def probability_plot_data( def draw_probability_plot( - ax, - d, - y_transform, - inv_y_transform, - title, - plot_bounds=False, -): + ax: Any, + d: Any, + y_transform: Callable[..., Any], + inv_y_transform: Callable[..., Any], + title: str, + plot_bounds: Any = False, +) -> Any: """ Draw the probability plot described by the ``probability_plot_data`` dictionary ``d`` onto the matplotlib axes ``ax``. diff --git a/surpyval/univariate/parametric/royston_parmar.py b/surpyval/univariate/parametric/royston_parmar.py index fb663d2..849c386 100644 --- a/surpyval/univariate/parametric/royston_parmar.py +++ b/surpyval/univariate/parametric/royston_parmar.py @@ -1,3 +1,5 @@ +import numpy.typing as npt + """ Royston-Parmar flexible parametric survival models. @@ -35,7 +37,7 @@ each observation's contribution by ``S(t_l) - S(t_r)``. """ -from typing import Any +from typing import Any, Callable import numpy as np from scipy.optimize import brentq, minimize @@ -89,7 +91,7 @@ def _place_knots(x_events: np.ndarray, n_internal: int) -> np.ndarray: return np.quantile(lx, qs) -def _scale_terms(eta: np.ndarray, scale: str): +def _scale_terms(eta: np.ndarray, scale: str) -> tuple[Any, ...]: """``(log S, log(-dS/deta))`` at linear predictor ``eta`` for a scale.""" if scale == "hazard": log_S = -np.exp(eta) @@ -113,7 +115,7 @@ def _sf_from_eta(eta: np.ndarray, scale: str) -> np.ndarray: def _sf_at( times: np.ndarray, knots: np.ndarray, gamma: np.ndarray, scale: str -): +) -> npt.NDArray: """Survival at arbitrary times, with the boundary conventions the censoring/truncation likelihoods need: ``S = 1`` at times ``<= 0`` (and ``-inf``) and ``S = 0`` at ``+inf``. Finite positive times go through the @@ -325,7 +327,9 @@ def from_dict(cls, model_dict: dict) -> "RoystonParmarModel": return out -def _numerical_hessian(f, x, eps=1e-5): +def _numerical_hessian( + f: Callable[..., Any], x: npt.NDArray, eps: float = 1e-05 +) -> npt.NDArray: n = len(x) H = np.zeros((n, n)) steps = np.maximum(np.abs(x), 1.0) * eps @@ -480,7 +484,7 @@ def fit( B_il = _rcs_basis(np.log(x_il), knots) if x_il.size else None B_ir = _rcs_basis(np.log(x_ir), knots) if x_ir.size else None - def neg_ll(g): + def neg_ll(g: npt.NDArray) -> Any: ll = 0.0 if B_o is not None: # events: log f = log(-dS) + log s' - log t eta = B_o @ g diff --git a/surpyval/univariate/regression/accelerated_life/accelerated_life.py b/surpyval/univariate/regression/accelerated_life/accelerated_life.py index 4626f27..8b3e131 100644 --- a/surpyval/univariate/regression/accelerated_life/accelerated_life.py +++ b/surpyval/univariate/regression/accelerated_life/accelerated_life.py @@ -1,5 +1,10 @@ import autograd.numpy as np +from surpyval.univariate.parametric.parametric_fitter import ( + OptimisedFitMixin, +) + +from .lifemodel import LifeModel from .parameter_substitution import ParameterSubstitutionFitter # Map each supported distribution to its life parameter name and any @@ -19,7 +24,9 @@ } -def AcceleratedLife(distribution, life_model): +def AcceleratedLife( + distribution: OptimisedFitMixin, life_model: LifeModel +) -> ParameterSubstitutionFitter: """ Create an Accelerated Life fitter for the given distribution and life model. diff --git a/surpyval/univariate/regression/accelerated_life/dual_exponential.py b/surpyval/univariate/regression/accelerated_life/dual_exponential.py index b4c3775..7c8f3ca 100644 --- a/surpyval/univariate/regression/accelerated_life/dual_exponential.py +++ b/surpyval/univariate/regression/accelerated_life/dual_exponential.py @@ -30,7 +30,7 @@ class DualExponential_(LifeModel): on observed data. """ - def __init__(self): + def __init__(self) -> None: """ Initialize the DualExponential_ class. diff --git a/surpyval/univariate/regression/accelerated_life/dual_power.py b/surpyval/univariate/regression/accelerated_life/dual_power.py index e93e8c7..edbea8e 100644 --- a/surpyval/univariate/regression/accelerated_life/dual_power.py +++ b/surpyval/univariate/regression/accelerated_life/dual_power.py @@ -5,7 +5,7 @@ class DualPower_(LifeModel): - def __init__(self): + def __init__(self) -> None: super().__init__( "DualPower", {"c": 0, "m": 1, "n": 2}, diff --git a/surpyval/univariate/regression/accelerated_life/exponential.py b/surpyval/univariate/regression/accelerated_life/exponential.py index 4b0d3f6..17b95c7 100644 --- a/surpyval/univariate/regression/accelerated_life/exponential.py +++ b/surpyval/univariate/regression/accelerated_life/exponential.py @@ -5,7 +5,7 @@ class InverseExponential_(LifeModel): - def __init__(self): + def __init__(self) -> None: super().__init__( "InverseExponential", {"a": 0, "b": 1}, @@ -27,7 +27,7 @@ def phi_init(self, life: float, Z: ndarray) -> list[float]: class ExponentialLifeModel_(LifeModel): - def __init__(self): + def __init__(self) -> None: super().__init__( "Exponential", {"a": 0, "b": 1}, diff --git a/surpyval/univariate/regression/accelerated_life/eyring.py b/surpyval/univariate/regression/accelerated_life/eyring.py index 41ab9cf..0a8c3c9 100644 --- a/surpyval/univariate/regression/accelerated_life/eyring.py +++ b/surpyval/univariate/regression/accelerated_life/eyring.py @@ -5,7 +5,7 @@ class Eyring_(LifeModel): - def __init__(self): + def __init__(self) -> None: super().__init__( "Eyring", {"a": 0, "b": 1}, @@ -27,7 +27,7 @@ def phi_init(self, life: float, Z: ndarray) -> list[float]: class InverseEyring_(LifeModel): - def __init__(self): + def __init__(self) -> None: super().__init__( "InverseEyring", {"a": 0, "c": 1}, diff --git a/surpyval/univariate/regression/accelerated_life/general_log_linear.py b/surpyval/univariate/regression/accelerated_life/general_log_linear.py index e938d5e..aba9860 100644 --- a/surpyval/univariate/regression/accelerated_life/general_log_linear.py +++ b/surpyval/univariate/regression/accelerated_life/general_log_linear.py @@ -5,11 +5,22 @@ class GeneralLogLinear_(LifeModel): - def __init__(self): + def __init__(self) -> None: super().__init__( "GeneralLogLinear", - lambda Z: (((None, None),) * Z.shape[1]), - lambda Z: {"beta_" + str(i): i for i in range(Z.shape[1])}, + # Swapped until 0.19.1: the bounds lambda sat in the + # phi_param_map slot and vice versa. LifeModel takes + # (name, phi_param_map, phi_bounds). + # + # Both are callables of Z rather than the dict and tuple the + # base declares, because this model's parameterisation + # depends on the covariate dimension. That is why it is left + # out of LIFE_MODELS and cannot be rebuilt from a name -- and + # why this module is not in the type ratchet. + lambda Z: { # type: ignore[arg-type] + "beta_" + str(i): i for i in range(Z.shape[1]) + }, + lambda Z: (((None, None),) * Z.shape[1]), # type: ignore[arg-type] ) def phi(self, Z: ndarray, *params: float) -> ndarray: diff --git a/surpyval/univariate/regression/accelerated_life/lifemodel.py b/surpyval/univariate/regression/accelerated_life/lifemodel.py index 31f550f..11268c8 100644 --- a/surpyval/univariate/regression/accelerated_life/lifemodel.py +++ b/surpyval/univariate/regression/accelerated_life/lifemodel.py @@ -8,8 +8,8 @@ def __init__( self, name: str, phi_param_map: dict[str, int], - phi_bounds: tuple[tuple[int | None, int | None]], - ): + phi_bounds: tuple[tuple[int | None, int | None], ...], + ) -> None: self.name = name self.phi_param_map = phi_param_map self.phi_bounds = phi_bounds diff --git a/surpyval/univariate/regression/accelerated_life/linear.py b/surpyval/univariate/regression/accelerated_life/linear.py index 3a8d74e..ad25309 100644 --- a/surpyval/univariate/regression/accelerated_life/linear.py +++ b/surpyval/univariate/regression/accelerated_life/linear.py @@ -5,7 +5,7 @@ class Linear_(LifeModel): - def __init__(self): + def __init__(self) -> None: super().__init__( "Linear", {"a": 0, "b": 1}, diff --git a/surpyval/univariate/regression/accelerated_life/parameter_substitution.py b/surpyval/univariate/regression/accelerated_life/parameter_substitution.py index 613c28b..761f8f7 100644 --- a/surpyval/univariate/regression/accelerated_life/parameter_substitution.py +++ b/surpyval/univariate/regression/accelerated_life/parameter_substitution.py @@ -1,17 +1,23 @@ -import inspect import warnings +from typing import Callable import autograd.numpy as np import numpy.typing as npt from scipy.optimize import minimize from surpyval.univariate.parametric.fitters import bounds_convert +from surpyval.univariate.parametric.parametric_fitter import ( + Boxable, + Numeric, + OptimisedFitMixin, +) from surpyval.utils.surpyval_data import SurpyvalData from .._fit_skeleton import HazardIdentitiesMixin from .._likelihood import regression_neg_ll from ..parametric_regression_model import ParametricRegressionModel from ..regression_data import DataFrameRegressionMixin +from .lifemodel import LifeModel class ParameterSubstitutionFitter( @@ -19,15 +25,15 @@ class ParameterSubstitutionFitter( ): def __init__( self, - kind, - name, - distribution, - life_model, - life_parameter, - baseline=None, - param_transform=None, - inverse_param_transform=None, - ): + kind: str, + name: str, + distribution: OptimisedFitMixin, + life_model: LifeModel, + life_parameter: str, + baseline: list[str] | str | None = None, + param_transform: Callable[[Boxable], Boxable] | None = None, + inverse_param_transform: Callable[[Boxable], Boxable] | None = None, + ) -> None: if baseline is None: baseline = [] elif not isinstance(baseline, list): @@ -58,25 +64,28 @@ def __init__( self.param_transform = lambda x: x self.inverse_param_transform = lambda x: x else: + # Supplied as a pair -- accelerated_life.py passes both or + # neither -- so the inverse is not None here. + assert inverse_param_transform is not None self.param_transform = param_transform self.inverse_param_transform = inverse_param_transform - def Hf(self, x, Z, *params): + def Hf(self, x: Numeric, Z: Numeric, *params: Boxable) -> Boxable: x = np.array(x) if np.isscalar(Z): - Z = np.ones_like(x) * Z + Z_arr = np.ones_like(x) * Z else: - Z = np.array(Z) - if Z.ndim == 1: + Z_arr = np.array(Z) + if Z_arr.ndim == 1: # A 1-D stress vector (one stress variable) becomes a single # column so the per-stress masking below works (#261). - Z = Z.reshape(-1, 1) + Z_arr = Z_arr.reshape(-1, 1) dist_params = np.array(params[0 : self.k_dist]) phi_params = np.array(params[self.k_dist :]) Hf = np.zeros_like(x) - stresses = np.unique(Z, axis=0) + stresses = np.unique(Z_arr, axis=0) for stress in stresses: life_param_mask = ( np.arange(len(dist_params)) @@ -87,27 +96,27 @@ def Hf(self, x, Z, *params): self.param_transform(self.phi(stress, *phi_params)), dist_params, ) - mask = (Z == stress).all(axis=1) + mask = (Z_arr == stress).all(axis=1) Hf = np.where(mask, self.Hf_dist(x, *dist_params_i), Hf) return Hf - def hf(self, x, Z, *params): + def hf(self, x: Numeric, Z: Numeric, *params: Boxable) -> Boxable: x = np.array(x) if np.isscalar(Z): - Z = np.ones_like(x) * Z + Z_arr = np.ones_like(x) * Z else: - Z = np.array(Z) - if Z.ndim == 1: + Z_arr = np.array(Z) + if Z_arr.ndim == 1: # A 1-D stress vector (one stress variable) becomes a single # column so the per-stress masking below works (#261). - Z = Z.reshape(-1, 1) + Z_arr = Z_arr.reshape(-1, 1) dist_params = np.array(params[0 : self.k_dist]) phi_params = np.array(params[self.k_dist :]) hf = np.zeros_like(x) - for stress in np.unique(Z, axis=0): + for stress in np.unique(Z_arr, axis=0): life_param_mask = ( np.arange(len(dist_params)) == self.param_map[self.life_parameter] @@ -117,7 +126,7 @@ def hf(self, x, Z, *params): self.param_transform(self.phi(stress, *phi_params)), dist_params, ) - mask = (Z == stress).all(axis=1) + mask = (Z_arr == stress).all(axis=1) hf = np.where(mask, self.hf_dist(x, *dist_params_i), hf) return hf @@ -126,42 +135,52 @@ def hf(self, x, Z, *params): # Hf and hf above already do the scalar/1-D stress coercion (#261), # so the identities need no preamble of their own. - def _parameter_initialiser_dist(self, x, c=None, n=None, t=None): - out = [] + def _parameter_initialiser_dist( + self, + x: Numeric, + c: Numeric | None = None, + n: Numeric | None = None, + t: Numeric | None = None, + ) -> list[float]: + out: list[float] = [] for low, high in self.bounds: if (low is None) and (high is None): - out.append(0) + out.append(0.0) elif high is None: - out.append(low + 1.0) + assert low is not None # both-None handled above + out.append(float(low) + 1.0) elif low is None: - out.append(high - 1.0) + out.append(float(high) - 1.0) else: - out.append((high + low) / 2.0) + out.append((float(high) + float(low)) / 2.0) return out - def mpp_inv_y_transform(self, y, *params): + def mpp_inv_y_transform(self, y: Numeric, *params: Boxable) -> Numeric: return y - def mpp_y_transform(self, y, *params): + def mpp_y_transform(self, y: Numeric, *params: Boxable) -> Numeric: return y - def mpp_x_transform(self, x, gamma=0): + def mpp_x_transform(self, x: Numeric, gamma: Boxable = 0) -> Boxable: return x - gamma - def random(self, size, Z, *params): + def random( + self, size: int, Z: Numeric | tuple[float, float], *params: Boxable + ) -> tuple[npt.NDArray, npt.NDArray]: dist_params = np.array(params[0 : self.k_dist]) phi_params = np.array(params[self.k_dist :]) x = [] Z_out = [] if isinstance(Z, tuple): + # A (low, high) pair draws the stresses uniformly. Z = np.random.uniform(*Z, size) - Z = np.asarray(Z) - if Z.ndim == 1: - Z = Z.reshape(-1, 1) + Z_arr = np.asarray(Z) + if Z_arr.ndim == 1: + Z_arr = Z_arr.reshape(-1, 1) - for stress in np.unique(Z, axis=0): + for stress in np.unique(Z_arr, axis=0): life_param_mask = ( np.arange(len(dist_params)) == self.param_map[self.life_parameter] @@ -181,7 +200,7 @@ def random(self, size, Z, *params): Z_out.append(np.ones((size, cols)) * stress) return np.array(x).flatten(), np.concatenate(Z_out) - def neg_ll(self, data, *params): + def neg_ll(self, data: SurpyvalData, *params: Boxable) -> Boxable: return regression_neg_ll(self, data, *params) def fit( @@ -244,15 +263,12 @@ def fit( parameter_data = self.inverse_param_transform(parameter_data) - if callable(self.life_model.phi_init): - if str(inspect.signature(self.life_model.phi_init)) == "(Z)": - phi_init = self.life_model.phi_init(Z) - else: - phi_init = self.life_model.phi_init( - parameter_data, stress_data - ) - else: - phi_init = self.life_model.phi_init + # Every life model's phi_init is (life, Z). There used to be + # a branch here for a "(Z)"-only signature, chosen by + # comparing str(inspect.signature(...)) == "(Z)", and another + # for a non-callable phi_init. Neither could run: all ten + # life models are callable with the two-argument signature. + phi_init = self.life_model.phi_init(parameter_data, stress_data) init = np.array([*dist_init, *phi_init]) else: init = np.array(init) @@ -295,7 +311,7 @@ def fit( with np.errstate(all="ignore"): - def fun(params): + def fun(params: npt.NDArray) -> Boxable: return self.neg_ll(data, *inv_trans(const(params))) res1 = minimize( diff --git a/surpyval/univariate/regression/accelerated_life/power.py b/surpyval/univariate/regression/accelerated_life/power.py index 78ad6c3..aba7d7b 100644 --- a/surpyval/univariate/regression/accelerated_life/power.py +++ b/surpyval/univariate/regression/accelerated_life/power.py @@ -5,7 +5,7 @@ class InversePower_(LifeModel): - def __init__(self): + def __init__(self) -> None: super().__init__( "InversePower", {"a": 0, "n": 1}, @@ -25,7 +25,7 @@ def phi_init(self, life: float, Z: ndarray) -> list[float]: class Power_(LifeModel): - def __init__(self): + def __init__(self) -> None: super().__init__( "Power", {"a": 0, "n": 1}, diff --git a/surpyval/univariate/regression/accelerated_life/power_exponential.py b/surpyval/univariate/regression/accelerated_life/power_exponential.py index 543b597..ea8a21b 100644 --- a/surpyval/univariate/regression/accelerated_life/power_exponential.py +++ b/surpyval/univariate/regression/accelerated_life/power_exponential.py @@ -5,7 +5,7 @@ class PowerExponential_(LifeModel): - def __init__(self): + def __init__(self) -> None: super().__init__( "PowerExponential", {"c": 0, "a": 1, "n": 2}, diff --git a/surpyval/univariate/regression/parametric_regression_model.py b/surpyval/univariate/regression/parametric_regression_model.py index efa9d8e..a80d56b 100644 --- a/surpyval/univariate/regression/parametric_regression_model.py +++ b/surpyval/univariate/regression/parametric_regression_model.py @@ -238,6 +238,7 @@ def from_dict(cls, model_dict: dict) -> "ParametricRegressionModel": """ import surpyval from surpyval.univariate.parametric.parametric_fitter import ( + OptimisedFitMixin, ParametricFitter, ) @@ -274,6 +275,19 @@ def from_dict(cls, model_dict: dict) -> "ParametricRegressionModel": "Cannot deserialise Accelerated Life model with life " "model {!r}".format(life_name) ) + # The guard above only establishes a ParametricFitter, which + # admits Bernoulli, Binomial and ExactEventTime -- none of + # them fittable, and an accelerated life model needs a + # distribution it can fit. The dict is untrusted input, so a + # name like that would otherwise get this far and fail deep + # inside the fitter on a missing attribute. + if not isinstance(dist, OptimisedFitMixin): + raise ValueError( + "Cannot deserialise Accelerated Life model with " + "distribution {!r}: it has no fitting machinery.".format( + model_dict["distribution"] + ) + ) reg_model = LIFE_MODELS[life_name] fitter = AcceleratedLife(dist, reg_model) elif kind in _SERIALISABLE_KINDS: