ADR-0123: #572 narrowed, not killed — log families refused on the math, the linear-Gaussian case still open - #637
Merged
Conversation
… own family's residual is linear in it, so the coupled scale/offset projection is not adopted (#572, ADR-0123) #572 proposed profiling an observable's additive offset -- and the coupled (scale, offset) pair -- out of the search by variable projection, and asked for an evaluation before anything was built, with a stated kill criterion. The evaluation was run. It kills the proposal, and the reason is one neither the issue's case-for nor its case-against anticipated. Variable projection is a linear-least-squares identity: it needs the family's residual to be `d - y` with `y` affine in the profiled coefficients. A PyBNF noise family declares the space its residual lives in, and on `lognormal` / `lnnormal` that residual is `log d - log(a*s + b)`, which is affine in `log a` only when `b == 0`. Every coupled pair in the corpus -- Borghans, Elowitz, Schwen -- is lognormal. Of the eight offset-role parameters in the 23 slugs, exactly one has a closed form, and it is one the issue does not mention. The two slugs where `Phi = [s, 1]` is the right object, Weber and Brannmark, carry no offset at all. The closed form is not the obstacle, so the landscape was measured with the profile taken numerically -- the exact conditional optimum whatever the family is -- through PyBNF's own evaluate_multiple, with sigma additionally profiled by ADR-0108. That is #572's evaluation item 2 verbatim. On Borghans, over 76 box draws that integrate, the searched objective spans [-160.9, +339.7] with an interquartile range of 129.5; the profiled objective spans [-167.15, -165.98] with an interquartile range of 0.081. Forty-six of 76 draws land exactly on the no-dynamics score -165.982113, 73 of 76 within 1.0 of it, and 39 of 76 profile the scale below 1e-6 -- discarding the dynamics outright, because on a log residual a wrong-shaped trajectory fits worse than no trajectory. Along a section from the reference optimum out to a box draw the profiled objective is bit-identical at six consecutive points. The reference optimum and the nominal point are untouched, exactly as the envelope theorem says. Profiling does not sharpen this landscape; it deletes it. Elowitz reproduces it on the same mechanism. Two further findings the issue did not have. Where the construction does apply, `noise_profiling` has already collected most of it: on Weber with ADR-0108 switched on, profiling all five observable scales changes the spread of the landscape by nothing (IQR 1667 -> 1715, rank correlation 0.87), and the thirteen-orders-of-magnitude improvement visible with a searched sigma belongs to the sigma profile rather than to the scale profile. And Fiedler is not the only double-bound slug: Raia_CancerResearch2011 binds five `scaling_*` parameters as both the observable scale and a factor inside a `prediction_formula` sigma, so a placeholder-name rule would catch Fiedler and miss Raia. Thirteen of the corpus's 47 affine observable parameters are double-bound. Item 3, head-to-head fits at matched budget, was deliberately not run, and the ADR says so and why: for the offset half a fit comparison on a landscape that is constant over half the box measures the sampler's tie-breaking rather than the objective, and for the pure-scale remainder it cannot be run without first building the feature that #572 exists to gate. It is named instead as the acceptance test for that follow-up, on Smith and Brannmark -- not on Weber, where there is nothing left for it to collect. No behaviour change, no new config key. The ADR exists so the measurement is not re-litigated and so the small, genuinely sound remainder -- a pure multiplicative scale on a linear-scale family, tied across the series that share it, 19 parameters in 4 slugs -- is scoped rather than lost. The two tools that produced these numbers, and their gotchas, live with the corpus in BNGL-Models: `Grein-2026-benchmark-subset-I/tools/linear_scope.py` and `.../linear_profile.py`.
… count (#572, ADR-0123) The self-check paragraph cites the section run's re-score of the reference optimum (-248.0524) and finding 3 cites the box run's gain (1.1e-4); the two came from different simulations, and finding 3 had the section's value against the box's gain. Use the box run's own pair, -248.054331 -> -248.054446, and give the exact number of scored points behind the never-worse check (279) rather than an approximation.
…used on the math, the linear-Gaussian half is untested (#572, ADR-0123) Rewrites ADR-0123. The previous version concluded "not adopted" on two arguments that do not hold, and the numbers underneath them were fine; it was the inference that was wrong. Both corrections are recorded in the ADR rather than quietly applied. The first bad argument was "the corpus has no instances". PyBNF is general-purpose, and a 23-slug corpus of PEtab imports is evidence about Benchmark-Models-PEtab, not about the merit of a facility. The sample skews lognormal because PEtab problems do; a conf written by hand uses sos or chi_sq, which is a LINEAR scale -- the case where the construction works. Smith, the one `objective = sos` slug, carries nine profilable scales. If anything the sample under-counts the feature's domain relative to PyBNF's native idiom. The second was that the landscape measurement kills it. It does not, because of where it was taken. The Borghans/Elowitz collapse happens BECAUSE the residual is on a log scale -- a wrong-shaped trajectory has more log-spread than a flat line, so the inner solve picks a = 0 -- and log families are exactly what the mathematics already excludes. The evaluation landscape-tested the method only in the regime where it cannot be used. The coupled (scale, offset) pair on a linear-scale family, which is the case #572 is actually about, was never tested, because the corpus has no instance of it. That collapse is now demoted to a footnote about what it does show: do not offer a numeric profile as a fallback where the closed form is absent. What survives, and now leads: * Finding 1, general and mathematical: the residual's space decides what is profilable. A log family has a closed form for a homogeneous parameter (the ADR-0066 geometric-mean ratio) and for nothing else, so the coupled form must be refused there rather than approximated. * Finding 2, a design constraint on any implementation: a parameter that any noise source also reads must be refused by RESOLVED NAME. The obvious rule -- detect the noiseParameter placeholder -- catches Fiedler and misses Raia, where the collision is inside a prediction_formula sigma. Thirteen of the corpus's 47 affine observable parameters are double-bound, so this is a quarter of the population, not a corner. * Finding 3, promoted from a buried aside: the only measurement taken inside the construction's real domain points FOR the feature. Laske's Int_nuc_off, a Gaussian offset in nine observables, profiles to about 57.13 at essentially every box draw regardless of theta, compresses the spread ~20x, and moves the nominal point by 0.02 -- a textbook redundant nuisance parameter. * Finding 4, the question that is actually open: how much of this is a re-delivery of ADR-0108. On Weber with noise_profiling on, profiling all five scales changes the landscape spread by nothing (IQR 1667 -> 1715, rank corr 0.87); Brannmark disagrees, but only because noise_profiling is refused there. Two slugs at n = 20 and n = 25 disagreeing for an explicable reason is not an answer. The status becomes "narrowed, not decided", and the ADR names the two measurements that would decide it: a synthetic linear-Gaussian fixture carrying a real coupled pair -- which the corpus cannot supply and which is cheap to build -- and #572's item 3 on Smith and Brannmark, the two slugs where the scale profile is not redundant with a profiled sigma. It also records the scope the feature would have if those come back positive. Also corrects the tied-parameter count to 9 of 19, across 2 to 11 experiments. No behaviour change. #572 stays open.
wshlavacek
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records the evaluation #572 asked for. No behaviour change; one ADR. #572 stays open, narrowed.
What the evaluation settles
1. The residual's space decides what is profilable — and this is general, not about our models. Variable projection needs the residual to be
d − ywithyaffine in the coefficients. PyBNF's log families scorelog d − log(a·s + b):a·ss + ba·s + bgaussian,laplace,sos)lognormal,lnnormal)So the coupled form must be refused on a log family, not approximated. Note "homogeneous" is a property of the whole formula:
Borghans'sscaleinZ_state*scale + offsetis not a pure scale.2. Double-binding must be refused by resolved parameter name, and the obvious rule is wrong.
Fiedleris confirmed as described — butRaia_CancerResearch2011is a second instance the issue doesn't mention, with fivescaling_*colliding inside aprediction_formulaσ. Detecting thenoiseParameterplaceholder catchesFiedlerand missesRaia. 13 of the corpus's 47 affine observable parameters are double-bound — a quarter of the population, not a corner.What stays open — and why the first version was wrong to close it
The corpus-instance-count argument doesn't hold. PyBNF is general-purpose; 23 PEtab imports are evidence about Benchmark-Models-PEtab. That sample skews
lognormalbecause PEtab problems do, while a hand-written conf usessosorchi_sq— linear scale, the case where this works.Smith, our onesosslug, carries nine profilable scales. The sample arguably under-counts the domain.And the landscape measurement was taken in the wrong regime. The Borghans/Elowitz collapse happens because the residual is logarithmic — a wrong-shaped trajectory has more log-spread than a flat line, so the inner solve picks
a = 0. Log families are what point 1 already excludes. The coupled pair on a linear-scale family — the case #572 is actually about — was never tested, because the corpus has no instance. That collapse is now a footnote showing one narrow thing: don't offer a numeric profile as a fallback where the closed form is absent.The one in-domain measurement points for the feature.
Laske'sInt_nuc_off— a Gaussian offset in nine observables, exactly #572's case minus the coupling — profiles to ≈57.13at essentially every box draw regardless of θ, compresses the spread ~20×, and moves the nominal point by0.02. A textbook redundant nuisance parameter.The real open question is redundancy with ADR-0108.
Weber, 20 draws:noise_profiling = 1)With ADR-0108 on, profiling all five scales buys nothing.
Brannmarkdisagrees (IQR 1.01e17 → 3.94e12) — but only becausenoise_profilingis refused there. Two slugs at n=20 and n=25 disagreeing for an explicable reason is not an answer.What would decide it
Named in the ADR, neither needing the full feature: (1) a synthetic linear-Gaussian fixture carrying a real coupled
(scale, offset)pair — the corpus can't supply one and it's cheap to build; (2) #572's item 3 onSmithandBrannmark, the two slugs where the scale profile isn't redundant with a profiled σ — not onWeber. The ADR also records the scope the feature would have if both come back positive.Verification
Three self-checks, all passed: the independently computed flat-line reference reproduces the recorded
-165.982113at every point with spread0; the reference optimum and nominal point re-score to their recorded values; and the profiled score never exceeded the searched score at any of the 279 points measured, which it cannot.Tools in wshlavacek/BNGL-Models#47 (
linear_scope.py,linear_profile.py). They run no fits — aBorghanspoint is one simulation plus a few hundred in-memory re-scorings of it.