Post-launch external discovery, review, and design-partner follow-ups

[seva9523]

Objective

Track adoption and third-party outcomes that continue after the launch sprint in #14. These items depend on external services, maintainers, or real users and must not be reported as completed before public evidence exists.

Contributor discovery

• Retry the Good First Issue submission only after the authentication mismatch reported in goodfirstissueorg Discussion #5 is publicly resolved.
  • Public bug report and privacy-cropped screenshot are present. No successful listing or verifiable resolution is claimed.
• Monitor Up For Grabs PR #6057 until upstream review and merge.
  • Status checked 2026-08-15: open, mergeable, and not a draft. The validator found the required label and two matching issues, and the Netlify preview is ready. No human review or merge yet.
• Submit EvalRepro to GitOpen only when the service is available and the signed-in submission flow works; record the resulting public listing.
  • Status checked 2026-08-15: the host still returns 502 Bad Gateway. No submission or listing is claimed.
• Evaluate CodeTriage accurately as a general open-issue channel rather than a beginner-only listing.
  • Evaluation completed in Launch sprint: package release, Marketplace discovery, contributor listings, and case-study outreach #14. It remains lower priority than curated beginner listings; no CodeTriage submission is claimed.
• Complete the initial review of optional discovery-source responses collected through the pull request template added in PR #28.
  • Status checked 2026-08-15: no open pull requests and no post-template referral response to record. Future responses will be logged when contributors provide them.

Upstream review

• Obtain technical human review or merge for Inspect Evals draft PR #2113.
  • Status checked 2026-08-15: open, mergeable draft. Automated findings were addressed; a maintainer acknowledged the queue, but no human technical review or merge has occurred.
• Complete the 2026-08-15 case-study status review.
  • No qualifying public upstream evidence changed, so the result remains EvalRepro workflow-validated and not upstream-reviewed. Recheck only when PR #2113 receives substantive human review, is merged, or is closed.

Design partners and adoption

• Invite three relevant public evaluation workflows through Beta: seeking three public evaluation workflows for design-partner testing #11 using narrow, evidence-led requests; do not request stars.
  • LM Evaluation Harness presets/MMLU pilot
  • Lighteval TriviaQA reproducibility pilot
  • LMMs-Eval MMMU local-test pilot
  • Status checked 2026-08-15: no maintainer response on any invitation. These remain invitations only; no participation, adoption, review, or endorsement is claimed.
• Record each participating public workflow, independently submitted issue/discussion, and accepted external contribution.
  • Current participating design partners: 1.
  • Awesome Agentic Engineering v0.15.0 → v0.16.0 public-file contract was independently submitted in #11 and accepted on 2026-08-17. Immutable revisions, ancestry, all sixteen blob hashes, and the expected three-workflow drift were independently reproduced. The implementation was merged in PR #31 as commit d843b4f after a public source-maintainer scope check; both the standard CI run and pinned public-source reproduction passed. The source comment is recorded as scope validation, not formal code approval, adoption, or endorsement.
• Record the day-zero baseline for the 60–90 day success review.
• Conduct the 60-day review on or after 2026-10-14 and the 90-day review on or after 2026-11-13 using public evidence only.

Baseline snapshot — 2026-08-15

Primary metric from #14	Baseline	Target
Independently verified public downstream workflows/repositories	0	3
Accepted external contributors	1	3
Upstream technical maintainer reviews/references	0	1
Versioned public releases	2	3
Independently participating design-partner cases or independently submitted issues/discussions	0	3

Additional evidence and boundaries:

• PyPI Trusted Publishing and a project-controlled GitHub Action consumer smoke run are verified in Launch sprint: package release, Marketplace discovery, contributor listings, and case-study outreach #14; independent usage evidence is not yet claimed.
• GitHub Marketplace listing is public.
• Up For Grabs listing: 1 open upstream PR, 0 merged listings.
• Good First Issue listing: blocked by reported authentication mismatch.
• GitOpen listing: blocked by service availability.
• Three public design-partner invitations were sent; those invitation threads still have no maintainer response.
• One separate, independently submitted public design-partner case is now participating and tracked in Design partner: verify Awesome Agentic Engineering v0.15.0 → v0.16.0 contract #30.
• Secondary metrics such as genuine stars/forks, article engagement, and private repository traffic are not asserted here because no verified snapshot was captured through the available public evidence.

Progress update — 2026-08-17

• Independently participating public design-partner cases: 1 of 3.
• The first case covers eight public release artifacts and locally reproduced semantic_drift in exactly three expected generated workflows.
• The case-study implementation was merged in PR #31 as commit d843b4f; standard CI and the pinned public-source reproduction passed on 2026-08-17. The completed verification record is in Design partner: verify Awesome Agentic Engineering v0.15.0 → v0.16.0 contract #30.
• Public source-maintainer scope validations for participating design-partner cases: 1. This is separate from the Inspect Evals upstream technical-review metric and is not counted as adoption or endorsement.
• Up For Grabs PR #6057 remains open and validator-ready with no human review.
• GitOpen still returns 502 Bad Gateway.
• The three outbound design-partner invitation threads and Harvey LAB issue #143 have no maintainer response.
• Inspect Evals PR #2113 remains an open, mergeable draft awaiting the maintainer's stated review window.

Evidence update — 2026-08-19

• MTEB #5228 received substantive technical clarification from maintainers and the original reporter.
• The Hagrid and Wikipedia scope assumptions were withdrawn after the source-loading paths were clarified.
• Samoed's multi-version rerun produced stable TwitterHjerne scores across MTEB 2.19.5, 1.39.7, and 1.12.70.
• The reporter explained the local 0.7257 result: direct SentenceTransformer loading omitted the registry query instruction; mteb.get_model() restored the prompt and produced 0.7846, matching the stable rerun.
• EvalRepro Usage validation: public reproducibility clinic candidate pipeline #33 is therefore closed as a candidate lane and retained only for observation. No cross-version/evaluator drift is claimed; the lower historical result files remain an upstream provenance/cleanup matter.
• No EvalRepro artifact or further MTEB outreach is needed, and this clarification is not counted as design-partner participation, adoption, artifact review, or endorsement. Baseline adoption counts remain unchanged.

Implementation update — 2026-08-20

• Reproducibility Clinic PR #32 passed CI, was marked ready, and was squash-merged as commit 6a9621c. The low-friction public request form and Clinic documentation are now live on main.
• MTEB's latest maintainer conclusion is to spot-check the historical result files from results PR release: prepare v0.1.0a1 trusted publishing #15 and rerun problematic submissions. EvalRepro remains observe-only on this case; no artifact or further outreach is needed.
• Promptfoo PR #10336 remains conflict-free with green CI and is awaiting human review after the 2026-08-18 follow-up. Inspect Evals PR #2113 remains an open, mergeable draft inside the maintainer's stated review window. Up For Grabs PR #6057 remains validator-ready and awaiting human review. No repetitive reminders were posted.

Follow-up policy

Update this issue only when public evidence changes. Do not post repetitive reminders to external maintainers while their stated review window is active. Preserve #6 and #38 as the two bounded, mentorable good first issue tasks while contributor-listing reviews are pending.

Integrity

No bought, reciprocal, automated, or friend-network activity. No fabricated users, issues, testimonials, downloads, downstream repositories, or adoption claims.

Progress update — 2026-08-29

• PR #35 supplied the copy-paste terminal-demo commands requested in Docs: add copy-paste commands for the terminal demo #26 and was squash-merged as commit 5610d43 after maintainer review and a local reproduction of the documented commands.
• Accepted external contributors: 2 of 3. This counts merged public contributions only; open PRs and comments are not counted.
• Issues Docs: add copy-paste commands for the terminal demo #26 and Docs: add Windows development setup commands #36 are complete. Issue #38 is now the second bounded, unpaid good first issue, so contributor discovery still exposes two genuine starter tasks: Tests: add Windows path and local-content normalisation fixtures #6 and Docs: add development setup troubleshooting #38.
• PR #34 is an external contribution for Tests: add Windows path and local-content normalisation fixtures #6. Head 011353c passed strict mypy and all 84 tests with 95.60% coverage. On 2026-09-03, head f7339a2 removed the final comment but introduced two mechanical formatting failures: trailing spaces in normalise.py and Ruff E501 on a 110-character test signature. The contributor was asked to run ruff format .; the PR is not counted as accepted until clean checks and merge.
• Up For Grabs PR #6057 therefore still has two qualifying open good first issue tasks to reference. No upstream merge is claimed.
• Inspect Evals follow-up was posted on 2026-08-29 after the maintainer's stated review window passed. PR #2113 remains an open, mergeable draft; no new maintainer response, technical review, or merge is claimed.
• PR #37 completed the cross-platform development setup documentation and was squash-merged as commit 9cb70bd after maintainer review and a clean diff check. Accepted external contributors are now 3 of 3, meeting the public target; this counts merged contributions only.

[seva9523]

Usage-validation intake update — 2026-08-17

• Draft PR #32 adds a low-friction Reproducibility check request and publishes the Reproducibility Clinic scope.
• Issue #33 records an initial 20-candidate public-problem scan, five pre-analysis lanes, conservative adoption metrics, and a no-spam outreach policy.
• The strongest new candidate is MTEB #5228, which supplies exact dataset revisions and separates evaluator-version drift from dataset-contract drift.
• External commenting was not available to the connected GitHub installation for MTEB, so no outreach or participation is claimed. The bounded pre-analysis and unsent draft are recorded in Usage validation: public reproducibility clinic candidate pipeline #33.

Baseline adoption counts remain unchanged until independent public evidence changes.