Skip to content

Backfill Project Sidewalk contributors from the NSF annual reports#1421

Merged
jonfroehlich merged 1 commit into
masterfrom
seed-sidewalk-participants
Jul 22, 2026
Merged

Backfill Project Sidewalk contributors from the NSF annual reports#1421
jonfroehlich merged 1 commit into
masterfrom
seed-sidewalk-participants

Conversation

@jonfroehlich

Copy link
Copy Markdown
Member

Why

Project Sidewalk now consumes /api/v1/projects/sidewalk/people/ as its single source of truth for who contributed to PS. That endpoint is driven by ProjectRole rows, so a contributor without one is invisible to Sidewalk — even if they have a member page.

Diffing the four NSF Crowd+AI annual report participant spreadsheets (award #2125087, Y1–Y4 = Oct 2021 → Sep 2025) against the live API found 115 participant rows = 74 unique people. 45 were already linked correctly; the other 29 split into two gaps:

Gap Count Who
Member page exists, but no Sidewalk ProjectRole 10 Y1 undergrads + faculty/collaborators (Heer, Hosseini, Zhihan Zhang, Kuang, Milandin, Lakshmanan, Duanhao Zhang, Alex Liu, Rapo, Buchanan)
No Person record at all 17 every one from the UIC subaward (Eisenberg / Labbé / Hammel's team)

What

Adds seed_sidewalk_participants, modelled on seed_project_aliases (#944): a baked-in seed table plus a create-only handler, wired into docker-entrypoint.sh as step 4.10d. There is no shell or DB access on the test/prod servers, so a startup one-shot is the only path for this.

Create-only is the load-bearing property, since this runs on every container start long after editors have tuned these records in /admin. It never modifies an existing Person, never adds a Position to anyone who already has one, and never touches an existing ProjectRole. A link-only entry whose Person can't be found is skipped with a warning rather than recreated, so a rename or merge can't spawn a duplicate.

Dates come from the NSF reporting periods (Oct 1 – Sep 30) — the finest granularity the spreadsheets offer, since they record which annual report someone appeared in, not the day they started. Consecutive appearances merge into a single role.

Also adds Title.PROJECT_COORDINATOR for the UIC project coordinator, whose role "Research Scientist" overstated. Worth knowing for future contributors: a new Title requires a TITLE_ORDER_MAPPING entry — admin/position_admin.py sorts the dropdown by it and would KeyError otherwise — so a test now pins every Title as mapped. No migration is committed (website/migrations/ is gitignored per-environment); the entrypoint's makemigrations/migrate generates the choices-only AlterField on deploy, leaving the column untouched.

Data judgment calls

  • Lauren Frame gets two Positions — undergrad through May 2024, then UIC's Occupational Therapy Doctorate from Aug 2024. The NSF sheets had her Y2/Y3 status backwards (Graduate in Y2, Undergraduate in Y3).
  • Xia Su and Lisa Orii are deliberately excluded — they appear in the reports but shouldn't be listed as PS contributors.
  • Headshots are out of scope. Person.save() supplies the Star Wars fallback; real headshots get uploaded via /admin, which is what sets the Cropper.js crop box (Would be so nice to get an instant preview of an image that you add for cropping #1299). 16 are still being collected.

Testing

python manage.py test website --settings=makeabilitylab.settings_test620 passed, 8 skipped. 12 new tests cover idempotency, that admin edits survive a re-run, the held-back sentinel for unconfirmed titles, the mid-grant title change, the new Title, and that seeded people surface through the real API route.

Dry run against a local DB:

seed_sidewalk_participants (dry run): 17 people, 18 positions, 18 project roles created; 0 held back pending a title.

18 positions because Lauren has two. On prod expect 17 people, 18 positions, 27 roles — the extra 9 roles are the link-only people the local dev DB doesn't have.

No UI changes, so no Pa11y run or screenshots (data-only).

Follow-ups (not in this PR)

  • 4 people whose existing ProjectRole dates don't cover the years they were reported — Michael Duan, Devon Snyder, Dylan Bunarto, Manjiri Dehankar. /admin edits; they already appear in the API, only the dates are wrong.
  • Headshot uploads for the 16 outstanding UIC folks.

🤖 Generated with Claude Code

Project Sidewalk now consumes /api/v1/projects/sidewalk/people/ as its
single source of truth for who contributed to PS. That endpoint is driven
by ProjectRole rows, so a contributor without one is invisible to Sidewalk
even if they have a member page.

Diffing the four NSF Crowd+AI annual report participant spreadsheets
(award #2125087, Y1-Y4 = Oct 2021 - Sep 2025) against the live API found
115 participant rows = 74 unique people, of which 45 were already linked.
The other 29 split into two gaps:

  - 10 had a member page but no Sidewalk ProjectRole (Y1 undergrads plus
    the faculty/collaborators)
  - 17 had no Person record at all -- every one from the UIC subaward

Adds seed_sidewalk_participants, modelled on seed_project_aliases (#944):
a baked-in seed table plus a create-only handler wired into
docker-entrypoint.sh as step 4.10d. There's no shell or DB access on the
test/prod servers, so a startup one-shot is the only path.

Create-only is the important property, since this runs on every container
start long after editors have tuned these records in /admin: it never
modifies an existing Person, never adds a Position to anyone who has one,
and never touches an existing ProjectRole. A link-only entry whose Person
can't be found is skipped with a warning rather than recreated, so a rename
or merge can't spawn a duplicate.

Dates come from the NSF reporting periods (Oct 1 - Sep 30), the finest
granularity the spreadsheets offer; consecutive appearances merge into one
role. Headshots are out of scope -- Person.save() supplies the Star Wars
fallback and real ones get uploaded via /admin, which is what sets the
Cropper.js crop box.

Also adds Title.PROJECT_COORDINATOR for the UIC project coordinator, whose
role "Research Scientist" overstated. Note that a new Title REQUIRES a
TITLE_ORDER_MAPPING entry -- admin/position_admin.py sorts the dropdown by
it and would KeyError otherwise -- so a test now pins every Title as mapped.

Two data corrections the spreadsheets got wrong: Lauren Frame is modelled
with two Positions (undergrad through May 2024, then UIC's Occupational
Therapy Doctorate from Aug 2024; the sheets had her Y2/Y3 status
backwards), and Xia Su and Lisa Orii are deliberately excluded despite
appearing in the reports.

Tests cover idempotency, that admin edits survive a re-run, the held-back
sentinel for unconfirmed titles, the mid-grant title change, and that
seeded people surface through the real API route.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jonfroehlich
jonfroehlich merged commit 5219dcb into master Jul 22, 2026
3 checks passed
@jonfroehlich
jonfroehlich deleted the seed-sidewalk-participants branch July 22, 2026 18:57
jonfroehlich added a commit that referenced this pull request Jul 22, 2026
Ships the seed_sidewalk_participants backfill plus the Pillow 12.3.0 bump.

Note: this release has NOT been verified on makeabilitylab-test -- the
test host's auto-deploy has been rebuilding from a source tree stuck at
578d643 since Jul 18, so nothing merged since then has landed there. The
backfill was instead validated against live prod data (all 27 seed
entries checked for existence/duplicates) plus the full test suite.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant