Promote development → staging: media→Mongo, IODA recharts, Connections redesign, access control, alert/incident filtering - #146
Open
ronniegross wants to merge 52 commits into
Open
Promote development → staging: media→Mongo, IODA recharts, Connections redesign, access control, alert/incident filtering#146ronniegross wants to merge 52 commits into
ronniegross wants to merge 52 commits into
Conversation
Fold the not-yet-implemented steps from seven completed/superseded plan docs into docs/claude/plans/todo.md (with file refs and context), then remove those docs. Keep the still-outstanding plans and the media-storage architecture reference. Kept: - architecture/media-image-storage.md (evergreen reference) - plans/fix-assign-user-in-operator-crash.md (unfixed live bug) - plans/cloudflare-chart-caching.md (design-only, not built) Removed (leftover steps moved to todo.md): - move-ioda-svg-to-media-storage, fix-reports-timeout-double-send, report-dedup-timeout-fix, table-views, table-toolbar-relayout, incidents-list-scroll-retention, ioda-v2-chart-investigation
…g on alerts and incidents
A — Alerts date filter fix
- Cast before/after bounds to Date in report-query so the deduped path's
aggregate $match total agrees with find() (aggregation doesn't cast),
fixing phantom/empty pages. Guard invalid dates.
- FilterDateTime gains an optional label prop; alerts label it "Outage start".
Remove stray console.log.
B — Incident start-time filter
- parseQueryData maps before/after to incidentStartedAt (cast to Date) instead
of storedAt; add incidentStartedAt to Group.filterAttributes so queryGroups
copies it into the Mongo filter.
- Add a FilterDateTime control ("Incident start") to IncidentsFilters.
C — Source badges on incident cards
- addReportSourcesToGroups aggregates distinct _media per group and attaches
reportSources[]; chained after coverage enrichment in group_groups.
- Add reportSources to Group type; render per-source badges via SocialMediaIcon
on incident cards.
- FilterDropdown: add persistLabel so the trigger keeps its label (and fixed size) when a value is set instead of swapping to the long range string — stops the filter bar from wrapping/resizing. - FilterDateTime: pin the dropdown panel to fit-content so it no longer grows when dates are picked; label the control clearly. - DateSelector: fixed-width date buttons; disable future dates; add min/max bounds so the "before" date can't precede the "after" date (and vice-versa), preventing empty invalid ranges.
When picking the "after" or "before" date, open the calendar on the other field's month (via defaultMonth) so the user lands near their target instead of today's month. Falls back to the field's own value, then today.
Repurpose the redundant alerts "Source" column (which just re-printed the platform name already shown by the Platform icon) into an "ASN / Network / Geo Scope" column for IODA/Cloudflare outage alerts. The ASN, network name, and geo scope are all derived from data already on each report (report.asn + rawAPIResponse.entityName/entityScope), so no backend change is needed and it works on existing rows. Also top-align the DataTable expand caret (items-start) so it lines up with the first row of text now that outage rows are multi-line.
Shrink the incidents table to text-xs so it matches the alerts table: pass tableClassName=text-xs (it was falling back to the DataTable default text-sm), drop two oversized hardcoded sizes (the "alerts" suffix and the "N reports" subtitle), and set the expanded detail panel to text-xs (scoped to incidents so the alerts ReportDetail is unaffected). Also center the DataTable expand caret on a one-line-tall box (items-center h-4) instead of pinning it to the top of a 30px box, so the chevron lines up with the first row of text in both the alerts and incidents tables.
Add per-user display preferences (12h/24h clock, MM/DD/YYYY vs DD/MM/YYYY date order, local vs UTC timezone) surfaced in the account profile. - User model gains a preferences sub-object with enum defaults (24h/DMY/local); user_update whitelists it and the session endpoint returns it app-wide. - Centralize date/time formatting into utils/dateFormat.ts (Intl-based) plus a useFormatters hook bound to the current session's preferences. - Convert all date/time display sites (incidents, reports, IODA/Cloudflare event windows, filters, WebAuthn rows, comments) to honor the preference, removing the scattered formatIsoTime/formatStamp/toLocale* helpers. - Add a self-service Display preferences section that saves via a dedicated updateUserPreferences call and refetches the session so dates update live.
Frees space on an over-quota cluster by deleting reports that are both orphaned (no group) and older than 30 days, keeping all grouped reports and the last month of everything. Dry-run by default; --apply to delete.
Add ExpandableChart so IODA/Cloudflare outage charts in the alerts table expanded row can be clicked to enlarge to full width and collapsed again (click chart or button). Deduplicates the chart-rendering JSX previously copied in IodaEvent and TrafficEvent. Also surface the ASN in the alert detail author header (ASN / network / geo scope) by moving reportNetwork into the shared reportParser.
Replace the incidents list Open/Closed/All radio with a multi-select "Stage" dropdown (Verification / Confirmation / Published) plus an "Include closed" toggle, matching the alerts filter pattern. - backend: add `stages` to Group.filterAttributes and translate it into strict-pipeline predicates in Group.queryGroups, OR-combined and pushed as an $and clause so it doesn't clobber the assignedTo $or. Existing closed handling is unchanged. - frontend: add `stages` to GroupQueryState; swap the radio for a FilterListbox multi-select with an "Include closed" toggle (Entity Level pattern); clear the stage filter on title search so search spans all stages.
- FilterDropdown: remove hardcoded w-[20em] that collided with each caller's width class (unpredictable panel widths); size panels with w-max min-w-[150px] so they fit the widest row without wrapping. - FilterListBox: nowrap option rows; cap the toggle description width so long helper text wraps instead of stretching the panel. - FilterDateTime: drop the now-unneeded !w-fit override. - Sort By dropdown: show human-readable labels (GROUP_SORTBY_LABELS) instead of camelCase keys, mapping label<->key on change. - Incidents filters: group the Stage dropdown inline with the other filter dropdowns.
The side-by-side Compare feature previously only worked in the table view of the Alerts and Incidents pages. Turn it on in list view too, reusing the existing page-level compare state, cap, and modals. Alerts (AllReportsList): - Show the Compare button in list view (drop the table-only guard). - Extract toggleReportForCompare (cap + chart prefetch) shared by the table and list rows. - Clicking a list row checkbox now enters compare mode and selects the row, while keeping the mark relevant/irrelevant bar available on the same selection (relevance bar renders in list view even in compare mode; table view stays pure compare). - "Select all on this page" leaves compare mode, since it's an uncapped relevance action incompatible with the 6-item compare cap. Incidents (index + IncidentListItem): - Show the Compare button in list view. - Add selection support to IncidentListItem via the shared MultiSelectListItem wrapper (checkbox gutter + select-mode row-click routing). - Clicking a list row checkbox enters compare mode and selects the incident. CompareActionBar and the compare modals are unchanged — they already render off page-level compareMode for both views.
…toolbar Table view (alerts + incidents): - Add an always-visible checkbox column to the shared DataTable via a new selection.alwaysShow flag; idle checkboxes reveal on row hover. - Clicking a table checkbox launches compare the same way the list does (enters compare mode, capped at 6, prefetch charts for alerts). On alerts it also surfaces the mark relevant/irrelevant bar on the same selection; the relevance select mode still does a plain toggle. Compare control: - Replace the floating CompareActionBar with an inline CompareToolbar rendered next to the view toggle: idle shows a Compare button, active turns blue (like the view toggle) and becomes the launch + clear + cancel bar, separated from the view toggle by a divider. Bug fix: - On the alerts table, 'Select all on this page' followed by unchecking it left the page stuck in an empty select mode, so a subsequent single check no longer launched compare. Unchecking select-all now returns fully to idle.
- Add rerender-frequency-findings.md: writeup of why the Alerts/Incidents pages re-render frequently (react-time-ago tickers, socket-listener churn, isFetching/StrictMode pulses, whole-list cache replacement) with ranked fixes. - Update incidents-alerts-filtering.md and todo.md planning notes.
…ection Two related fixes for the Alerts and Incidents list/table views. ResizeObserver overlay crash: - Switching list -> table with rows selected triggered "ResizeObserver loop completed with undelivered notifications", which CRA's dev error overlay surfaced as a full-screen crash. Both pages measured their sticky filters bar with a ResizeObserver whose callback set state synchronously; clearing the selection shrank the bar in the same frame, retriggering layout mid- delivery. - Extract a shared useMeasuredHeight hook that defers the measurement to requestAnimationFrame and skips no-op updates, breaking the loop. Reuse it in AllReportsList and incidents/index, replacing the duplicated inline observers. Persist selection across list/table: - Selecting rows then toggling the view previously cleared the selection. The underlying results are identical across both views, so keep the selection instead. - Remove the [view] reset effect on both pages and re-key the search-params reset effect on the non-view params, so a view toggle no longer clears the selection (or refetches, on incidents) while real filter/search/page changes still do.
Add planning doc for building a dev Aggie environment seeded with real production data at zero risk to prod. Approach: each VM runs its own local MongoDB, and a nightly cron on the dev VM overwrites dev's database with a fresh mongodump/mongorestore snapshot of prod. Covers the rationale (why not a shared DB, dedicated DB VM, or per-record origin field), the refresh script design (dump-first ordering, PM2 stop/restore/start, single-instance lock), two connection options (SSH tunnel vs direct TLS + IP allowlist), the config required for a full "copy everything" dump (ENCRYPTION_KEY parity, MFA/WebAuthn caveats), dev-fetching implications, and a verification checklist.
Drop the per-poll headless Chromium (Playwright) that scraped IODA's dashboard SVG. Fetch IODA's raw signal series instead, store them inline on the report (metadata.rawAPIResponse.chart), and render them client-side with recharts. Backend: - fetchSignals() replaces the SVG scrape; removes playwright + dompurify (jsdom kept for Mastodon); drops the now-dead persistSvgChart. - Signals use the raw event.location entity so geoasn works (the dashboard-stripped form is rejected by the signals API). - List serializer strips the chart series from rows; detail keeps it and the frontend lazy-loads per report. Frontend: - New IodaChart (recharts): per-signal normalized lines (BGP, Active Probing, Telescope, Mozilla), outage-window shading, and a zoomable navigator with a responsive full-range 12-hour time axis. - New useReportChartSeries hook; IodaEvent and CompareCardBody branch to IodaChart, keeping the legacy image path as a fallback. Also adds dev-only fixture/backfill scripts (gitignored scratchpad) and updates the plan docs; IODA is now out of scope for the media-bytes migration.
The signal-JSON/recharts migration removed persistSvgChart, but the one-off backfill scripts/migrate-ioda-svg-to-storage.js still needs it to run in production. Reinstate the helper (live channel no longer calls it), correct the list-projection comment in report.js, and update the media/deployment docs to describe the current model and the pending production backfill.
Maps every MongoDB collection, model, and notably-renamed field to its user-facing UI label (Group=Incident, one reports collection split into Alerts vs Social Media Posts, SMTCTag=Tags, etc.). Verified against the live aggie DB. Includes a detailed explanation of the _sources String-vs-ObjectId caveat.
One-time, idempotent script that converts legacy IODA chart SVGs (metadata.rawAPIResponse.image) into the compact signal JSON shape (metadata.rawAPIResponse.chart) by re-fetching from IODA's signals API. Converted disk SVGs are moved to an out-of-media backup tree; charts with no usable signals keep the SVG fallback (inline SVGs promoted to disk) for the later media-bytes->Mongo migration. Supports --dry-run.
- Rework Step 2 backfillMediaToMongo.js spec to migrate only report-referenced keys, skip and report orphans, and count dangling refs — avoids ingesting the ~322 MB of orphaned IODA chart SVGs into the mediaassets collection. - Gitignore public/ioda-charts-backup (rollback copies from the SVG→JSON backfill).
Image bytes for Mastodon/Telegram social attachments and residual IODA chart SVGs now live in a new 'mediaassets' collection (inline BinData) instead of public/media/, so media travels with the database. The key namespace and the unauthenticated GET /media/<key> URL contract are unchanged, so reports, controllers, channels, hooks, and the frontend are untouched. - backend/models/mediaAsset.js: new model / mediaassets collection (key unique, data Buffer, contentType, byteSize, kind indexed, sourcePlatform, timestamps). - socialImageStorage.js: disk -> Mongo, same public API. Thumbnails are built in-memory with sharp (replaces the macOS-only sips shell-out, which silently produced full-size copies on the Ubuntu prod VM). persistSocialImage inserts social-full + social-thumb docs; persistSvgChart upserts an ioda-chart doc; deleteMediaByKey removes the doc; normalizeKey is now exported. getMediaRoot/MEDIA_ROOT retained for the backfill. - api.js: serve GET /media/* from Mongo via one streaming route (replacing both express.static mounts), with per-kind cache/ETag and a Binary->Buffer coercion for .lean() results. - backfillMediaToMongo.js: one-time, idempotent, reference-aware backfill that migrates only report-referenced keys and skips/reports orphans. - Add sharp dependency; update architecture + plan/runbook docs.
Document how to obtain external credentials and register Mastodon, Telegram (User and Bot), and Cloudflare sources in Aggie, with per-source troubleshooting.
The FETCH process writes social images and IODA SVG charts under MEDIA_ROOT, which defaulted to public/media. CRA's dev server watches public/ and does a full browser reload on every write, so background fetch cycles reloaded the app on a loop (full page reload, no 401) on every page while idle. Default MEDIA_ROOT to media-store/ (outside public/) in dev, keeping public/media only in production, which has no such watcher. /media is still served by the backend via getMediaRoot(), so images resolve unchanged with no frontend or env changes. Gitignore media-store; document the optional override in .env.example.
Fold the six shipped plan docs (lifecycle-stage filter, ASN/Network column, compare-in-list-view, date/time preferences, and the alerts reload/re-render work) into incidents-alerts-filtering.md as a consolidated branch record, and delete the standalone files. Update todo.md with a branch-record pointer, an alerts/incidents re-render perf backlog, and a flag for the live assign-user crash. Leave cloudflare-chart-caching.md and fix-assign-user-in-operator-crash.md standalone since they did not ship.
Check in scripts/backfill/ (reports outage fields, entity level, event keys, incidents import, IODA chart signals). Promote the IODA chart backfill out of scratchpad/ into scripts/backfill/ and fix its require paths. Gitignore media-store/, public/ioda-charts-backup/, and scratchpad/ — runtime media, chart backups, and dev scratch.
…rovider - New /settings/connections page (ConnectionsIndex + ApiTypeSection) that groups credentials and sources by API type, each with add-credential/add-source actions - Extract reusable SourcesSection, CredentialsSection, and SourceDetailsView from the standalone pages - Responsive Settings sidebar (fixed 300px at >=1080, wraps below) and teal AggieButton variant + aggie.secondary.650 token - Source form: defaultType prop to pre-scope by type, access-policy fields, multi-hashtag Mastodon input; auto-filled credential name - Backend: credential-name fallback in credential_create; Mastodon multi-hashtag fetch; media storage/env tweaks
…ined details Frontend - Feed row: title is a plain bold heading (no longer a link); add "View details" to the row's action menu. - Details popup: "View details" now shows a combined read-only info + editable form; warnings pill shows a summary toast (react-hot-toast) instead of opening the popup. - Add-feed form is locked to its provider (no provider dropdown) and the provider is emphasized in the dialog title; removed the inline "Add connection" button - connections are configured beforehand. - Cap connections at one per provider via ALLOW_MULTIPLE_CONNECTIONS_PER_PROVIDER (flag kept for re-enabling). Backend - Compute distinctErrorCount for sources (controller + model), exposed on the source list payload. Docs - Replace older Connections plan docs with connections-page.md.
…icher details view - Replace react-hot-toast warnings peek with a Warnings dialog and a Recent activity table in the details view; drop react-hot-toast - Warning badge now counts recent events (last 50) so it matches the popup list, instead of distinct-message dedup (backend + types) - Add a runtime "Allow multiple connections per provider" toggle, replacing the compile-time flag; hide/auto-select the connection picker when off - Expand SourceDetailsView into a read-only view with per-provider config, access policy, and allowed teams - FormikInput hint prop; block submit until dirty; dialog/spacing polish
Recover from a full MongoDB (Atlas free-tier) storage quota by keeping the newest ~N MB of reports and deleting the rest. Walks reports newest-first by _id, measures real BSON size via $bsonSize, and deletes everything older than the cutoff. Deletes run before the group back-ref cleanup so recovery works even while writes are quota-blocked. Defaults to a 350MB budget (KEEP_BYTES-overridable) and is dry-run unless --yes.
- Rename page to "Providers and Feeds"; rewrite intro to define provider → connection → feed; "Reports" → "Alerts" copy; capitalize Provider; rework Mastodon hashtag helper text; drop em dashes - Feed rows: lead with mode/country code, feed name beneath; keep the connection key chip - View details: "View <name> details" header (green name, sized like Add feed); static (non-link) connection; collapsible Recent activity with inline warnings pill; respect single-connection setting when editing - Conditional "Hashtag"/"Hashtags" label by tag count - Smaller "Enable Fetching" heading - Remove dev-only Style link from the navbar - Consistent light-grey "more" menu buttons on the feeds list and details
Folds the 5 scripts/backfill/*.js tools onto media-migration. feat/backfill-scripts carried a merge of development, so this also refreshes media-migration onto development. Conflict resolution (package.json / package-lock.json / .gitignore): - Kept development's structure: cross-env in devDependencies + .npmrc (include=dev) prod-build fix; dompurify and playwright retained. - Kept media-migration's sharp dependency; regenerated lockfile (--package-lock-only) so sharp's subtree is consistent. - .gitignore: union of both ignore sets (media-store, public/ioda-charts-backup, scratchpad/).
The canonical IODA backfill is scripts/backfill/backfill-ioda-charts.js (re-fetches signal series from the IODA API). Add a header banner pointing there and reserve the SVG->JSON converter for legacy reports the API re-fetch can't reconstruct.
Brings the landed api-redesign work (Settings/Connections redesign, mastodon multi-hashtag, distinctErrorCount, trim-reports script) onto media-migration. Conflict resolution: - socialImageStorage.js: keep media-migration's Mongo storage. MEDIA_ROOT stays the legacy disk path (public/media) used only by backfillMediaToMongo.js; api-redesign's dev 'media-store' reload-loop workaround is unnecessary here because fetch no longer writes media to disk. - .env.example: MEDIA_ROOT comment updated to reflect Mongo storage. - package.json: auto-merged (sharp dependency + trim-reports script both kept).
Media migration
One-off, idempotent repair that re-fetches signal series for IODA reports rendering blank — those whose metadata.rawAPIResponse.chart is null/empty-series with no legacy image fallback (a transient IODA signals-API error at fetch time stored chart=null with no retry). Distinct from backfill-ioda-charts.js, which repairs legacy reports that still carry a scraped SVG image key and no chart field. Supports --dry-run.
Add backfill-ioda-null-charts.js: repair blank IODA charts
Refresh the branch onto development ahead of its PR (step 3 of the branch-test-merge-order plan). development had since landed media-migration (media->Mongo + recharts IODA), the ongoing-outage status filter, and api-redesign, which the branch predated. Conflict resolution: - Media storage: keep development's Mongo path (socialImageStorage.js, .env.example); drop the branch's stale on-disk writer. - report-query.js: keep isOutageOngoing filter AND the branch's Date-cast pagination fix for before/after. - reports/types.ts: keep chart? (IodaChartData) + add entityName/entityScope. - IODA/Cloudflare charts: recharts IodaChart stays the primary render and is now click-to-expand; ExpandableChart handles the legacy image/SVG fallback (with loading skeleton). Shared expand/collapse chrome extracted into ChartExpander (image = surface click; recharts = corner button only). - Dates: adopt the branch's display-preferences formatDateTime while restoring development's ongoing -> "Present" (SocialMediaListItem, IncidentInfo). - CompareCardBody: keep recharts + existing zoom button; formatStamp->formatDateTime.
The three backfill-reports-*.js scripts require()d "../database" / "../models/report" / "../fetching/utils/iodaUtils", which resolve relative to the script's own dir (scripts/backfill/), not the cwd — so they threw "Cannot find module '../database'" from every working directory. Repoint them at "../../backend/…" (matching backfill-incidents.js) so they run from the repo root. Also add a --dry-run flag to each (they previously wrote immediately with no preview): reports counts / would-modify totals without issuing updateMany / bulkWrite.
The dry-run tally previously printed "Matched: N / Modified: 0", identical to a real run that changed nothing — impossible to tell apart without the [DRY-RUN] banner. In dry-run these now say "would write (prepared updates): N" and "would update=N (no writes made)" instead of Matched/Modified. Real-run output is unchanged.
Feat/incident alert filtering
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes everything accumulated on
developmentsince it was last 1:1 withstaging.Scope: 52 commits · 112 files · +8,107 / −2,410, bundling five efforts that landed as separate PRs into
development.What's changing
mediaAssetmodel stores image/chart bytes in Mongo;socialImageStorage.jsrewritten (thumbnails viasharp, replacing the macOS-onlysips). Media served at/media/<key>.IodaChart+useReportChartSeries), rendering signal JSON. Legacy image/SVG reports still render via a fallback.scripts/backfill/*andbackend/scripts/*; inert until run.sourceController/credentialsController/source.js.isOutageOngoing+ alerts Status filter (All/Ongoing/Ended); "Present" for outages with no end.report-query.js(corrects phantom pagination pages when a date range is set).New dependency:
sharp@^0.35.3→ runnpm ci(recharts was already present).New env keys:
API_REQUEST_TIMEOUT,API_FETCH_INTERVAL, optionalMEDIA_ROOT(leave unset — media is in Mongo).Migrations
Deploy code. Then, from the repo root, staging
.env→ staging DB,--dry-runfirst (all idempotent):