feat(media): scrape indexed items and directory subtrees

[wizzomafizzo]

Add optional media ID, exact-file, and system-bound subtree scopes to media.scrape, preserving existing systems requests.

Narrow selection across all scrapers, persist exact scope for restart recovery, and keep force cleanup and progress scoped. Add regression coverage, path fuzzing, and API documentation.

Closes #1326

Summary by CodeRabbit

• New Features

  • Added scoped media scraping by indexed media ID, exact file or URI, or directory subtree.
  • Scoped scrapes now restrict selection, progress, metadata updates, force cleanup, and matching to the chosen target.
  • Scrape scopes persist across interruptions and are restored when operations resume.
  • Added validation and canonicalization for scoped paths, including safeguards against invalid or unsafe paths.

• Documentation

  • Documented scope parameters, validation, progress reporting, recovery, cancellation, and library-wide scrape counts.

• Tests

  • Added coverage for scoped scraping, path handling, recovery, cancellation, force mode, and fuzz testing.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 2121274f-4227-440a-a6b7-e26954737115

📥 Commits

Reviewing files that changed from the base of the PR and between 5aefeb6 and ecdf07b.

📒 Files selected for processing (26)

• Taskfile.dist.yml
• docs/api/methods.md
• docs/api/notifications.md
• docs/scraper.md
• pkg/api/methods/media_scrape.go
• pkg/api/methods/media_scrape_scope.go
• pkg/api/methods/media_scrape_scope_test.go
• pkg/api/methods/media_scrape_test.go
• pkg/api/models/params.go
• pkg/api/models/scrape_scope.go
• pkg/database/database.go
• pkg/database/mediadb/sql_scrape_scope.go
• pkg/database/mediadb/sql_scrape_scope_test.go
• pkg/database/scrape_scope.go
• pkg/database/scrape_scope_test.go
• pkg/database/scraper/gamelistxml/scope.go
• pkg/database/scraper/gamelistxml/scope_test.go
• pkg/database/scraper/gamelistxml/scraper.go
• pkg/database/scraper/localmedia/scope_test.go
• pkg/database/scraper/localmedia/scraper.go
• pkg/database/scraper/misterdocs/scope_test.go
• pkg/database/scraper/misterdocs/scraper.go
• pkg/database/scraper/scope.go
• pkg/database/scraper/scope_test.go
• pkg/database/scraper/scraper.go
• pkg/testing/helpers/db_mocks.go

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

---

📝 Walkthrough

Walkthrough

Changes

Scoped media.scrape requests can target one indexed media item, one exact file or URI, or a directory subtree. The resolved scope is canonicalized, persisted for resume, applied to database queries, and passed through scraper implementations with scoped progress and write validation.

Scoped API contract and resolution

Layer / File(s)	Summary
Scope contract and canonicalization pkg/api/models/*, pkg/database/scrape_scope.go, pkg/api/models/params.go	Adds structured scope selectors, strict JSON decoding, canonical path validation, and MediaScrapeParams.Scope.
API scope resolution pkg/api/methods/media_scrape_scope.go, pkg/api/methods/*_test.go	Resolves media IDs, files, URIs, and subtrees while rejecting invalid, missing, or conflicting selectors.
API documentation and fuzz coverage docs/api/methods.md, Taskfile.dist.yml	Documents scope behavior and adds FuzzCanonicalScrapePath to the fuzz task.

Database selection

Layer / File(s)	Summary
Scoped persistence and queries pkg/database/database.go, pkg/database/scrape_scope.go, pkg/database/mediadb/sql_scrape_scope.go	Persists operation scope and adds exact-scope media and completion-marker queries.
Database validation tests and mocks pkg/database/mediadb/sql_scrape_scope_test.go, pkg/testing/helpers/db_mocks.go	Tests path boundaries, URI handling, identity guards, stored scopes, and run-marker filtering.

Operation scope lifecycle

Layer / File(s)	Summary
Start and resume wiring pkg/api/methods/media_scrape.go, pkg/api/methods/media_scrape_test.go	Resolves or validates scope under the indexing lease, persists it, overrides systems with the scoped system, and restores it during resume.

Shared scraper execution

Layer / File(s)	Summary
Scoped selection and target application pkg/database/scraper/scraper.go, pkg/database/scraper/scope.go, pkg/database/scraper/scope_test.go	Loads scoped media, filters completed items, resolves containers, validates target boundaries, applies results, and reports progress, cancellation, and completion.

Scraper integrations and documentation

Layer / File(s)	Summary
GamelistXML scoped matching pkg/database/scraper/gamelistxml/*	Adds scoped gamelist processing and disables unrestricted path, slug, container, and filename fallbacks in scoped mode.
Local media and MiSTer docs integration pkg/database/scraper/localmedia/*, pkg/database/scraper/misterdocs/*	Uses scoped selection, completion markers, scoped containers, and scoped target application in both scrapers.
Scraper documentation and notifications docs/scraper.md, docs/api/notifications.md	Documents scoped execution and clarifies that totalScraped remains library-wide.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: ⚪ Minimal · up to ecdf0

Scoped scraping preserves validated selection, scoped persistence and resume behavior, and scoped scraper writes without an identified merge-blocking regression.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant media.scrape
  participant MediaDB
  participant Scraper
  participant MediaWriter
  Client->>media.scrape: submit scope
  media.scrape->>MediaDB: resolve and validate scope
  media.scrape->>MediaDB: persist ScrapingOperation
  media.scrape->>Scraper: pass ScrapeOptions.Scope
  Scraper->>MediaDB: load scoped selection
  Scraper->>MediaWriter: apply scoped targets
  MediaWriter-->>Client: emit scoped progress and completion

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.81% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 31 functions across 22 files. (4 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the primary change: support for scraping indexed media items and directory subtrees.
Linked Issues check	✅ Passed	The changes satisfy the linked issue objectives. They add validated media ID, file, and subtree scopes; preserve system selection; persist scope for resume; narrow database and scraper work; preserve …
Out of Scope Changes check	✅ Passed	The changes remain within scope. The implementation, database updates, scraper changes, documentation, fuzzing, mocks, and tests directly support scoped media scraping and compatibility requirements.

Full details: Docstring Coverage

Explanation

Docstring coverage is 25.81% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 31 functions across 22 files. (4 skipped: 4 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 67.43119% with 142 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/database/scraper/scope.go	32.97%	63 Missing ⚠️
pkg/database/scraper/gamelistxml/scope.go	79.79%	20 Missing ⚠️
pkg/database/scraper/localmedia/scraper.go	57.14%	15 Missing ⚠️
pkg/database/mediadb/sql_scrape_scope.go	85.29%	10 Missing ⚠️
pkg/database/scrape_scope.go	56.52%	10 Missing ⚠️
pkg/api/methods/media_scrape_scope.go	83.63%	9 Missing ⚠️
pkg/api/models/scrape_scope.go	0.00%	8 Missing ⚠️
pkg/database/scraper/misterdocs/scraper.go	79.16%	5 Missing ⚠️
pkg/database/scraper/gamelistxml/scraper.go	83.33%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!