fix(scraper): follow the MiSTer artwork pack format

docs/scraper.md

@@ -186,11 +186,11 @@ Game manuals can be selected through Update All's **Game Manuals (EN)** settings
 
 ### Discovery
 
-Core derives `docs` roots from MiSTer's configured SD, USB, network/CIFS, and custom index roots. It recognizes content by installed format rather than repository name:
+Core derives `docs` roots from MiSTer's configured SD, USB, network/CIFS, and custom index roots, and also probes `/media/usb6` and `/media/usb7`, which artwork packs may be installed to but MiSTer's games-folder list does not reach. It recognizes content by installed format rather than repository name, following the [MiSTer Artwork Pack format](https://github.com/chipster6502/MiSTer_artwork_pack/blob/main/PACK_FORMAT.md):
 
-- Artwork: `docs/<system>/Artwork/index.tsv` plus image files in the same directory.
-- Optional title metadata: `gameinfo.tsv` beside the artwork index.
-- Optional description: `synopsis_en.tsv` beside the artwork index.
+- Artwork: `docs/<System>/Artwork/` holding one `<key>.jpg` per game, normally with an `index.tsv` beside them that maps every known dump to its key. `<System>` is the MiSTer `games/` folder name. A directory with images and no index still resolves games filed under their exact key.
+- Optional title metadata: `gameinfo.tsv` beside the images. Games it lists without an image still receive their metadata.
+- Optional description: `synopsis_<lang>.tsv` beside the images. Which languages a pack ships varies per system, so Core reads whichever files exist and picks the first match from `media.default_langs`, then English, then the first available language.
 - Manuals: direct PDF files in a child directory whose name contains `manual`, for example `docs/SNES/Manuals/` or `docs/NES/Famicom Disk System Manuals/`.
 
 This format-based discovery means future compatible databases need no Core update. Run `mister-docs` again after Downloader installs or updates content. Normal runs rescan installed records idempotently; force runs additionally delete stale box-art/manual properties whose old paths are proven to belong to a discovered MiSTer docs convention.
@@ -199,7 +199,14 @@ Metadata files are treated as untrusted input. Core bounds their size and record
 
 ### Matching And Fields
 
-`index.tsv` maps ROM or MRA basenames to canonical artwork keys. Core prefers an exact media basename match. When no exact media match exists, a unique title-slug match may receive title-level artwork; ambiguous matches are skipped. CRC and size columns are not used because hashing every installed ROM would impose substantial MiSTer I/O.
+`index.tsv` maps catalogued dump names to artwork keys: No-Intro names for cartridges, Redump names for CD systems, and MAME parent setnames for arcade. Core resolves each pack entry to installed media in the pack format's order, stopping at the first hit:
+
+1. The catalogued name as a media basename, at media scope.
+2. For arcade, the `<setname>` inside each installed `.mra`, at media scope. MRA filenames are titles, so the setname is the only handle an arcade key has; Core reads it from the MRA only for systems that have an arcade artwork source.
+3. A media basename whose trailing parenthesised tag is itself a pack key, such as `Shock Troopers (set 1) (shocktro)`, at media scope.
+4. A unique bare-title match, at title scope. This step is skipped when the stripped title is not unique among the pack's keys or among the library's titles, and it is only available to index rows: images the index does not mention resolve by exact name alone.
+
+CRC and size columns are not used because hashing every installed ROM would impose substantial MiSTer I/O; the pack format treats that step as optional.
 
 | Source | Destination |
 |---|---|
@@ -208,10 +215,12 @@ Metadata files are treated as untrusted input. Core bounds their size and record
 | `gameinfo.tsv` genre | title tag `genre` |
 | `gameinfo.tsv` developer | title tag `developer` |
 | `gameinfo.tsv` players | title tag `players` using highest numeric value |
-| `synopsis_en.tsv` synopsis | title property `property:description` |
+| `synopsis_<lang>.tsv` synopsis | title property `property:description` |
 | Manual PDF | title property `property:manual` |
 
-Manual filenames are matched with the same game-title slug normalization used by MediaDB, including leading/trailing article handling. Basenames with no matching title, or whose slug collision remains ambiguous after normalized-name matching, are left unmatched. Category-like names such as system manuals, overlays, or charts are not filtered separately. Base-system sources can enrich compatible fallback systems such as SNES MSU and Genesis variants.
+Manual filenames are matched with the same game-title slug normalization used by MediaDB, including leading/trailing article handling. Basenames with no matching title, or whose slug collision remains ambiguous after normalized-name matching, are left unmatched. Category-like names such as system manuals, overlays, or charts are not filtered separately.
+
+Base-system sources enrich their variants, such as SNES MSU-1, Genesis MSU, and the granular arcade systems. On top of that, Core applies the pack format's shared-catalogue rules: Game Boy and Game Boy Color each fall back to the other, Super Game Boy reads both, and FDS falls back to NES but never the reverse. Systems the pack catalogues separately do not fill each other's gaps, so SG-1000 never receives ColecoVision art and Neo Geo Pocket Color never receives Neo Geo Pocket art, even though the general system fallbacks allow it.
 
 If multiple docs roots provide the same property, MiSTer root order decides which source wins. As with other scrapers, running a different scraper later may replace exclusive tags or same-type properties.
 

pkg/database/mediadb/sql_property_plan_test.go

@@ -0,0 +1,97 @@
+// Zaparoo Core
+// Copyright (c) 2026 The Zaparoo Project Contributors.
+// SPDX-License-Identifier: GPL-3.0-or-later
+//
+// This file is part of Zaparoo Core.
+//
+// Zaparoo Core is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// Zaparoo Core is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with Zaparoo Core.  If not, see <http://www.gnu.org/licenses/>.
+
+package mediadb
+
+import (
+	"context"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// TestPropertyLookupsDriveFromPropertyTable pins the join order of the bulk
+// property lookups. Scrapers fill the property tables after indexing has run
+// ANALYZE, so the planner usually has no statistics for them; without a fixed
+// order it drove a many-ID lookup from TagTypes outward and probed the
+// property index once per tag per requested ID, which took minutes on a
+// MiSTer for a few thousand IDs.
+func TestPropertyLookupsDriveFromPropertyTable(t *testing.T) {
+	t.Parallel()
+	mediaDB, cleanup := setupTempMediaDB(t)
+	defer cleanup()
+	ctx := context.Background()
+
+	const ids = 2000
+	args := make([]any, ids)
+	for i := range args {
+		args[i] = int64(i + 1)
+	}
+	inList := prepareVariadic("?", ",", ids)
+
+	tests := []struct {
+		name      string
+		query     string
+		wantFirst string
+	}{
+		{
+			name:      "media properties",
+			query:     mediaPropertyQuery("WHERE mp.MediaDBID IN ("+inList+")", propertyGroupInclude),
+			wantFirst: "SEARCH mp USING INDEX mediaproperties_media_idx (MediaDBID=?)",
+		},
+		{
+			name:      "media property metadata",
+			query:     mediaPropertyMetadataQuery("WHERE mp.MediaDBID IN ("+inList+")", propertyGroupInclude),
+			wantFirst: "SEARCH mp USING INDEX mediaproperties_media_idx (MediaDBID=?)",
+		},
+		{
+			name:      "title properties",
+			query:     mediaTitlePropertyQuery("WHERE mtp.MediaTitleDBID IN ("+inList+")", propertyGroupInclude),
+			wantFirst: "SEARCH mtp USING INDEX mediatitleproperties_title_idx (MediaTitleDBID=?)",
+		},
+		{
+			name: "title property metadata",
+			query: mediaTitlePropertyMetadataQuery(
+				"WHERE mtp.MediaTitleDBID IN ("+inList+")", propertyGroupInclude,
+			),
+			wantFirst: "SEARCH mtp USING INDEX mediatitleproperties_title_idx (MediaTitleDBID=?)",
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			rows, err := mediaDB.sql.Load().QueryContext(ctx, "EXPLAIN QUERY PLAN "+tt.query, args...)
+			require.NoError(t, err)
+			defer func() { _ = rows.Close() }()
+			var plan []string
+			for rows.Next() {
+				var id, parent, notUsed int
+				var detail string
+				require.NoError(t, rows.Scan(&id, &parent, &notUsed, &detail))
+				plan = append(plan, detail)
+			}
+			require.NoError(t, rows.Err())
+			require.NotEmpty(t, plan)
+			assert.Equal(t, tt.wantFirst, plan[0], "plan: %v", plan)
+			for _, step := range plan {
+				assert.NotContains(t, step, "SCAN tt", "plan drives from TagTypes: %v", plan)
+			}
+		})
+	}
+}

pkg/database/mediadb/sql_scraper.go

@@ -2609,7 +2609,9 @@ func (db *MediaDB) loadMediaTitlePropertiesByMediaTitleDBIDs(
 	args := int64Args(mediaTitleDBIDs)
 	where := `WHERE mtp.MediaTitleDBID IN (` + prepareVariadic("?", ",", len(mediaTitleDBIDs)) + `)`
 	//nolint:gosec // Safe: prepareVariadic only generates SQL placeholders like "?, ?, ?".
-	rows, err := db.sql.Load().QueryContext(ctx, mediaTitlePropertyQuery(where, propertyGroupInclude), args...)
+	rows, err := db.sql.Load().QueryContext(
+		ctx, mediaTitlePropertyQuery(where, propertyGroupInclude), args...,
+	)
 	if err != nil {
 		return nil, fmt.Errorf("failed to query GetMediaTitlePropertiesByMediaTitleDBIDs: %w", err)
 	}
@@ -2667,7 +2669,9 @@ func (db *MediaDB) GetMediaTitlePropertyMetadataByMediaTitleDBIDs(
 	args := int64Args(mediaTitleDBIDs)
 	where := `WHERE mtp.MediaTitleDBID IN (` + prepareVariadic("?", ",", len(mediaTitleDBIDs)) + `)`
 	//nolint:gosec // Safe: prepareVariadic only generates SQL placeholders like "?, ?, ?".
-	rows, err := db.sql.Load().QueryContext(ctx, mediaTitlePropertyMetadataQuery(where, propertyGroupInclude), args...)
+	rows, err := db.sql.Load().QueryContext(
+		ctx, mediaTitlePropertyMetadataQuery(where, propertyGroupInclude), args...,
+	)
 	if err != nil {
 		return nil, fmt.Errorf("failed to query GetMediaTitlePropertyMetadataByMediaTitleDBIDs: %w", err)
 	}
@@ -2737,7 +2741,9 @@ func (db *MediaDB) loadMediaPropertiesByMediaDBIDs(
 	args := int64Args(mediaDBIDs)
 	where := `WHERE mp.MediaDBID IN (` + prepareVariadic("?", ",", len(mediaDBIDs)) + `)`
 	//nolint:gosec // Safe: prepareVariadic only generates SQL placeholders like "?, ?, ?".
-	rows, err := db.sql.Load().QueryContext(ctx, mediaPropertyQuery(where, propertyGroupInclude), args...)
+	rows, err := db.sql.Load().QueryContext(
+		ctx, mediaPropertyQuery(where, propertyGroupInclude), args...,
+	)
 	if err != nil {
 		return nil, fmt.Errorf("failed to query GetMediaPropertiesByMediaDBIDs: %w", err)
 	}
@@ -2792,7 +2798,9 @@ func (db *MediaDB) GetMediaPropertyMetadataByMediaDBIDs(
 	args := int64Args(mediaDBIDs)
 	where := `WHERE mp.MediaDBID IN (` + prepareVariadic("?", ",", len(mediaDBIDs)) + `)`
 	//nolint:gosec // Safe: prepareVariadic only generates SQL placeholders like "?, ?, ?".
-	rows, err := db.sql.Load().QueryContext(ctx, mediaPropertyMetadataQuery(where, propertyGroupInclude), args...)
+	rows, err := db.sql.Load().QueryContext(
+		ctx, mediaPropertyMetadataQuery(where, propertyGroupInclude), args...,
+	)
 	if err != nil {
 		return nil, fmt.Errorf("failed to query GetMediaPropertyMetadataByMediaDBIDs: %w", err)
 	}
@@ -3148,12 +3156,19 @@ func propertyMetadataSelectColumns(entityIDColumn string, groupMode propertyGrou
 	return strings.Join(parts, ", ")
 }
 
+// The property query builders below join with CROSS JOIN, which SQLite
+// treats as an inner join whose nesting order is fixed left to right. Without
+// it, and without fresh statistics on the property tables, the planner drives
+// a lookup of many IDs from TagTypes outward and probes the property index
+// once per tag per requested ID - a few thousand IDs then take minutes on a
+// MiSTer. Starting from the property table keeps every lookup at one index
+// search per requested ID whatever the statistics say.
 func mediaTitlePropertyQuery(where string, groupMode propertyGroupMode) string {
 	return `
 		SELECT ` + propertySelectColumns("mtp.MediaTitleDBID", groupMode) + `
 		FROM MediaTitleProperties mtp
-		JOIN Tags t      ON mtp.TypeTagDBID = t.DBID
-		JOIN TagTypes tt ON t.TypeDBID = tt.DBID
+		CROSS JOIN Tags t      ON mtp.TypeTagDBID = t.DBID
+		CROSS JOIN TagTypes tt ON t.TypeDBID = tt.DBID
 		LEFT JOIN MediaBlobs mb ON mtp.BlobDBID = mb.DBID
 		` + where
 }
@@ -3162,8 +3177,8 @@ func mediaTitlePropertyMetadataQuery(where string, groupMode propertyGroupMode)
 	return `
 		SELECT ` + propertyMetadataSelectColumns("mtp.MediaTitleDBID", groupMode) + `
 		FROM MediaTitleProperties mtp
-		JOIN Tags t      ON mtp.TypeTagDBID = t.DBID
-		JOIN TagTypes tt ON t.TypeDBID = tt.DBID
+		CROSS JOIN Tags t      ON mtp.TypeTagDBID = t.DBID
+		CROSS JOIN TagTypes tt ON t.TypeDBID = tt.DBID
 		LEFT JOIN MediaBlobs mb ON mtp.BlobDBID = mb.DBID
 		` + where
 }
@@ -3172,8 +3187,8 @@ func mediaPropertyQuery(where string, groupMode propertyGroupMode) string {
 	return `
 		SELECT ` + propertySelectColumns("mp.MediaDBID", groupMode) + `
 		FROM MediaProperties mp
-		JOIN Tags t      ON mp.TypeTagDBID = t.DBID
-		JOIN TagTypes tt ON t.TypeDBID = tt.DBID
+		CROSS JOIN Tags t      ON mp.TypeTagDBID = t.DBID
+		CROSS JOIN TagTypes tt ON t.TypeDBID = tt.DBID
 		LEFT JOIN MediaBlobs mb ON mp.BlobDBID = mb.DBID
 		` + where
 }
@@ -3182,8 +3197,8 @@ func mediaPropertyMetadataQuery(where string, groupMode propertyGroupMode) strin
 	return `
 		SELECT ` + propertyMetadataSelectColumns("mp.MediaDBID", groupMode) + `
 		FROM MediaProperties mp
-		JOIN Tags t      ON mp.TypeTagDBID = t.DBID
-		JOIN TagTypes tt ON t.TypeDBID = tt.DBID
+		CROSS JOIN Tags t      ON mp.TypeTagDBID = t.DBID
+		CROSS JOIN TagTypes tt ON t.TypeDBID = tt.DBID
 		LEFT JOIN MediaBlobs mb ON mp.BlobDBID = mb.DBID
 		` + where
 }

pkg/database/scraper/misterdocs/discovery.go

@@ -69,9 +69,21 @@ func candidateDocsRoots(roots []string) []string {
 		}
 		appendRoot(filepath.Join(root, "docs"))
 	}
+	for _, root := range extraDocsRoots {
+		appendRoot(root)
+	}
 	return result
 }
 
+// extraDocsRoots are mount points artwork packs tell consumers to probe that
+// MiSTer's own games-folder list does not reach. Packs install with the
+// Downloader's "pext" path, so docs can land on any USB drive. Roots that do
+// not exist are skipped during discovery, so probing costs one stat each.
+var extraDocsRoots = []string{ //nolint:gochecknoglobals // Fixed MiSTer mount points.
+	"/media/usb6/docs",
+	"/media/usb7/docs",
+}
+
 func discoverSources(fs afero.Fs, roots []string) ([]sourceDir, error) {
 	var result []sourceDir
 	seen := make(map[string]struct{})
@@ -100,8 +112,7 @@ func discoverSources(fs afero.Fs, roots []string) ([]sourceDir, error) {
 				var kind sourceKind
 				var systemID string
 				switch {
-				case strings.EqualFold(child.Name(), artworkDirName) &&
-					isRegularFile(fs, filepath.Join(path, indexFileName)):
+				case strings.EqualFold(child.Name(), artworkDirName) && hasArtworkContent(fs, path):
 					kind = sourceArtwork
 					systemID = resolveSourceSystem(systemEntry.Name(), "")
 				case strings.Contains(strings.ToLower(child.Name()), "manual"):
@@ -172,6 +183,33 @@ func pathWithin(path, root string) bool {
 	return err == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator))
 }
 
+// hasArtworkContent reports whether an Artwork directory holds anything worth
+// loading. The index resolves every dump not filed under its own key, but a
+// pack shipped without one still serves exact-key images, so a directory with
+// images and no index is a valid source rather than a directory to ignore.
+func hasArtworkContent(fs afero.Fs, dir string) bool {
+	if isRegularFile(fs, filepath.Join(dir, indexFileName)) {
+		return true
+	}
+	directory, err := fs.Open(dir)
+	if err != nil {
+		return false
+	}
+	defer func() { _ = directory.Close() }()
+	for {
+		entries, readErr := directory.Readdir(directoryReadBatch)
+		for _, entry := range entries {
+			if !entry.IsDir() && entry.Mode()&os.ModeSymlink == 0 &&
+				supportedImageExt(filepath.Ext(entry.Name())) {
+				return true
+			}
+		}
+		if readErr != nil {
+			return false
+		}
+	}
+}
+
 func isRegularDir(fs afero.Fs, path string) bool {
 	info, err := lstat(fs, path)
 	return err == nil && info.IsDir() && info.Mode()&os.ModeSymlink == 0
@@ -205,6 +243,29 @@ func sourcesBySystem(sources []sourceDir) map[string][]sourceDir {
 	return result
 }
 
+// artworkSiblings adds the shared-catalogue fallbacks the artwork pack format
+// specifies and the general system definitions do not model. ScreenScraper
+// splits dual-mode cartridges between the Game Boy and Game Boy Color
+// catalogues on its own criteria, so each has to try the other, and Super Game
+// Boy ships no pack of its own. Famicom Disk System is asymmetric on purpose:
+// a disk release may borrow the cartridge box, but a cartridge must never
+// receive the disk release's.
+var artworkSiblings = map[string][]string{ //nolint:gochecknoglobals // Fixed artwork pack rules.
+	systemdefs.SystemGameboy:      {systemdefs.SystemGameboyColor},
+	systemdefs.SystemGameboyColor: {systemdefs.SystemGameboy},
+	systemdefs.SystemSuperGameboy: {systemdefs.SystemGameboy, systemdefs.SystemGameboyColor},
+	systemdefs.SystemFDS:          {systemdefs.SystemNES},
+}
+
+// artworkFallbackBlocks drops general system fallbacks between systems the
+// artwork pack catalogues separately. Filling an SG-1000 gap with a
+// ColecoVision box serves art for a different release, and the pack's own rule
+// is that an absent image beats a wrong one.
+var artworkFallbackBlocks = map[string]map[string]struct{}{ //nolint:gochecknoglobals // Fixed artwork pack rules.
+	systemdefs.SystemSG1000:            {systemdefs.SystemColecoVision: {}},
+	systemdefs.SystemNeoGeoPocketColor: {systemdefs.SystemNeoGeoPocket: {}},
+}
+
 func sourceIDsForTarget(targetID string) []string {
 	seen := make(map[string]struct{})
 	var result []string
@@ -215,12 +276,17 @@ func sourceIDsForTarget(targetID string) []string {
 		}
 		seen[id] = struct{}{}
 		result = append(result, id)
-		sys, err := systemdefs.GetSystem(id)
-		if err != nil {
-			return
+		blocked := artworkFallbackBlocks[id]
+		if sys, err := systemdefs.GetSystem(id); err == nil {
+			for _, fallback := range sys.Fallbacks {
+				if _, ok := blocked[fallback]; ok {
+					continue
+				}
+				visit(fallback)
+			}
 		}
-		for _, fallback := range sys.Fallbacks {
-			visit(fallback)
+		for _, sibling := range artworkSiblings[id] {
+			visit(sibling)
 		}
 	}
 	visit(targetID)