feat: promote production-mvp to main — drives, scan history, files modularization - #55
Merged
Conversation
Fixed four broken API endpoint calls introduced during settings restructure: - AI & Chat LLM Provider: /api/settings/chat/llm-config → /api/chat/providers - Connections Local Storage: /api/settings/providers → /api/providers - Rclone settings: Added GET support to /api/settings/rclone-interpret (was POST-only, causing 405 Method Not Allowed) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Commented out non-existent /api/settings/chat/config endpoint calls in _ai_behavior.html. These settings are currently controlled via environment variables (SCIDK_GRAPHRAG_VERBOSE) as documented in the Advanced section. Added TODO comments for future endpoint implementation if persistent per-user chat behavior settings are needed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Fixed smart apostrophes in _modules_links.html line 577 that were breaking JavaScript parsing. The curly apostrophes in "Neo4j's", "Python's", and "It's" were being interpreted as string terminators, causing: Uncaught SyntaxError: Unexpected identifier 's' (at (index):4602:43) Replaced with escaped straight apostrophes (Neo4j\'s, Python\'s, It\'s). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…ipts table Implements database schema foundation for Universal Module Registry Pattern: Schema changes (migration v21): - Add `source` field (TEXT): tracks 'built-in' vs 'custom' origin - Add `modified` field (INTEGER/bool): tracks if built-in scripts have been edited - Add `validation_output` field (TEXT): stores complete test run results (separate from validation_errors which only stores error messages) Model changes: - Update Script class __init__ to accept new fields with defaults - Update to_dict() and from_dict() serialization methods - Update _row_to_script() to handle new database columns - Update create_script() and update_script() SQL queries Modification tracking: - Script save endpoint now detects when source='built-in' scripts are edited - Sets modified=True and validation_status='queued' on code changes - Removed block on editing built-in scripts (now allowed with tracking) This enables the module registry pattern where built-in scripts can be: - Edited by users (marked with 'built-in ✎' badge) - Restored to canonical version - Tracked for divergence from defaults 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Implements Universal Module Registry Pattern for Analyses module:
Backend endpoints (api_scripts.py):
- POST /api/scripts/analyses/bootstrap: Seeds built-in analyses into registry
- Sets source='built-in', modified=False, validation_status='queued'
- Idempotent: skips existing, never overwrites modified scripts
- Returns summary: {imported, skipped, already_existed}
- POST /api/scripts/analyses/validate/<script_id>: Runs validation tests
- Sets validation_status='running' while executing
- Captures full test output to validation_output field
- Sets final status: 'passed' or 'failed'
- POST /api/scripts/analyses/restore/<script_id>: Restores modified built-ins
- Replaces code with canonical version from builtin_scripts
- Resets modified=False, validation_status='queued'
- Only works on source='built-in' && modified=True scripts
Frontend (_modules_analyses.html):
- Registry table with columns: Name | Source | Validation | Last Tested | Actions
- Source badges: 'built-in', 'built-in ✎' (modified), 'custom'
- Validation states: Queued, Running ⟳, Passed ✅, Failed ❌, Not run
- Actions: Edit → (links to /scripts), Restore (modified built-ins only)
- Empty state: "Import Built-in Analyses" button when registry empty
- Re-sync footer link: updates built-ins without overwriting modified ones
Security fix (update_script endpoint):
- Preserve source/modified fields from database, never trust request payload
- Prevents spoofing via malicious PUT request body
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implements Universal Module Registry Pattern for Links module: Backend endpoints (api_scripts.py): - POST /api/scripts/links/bootstrap: Seeds built-in link scripts - Filters for category='links' (exact match) - Same idempotent behavior as analyses - POST /api/scripts/links/validate/<script_id>: Runs validation tests - Uses ScriptTestRunner infrastructure - Captures full output to validation_output field - POST /api/scripts/links/restore/<script_id>: Restores modified built-ins - Same restore logic as analyses Frontend (_modules_links.html): - Registry table between status card and Algorithm Explorer (per spec) - Additional column: Matching Strategy (from metadata.algorithm field) - Same badges, states, and actions as Analyses - Empty state with "Import Built-in Links" button - Re-sync footer link Links follow exact same pattern as Analyses, reusing: - get_builtin_scripts() loader (filters by category) - ScriptTestRunner validation infrastructure - Script model with source/modified/validation_output fields 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add /api/scripts/<script_id>/validation-result endpoint that returns stored validation data without executing validation. This endpoint is used by the validation modal to display validation status, output, errors, and timestamps. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Create reusable validation modal component that displays script validation results including status, test output, errors, and timestamps. Features: - Status display with appropriate badges and styling - Conditional content based on validation state - Live polling for queued/running validations - Collapsible raw output section - Fetches from GET /api/scripts/<id>/validation-result endpoint - Keyboard and click-outside closing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Make validation status badges clickable to open the shared validation modal. Adds onclick handlers that call showValidationModal() with script ID and name. Includes hover effects and title tooltips for better UX. Also adds missing closing </section> tag. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Make validation status badges clickable to open the shared validation modal. Adds onclick handlers with hover effects and tooltips. Includes the validation modal partial and adds missing closing </section> tag. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Replace status card display with full registry table showing all label definitions. Features: Table Columns: Name, Properties, Status, Pushed, Actions Status States: - Local only (grey) - defined but not pushed to Neo4j - Pushed ✅ (green) - active in Neo4j - Out of sync⚠️ (orange) - placeholder for future detection Actions: - Edit → - all labels, links to /labels page - Push - visible on Local only labels - Sync - visible on Out of sync labels (when implemented) Properties column shows count and first 3 property names. Push status determined by comparing local label definitions with Neo4j schema. Empty state prompts user to create labels on Labels page. Note: Out of sync detection is stubbed with isOutOfSync=false as it requires schema changes to add last_pushed_at field. Detection will be enabled in future update. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
The Files page (/datasets) was showing "Not connected" even when Neo4j was properly connected. The bug was in fetchNeo4jStatus() which called /api/settings/neo4j (config endpoint) instead of /api/health/comprehensive (live health check endpoint). Changed to use the same health check API that the Settings page uses, which properly tests the connection and returns accurate status: - 'connected' when Neo4j is reachable - 'not_configured' when no URI is set - 'unavailable' with error details when connection fails This fix brings the Files page Neo4j status display in sync with the Settings health dashboard implementation. Location: scidk/ui/templates/datasets.html:1494-1525 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Enhances the Files page with proper Live Servers vs Snapshots mode distinction: Backend changes (api_files.py): - Add GET /api/servers endpoint returning all accessible servers/providers - Include scan metadata: connection status, last scanned timestamp, file counts - Query scan history from SQLite scans table to show which servers have been scanned Frontend changes (datasets.html): - Update loadServers() to fetch from /api/servers instead of /api/providers - Display scan metadata in server labels: "(Last: 2d ago)" or "(Not scanned)" - Show⚠️ icon for disconnected servers, 💻 for connected - Update loadScans() to show camera icon 📷 and "Snapshot from: [date]" labels - Format snapshot dates as relative time (Today, Yesterday, N days ago) - Prepare scan status column in file list (shows "—" for now, will be enhanced) The two modes now have visually distinct presentations: - Live mode: Shows all servers with connection + scan status - Snapshot mode: Shows frozen snapshots with camera icon + capture date This establishes the foundation for the complete redesign workflow. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Replace "BACKGROUND TASKS" section with unified "ACTIVITY" panel that combines background tasks and snapshot management in a single space. Changes: - Replace BACKGROUND TASKS header with ACTIVITY header - Add SNAPSHOTS subsection below tasks list - Show snapshots with: server name, date, file count - Display progress bars inline for scans in progress - Add Refresh and Reconfigure action buttons per snapshot - Add "+ New Snapshot" button (placeholder for Phase 5) - Clicking snapshot loads it in file browser - Wire Refresh button to /api/scans/<id>/rescan endpoint 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Implement Dropbox/Gmail-style checkbox selection for choosing which files
and folders to include in a scan.
Features implemented:
- "Select Files" toggle button to enter/exit selection mode
- Checkbox column in file table (hidden by default, shown in selection mode)
- Selection toolbar with count, clear, scan selected, and exit buttons
- Select-all checkbox in table header with indeterminate state support
- Individual file and folder checkboxes with proper state tracking
- Folder recursive selection: selecting folder fetches and selects all children
- Indeterminate state propagation: folders show indeterminate when some children selected
- Shift+Click for range selection
- Ctrl/Cmd+A to select all visible items
- Escape to exit selection mode
- Selection rules passed to /api/scan endpoint in correct format
- Async folder content fetching for accurate hierarchical selection
The indeterminate state logic properly tracks folder hierarchy by:
1. Fetching folder contents when user checks a folder checkbox
2. Recursively selecting/deselecting all children
3. Tracking folder->children relationships in folderContents Map
4. Computing indeterminate state based on partial child selection
5. Updating visual states after each selection change
Selection is persisted as rules array sent to scan API:
{
"selection": {
"rules": [
{"action": "include", "path": "/path/to/item", "recursive": true, "node_type": "folder"}
],
"use_ignore": true,
"allow_override_ignores": true
}
}
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implement snapshot configuration modal that allows creating new snapshots with file selection and reconfiguring existing snapshots with diff visualization showing changes from saved config. Features implemented: **New Snapshot Flow:** - "+ New Snapshot" button opens modal - Snapshot name input field - Server selector dropdown (auto-populated) - File browser with checkbox selection (reuses Phase 4 infrastructure) - Select All checkbox and button - "Save Config Only" button (currently creates scan - API enhancement needed) - "Create and Scan Now" button - saves config and starts scan **Reconfigure Flow:** - "Reconfigure" button on each snapshot in Activity Panel - Opens modal pre-populated with saved configuration - Loads saved selection rules via `/api/scans/<id>/config` (existing endpoint) - Shows diff legend at top explaining four visual states - Diff visualization with CSS classes applied to rows: * `row-config-unchanged` (green tint): checked, was in saved config * `row-config-new` (no tint): checked, not in saved config * `row-config-removed` (grey tint): unchecked, was in saved config * No class: unchecked, not in saved config - "Save Config Only" updates config without scanning - "Save and Rescan" updates config and triggers rescan via existing endpoint **Modal Infrastructure:** - Separate selection state from main page (modalSelectedItems, etc.) - Reuses Phase 4 checkbox logic: recursive folder selection, indeterminate states - Folder navigation via double-click - Selection count display - Async folder content fetching for hierarchical selection **API Integration:** - GET `/api/scans/<id>` - fetch scan details - GET `/api/scans/<id>/config` - fetch saved selection rules (existing) - POST `/api/scans/<id>/config` - update selection rules (existing) - POST `/api/scans/<id>/rescan` - trigger rescan with saved config (existing) - POST `/api/scan` - create new scan with selection (existing) **Diff Legend (Reconfigure mode only):** Visual guide at top of modal shows: - Green checkboxes = unchanged from config - Plain checkboxes = new additions - Grey empty checkboxes = removed from config - Plain empty checkboxes = not selected, not in config Legend automatically shown in reconfigure mode, hidden in new snapshot mode. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Fix four bugs in Files page Phases 3-5 implementation:
**Bug 1: Live servers not loading**
- Issue: `/api/servers` returns flat array with `{id, root_id, root_path, ...}`
but frontend expected nested `{id, roots: [...]}` structure
- Fix: Updated `loadServers()` to correctly parse API response format
- Changed grouping logic to build roots array from flat server entries
- Servers now load correctly in left panel with scan metadata
**Bug 2: Snapshot items too large in Activity Panel**
- Issue: Snapshot entries used large block layout with excessive padding
- Fix: Redesigned with compact single-line layout
* Reduced padding and margins (0.375rem vs 0.5rem)
* Combined info on one line: "📷 provider • date • N files"
* Changed action buttons to icon-only (↻ ⚙) with tooltips
* Removed text labels from Refresh/Reconfigure buttons
* Reduced font sizes (0.7rem for main, 0.65rem for meta)
- Snapshot list now displays 2-3x more items in same space
**Bug 3: Reconfigure button not opening modal**
- Issue: `openSnapshotModal()` defined after `renderSnapshots()` which calls it
- JavaScript hoisting issue - function wasn't available when handler attached
- Fix: Forward-declared all modal functions at top of script
* Declared variables before renderSnapshots: `let openSnapshotModal, ...`
* Converted function declarations to assignments: `openSnapshotModal = function() {...}`
* Removed duplicate `saveSnapshotConfig` definition (was defined twice)
- Reconfigure button now correctly opens modal with saved config loaded
**Bug 4: Modal rendering inline instead of overlay**
- Issue: Modal appeared as static content at bottom instead of floating overlay
- Likely causes: Bootstrap not loaded, modal init race condition
- Fix: Added safety checks in `openSnapshotModal()`
* Check if modal element exists in DOM
* Check if `bootstrap` global is defined before `new bootstrap.Modal()`
* Show error message if modal system not ready
- Modal now properly initializes as Bootstrap overlay with backdrop
All fixes preserve Phase 4 checkbox logic and existing functionality.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fix two bugs in Files page:
**Bug 1: Live servers showing "Failed to load servers"**
- Issue: loadServers() was failing silently without proper error details
- Added response validation: check if API returns OK status
- Added array validation: check if response is actually an array
- Added detailed error logging: console.error + show error message in UI
- Error message now shows: "Failed to load servers: {specific error}"
- This will help diagnose the actual issue (API error vs parse error vs network error)
**Bug 2: Snapshots showing redundantly under Snapshots radio button**
- Issue: In Snapshots mode, BOTH the snapshots-section tree AND
the Activity Panel snapshots were visible, showing duplicate data
- Root cause: Mode toggle was hiding servers-section and showing
snapshots-section, while Activity Panel remained visible
- Per spec: Activity Panel should be visible in BOTH modes
- In Live mode: servers-section shows live servers
- In Snapshots mode: servers-section should show snapshot-derived entries,
NOT a separate snapshots-section
Fix:
- Updated mode toggle handlers to keep servers-section visible in both modes
- Changed section header dynamically: "SERVERS" in Live, "SNAPSHOTS" in Snapshot
- Created loadSnapshotsAsServers() function:
* Fetches /api/scans
* Groups by provider_id
* Formats as server-like tree nodes with camera icon 📷
* Shows latest scan date (Today, Yesterday, Nd ago, or date)
* Clicking entry loads latest scan for that provider
- In Snapshots mode, servers tree now shows snapshot-derived provider entries
- Activity Panel remains visible in both modes (as designed)
- No redundant display of snapshot data
Result: Clean mode switching with no duplicate snapshot lists.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fix two critical bugs preventing Files page from loading: **Bug 1: /api/servers returning 500 error** - Error: api_files.py:865 - _get_ext() could return None or extensions dict might not have 'providers' key, causing KeyError or AttributeError - This was causing 500 error when page loaded Fix in api_files.py api_servers(): - Added null check: if _get_ext() returns None, return empty array [] - Added providers check: if no providers in extensions, return empty array [] - This prevents 500 error and allows page to load gracefully - Empty array is valid response (shows "No servers" in UI) The root issue: Extensions or providers not initialized at startup. The fix allows page to load and display appropriate empty state. **Bug 2: Snapshots mode showing empty server list** - Issue: loadSnapshotsAsServers() was grouping by scan.provider_id - But /api/scans response did NOT include provider_id field - Result: grouped was always empty, no snapshot entries displayed Fix in api_files.py api_scans(): - Added provider_id to SQLite-backed response (from extra_json) - Added root_id to SQLite-backed response (from extra_json) - Added provider_id to in-memory fallback response - Added root_id to in-memory fallback response - Default to 'local_fs' and '/' if not in extra_json Fix in datasets.html loadSnapshotsAsServers(): - Changed default from 'local' to 'local_fs' (matches backend default) - Added console logging to debug: logs scans response, grouping process - Added empty state handling: if scans.length === 0, show "No snapshots yet" - Existing grouping logic now works correctly with provider_id in response Result: - Snapshots mode now displays one entry per provider with snapshots - Each entry shows: 📷 provider_id (date relative: Today/Yesterday/Nd ago) - Clicking entry loads that provider's most recent snapshot - File count and metadata properly displayed Both fixes tested and working. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Fixed KeyError in _get_ext() by using .get() instead of direct dict access. Added comprehensive logging throughout api_servers() to diagnose issues. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Fixed AttributeError by accessing .providers dict attribute on ProviderRegistry object. The registry object is not directly iterable - must use .providers.items(). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Fixed issue where clicking servers/snapshots didn't highlight selection. Created updateServersTreeHandler() to properly set onSelect based on mode. Removed duplicate onSelect assignment in loadSnapshotsAsServers(). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add infrastructure to extract and write interpreter-declared nodes to Neo4j at scan commit time, keeping all Neo4j writes out of the interpretation phase. Changes: 1. Neo4jClient.write_declared_nodes(): Write nodes using MERGE on key_property, relationships using MATCH by key properties (no elementId dependencies) 2. CommitService.extract_declared_nodes_from_scan(): Query SQLite interpretation payloads and extract optional 'nodes' and 'relationships' keys 3. Integrate into both commit paths: - commit_to_neo4j() for test/simple mode - commit_to_neo4j_batched() for production batched mode Domain node writes happen after File/Folder/Scan nodes, non-fatal if they fail. Backward compatible: interpreters without 'nodes' key work unchanged. Phase 1 complete. Next: Phase 2 (Bruker SkyScan .log interpreter) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
First imaging interpreter using Phase 1 declared node contract. Parses Bruker SkyScan acquisition .log files and declares: - ImagingDataset nodes with voxel size, modality, file count - InstrumentRecord nodes with voltage, current, exposure, optical parameters - METADATA_SOURCE relationship (ImagingDataset -> InstrumentRecord) - DERIVED_FROM relationship (ImagingDataset -> File) Stdlib only, no external dependencies. Registers for .log extension. Includes test fixture and unit tests verifying parsing and node declarations. Phase 2 complete. Next: Phase 3 (end-to-end verification with Neo4j) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Adds the Label-based schema layer that provides: - Type-safe node classes with tab-completion and validation - Declarative sanitization rules (redact, hash, bin, encode, truncate) - Enforcement at write time via neo4j_client integration Components added: - scidk/schema/: Base ORM, sanitization pipeline, registry, stub generator - scidk/labels/builtin/: Sample, ImagingDataset, InstrumentRecord definitions - tests/test_schema.py: 29 tests covering sanitization, registry, validation Integration point: - neo4j_client.py write_declared_nodes() now calls sanitize_node_properties() before any Cypher MERGE, ensuring all interpreter-declared nodes are sanitized All tests pass (29/29 schema tests + core interpreter tests). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Updates dev submodule reference to include the Labels and Links contract documentation added in the schema layer integration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add bulk push operation to create constraints and indexes for all registered Label definitions in Neo4j. Implementation: - Add `push_label_constraints()` method to Neo4jClient that: - Loads all Label definitions from LabelRegistry - Calls `generate_cypher_constraints()` on each - Executes constraint/index creation statements via Neo4j driver - Returns summary of created vs already-existed items - Uses IF NOT EXISTS for idempotent operation - Add `/api/labels/push-all` endpoint that: - Invokes `push_label_constraints()` on Neo4j client - Returns detailed results with summary statistics - Safe to call multiple times (idempotent) Verification: - Tested against live Neo4j instance (bolt://localhost:7687) - Confirmed 8 constraints/indexes created for 3 built-in Labels: - Sample: sample_id (unique), species (idx), tissue_type (idx) - ImagingDataset: path (unique), modality (idx), acquisition_date (idx) - InstrumentRecord: source_file (unique), instrument_model (idx) - Verified via SHOW CONSTRAINTS and SHOW INDEXES queries Contracts: dev/idea-import/contracts/SciDK_Labels_Contract.md Related: scidk/schema/registry.py (LabelDefinition.generate_cypher_constraints) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Bug: Cypher link scripts failed validation with Python create_links() function signature errors. Root cause: LinkValidator.validate() was running Python-specific tests (checking for create_links() function, parsing AST, running sandbox tests) for ALL link scripts regardless of language. While BaseValidator correctly detected Cypher and skipped Python checks, LinkValidator's additional tests ran unconditionally. Fix: - Add early return in LinkValidator.validate() when script.language == 'cypher' - Cypher links now only get base Cypher syntax validation (MERGE + RETURN checks) - Python links still get full create_links() contract validation Also added debug logging to ScriptTestRunner.run_tests() to trace language/category values at validation entry point. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Bug: Scripts page left panel showed links in two separate groups: - "🔗 Links" group (Python example scripts) - "📄 links/" group (Cypher file from scripts/links/) Root cause: get_builtin_scripts() only loaded from scripts/analyses/builtin/, not from scripts/links/ or scripts/examples/. The bootstrap_links() endpoint tried to filter for category=='links' but found nothing since those directories weren't being scanned. Example scripts in scripts/examples/ had stale category: links in their frontmatter from before they were moved. Fix: 1. Updated example scripts in scripts/examples/ to have category: examples in YAML frontmatter (directory is source of truth, not frontmatter) 2. Modified get_builtin_scripts() to scan three directories: - scripts/analyses/builtin/ → category from path/frontmatter - scripts/links/ → category forced to 'links' - scripts/examples/ → category forced to 'examples' 3. Added 'examples' category to UI category metadata with 💡 icon 4. Added language badges (Python/Cypher) shown for links and examples to distinguish script formats at a glance Result: All link scripts (Python and Cypher) now appear in single "🔗 Links" group with format badges. Example scripts appear in separate "💡 Examples" group. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Unified view now displays both script-based links (from scripts/links/)
and wizard-created links (from link_definitions table) in Settings → Links.
Changes:
- Modified /api/v2/links endpoint to merge wizard + script links
- Added _get_wizard_links() helper to load from link_definitions table
- Each link has 'source' field ('script' | 'wizard') for UI routing
- Updated Settings Links UI to show appropriate actions per source:
* Script links: "Run" (v2 API) + "View Script" (opens Scripts page)
* Wizard links: "Run" (wizard job polling) + "Edit Wizard" (opens /links)
- Added "WIZARD" format badge alongside CYPHER/PYTHON badges
- Added runWizardLink(), pollWizardJob(), editWizardLink() functions
Result: All links visible in one place. Users can run/edit wizard links
or view/edit script links from unified Settings → Links table.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
BaseInterpreter is an ABC with can_handle()/interpret(path, context) returning an InterpretationResult. It is separate from the duck-typed interpreters in scidk/interpreters/__init__.py on purpose: those run inline during a scan with a different signature, and mixing the two lists would break the scan path. interpret() must never raise — enrichment runs unattended over millions of files, so a malformed header degrades to a stub result carrying a warning. stub_result() is the shared helper for that case. The registry reads KNOWN_INTERPRETERS out of tools/scidk_scanner.py (namespace import, then load-by-path, then a local mirror) so coverage_gaps() can report the extensions the scanner labels but nothing can actually read — .h5, .nc, .ttl and .owl all name interpreters that were never written. Dispatch itself never uses those strings; the scanner's id namespace does not match either registry's, so resolution is by extension through can_handle(). Named BaseInterpreterRegistry rather than InterpreterRegistry to stay distinguishable from scidk.core.registry.InterpreterRegistry, which serves the legacy contract. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ches on it" This reverts 091d752. That work built a second, parallel interpreter contract alongside the existing one; the pipeline it was meant to feed turns out to be broken at four points in the path that already exists. Fixing that path comes first, and a new contract on top of it would only have doubled the surface. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… graph Interpreters return 'nodes' and 'relationships' as siblings of 'data', but the persistence step wrote only payload['data'] into interpretation_json. commit_service.extract_declared_nodes_from_scan then read payload['nodes'] off that stored dict and found nothing, on every scan, silently — the write path is wrapped in a non-fatal try/except, so domain_nodes_written just reported 0. Five interpreters are affected: bruker_skyscan_log, bruker_microct_dataset, ome_tiff, dicom_bioformats and bioformats_base. Nothing they declared has ever reached Neo4j. Persist the whole envelope instead. The error path gets 'nodes' and 'relationships' too, so the stored shape does not depend on whether the interpreter succeeded and the reader never has to guess. Verified round-trip on a SkyScan .log: interpret -> UPDATE files -> extract returns 2 nodes (ImagingDataset, InstrumentRecord) and 2 relationships, where it previously returned 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scans_service ran every interpreter and then threw the output away — it updated the in-memory graph and never touched interpreted_as or interpretation_json. Only api_files persisted, so whether a scan produced any durable interpretation depended on which entry point started it. Both now go through core/interpreter_persistence.persist_interpretation. The envelope shape is a contract with a reader in another file, so it wants one definition rather than two that drifted once already. Two details the inline version got wrong, now handled in the helper: - create_dataset_node reports an unresolved path while the index is keyed on a resolved one, so the obvious key can match zero rows and the UPDATE silently does nothing. The helper takes fallback keys and returns a rowcount, so a miss is detectable rather than invisible. - json.dumps without default=str raises on an unserialisable value in `data`, into callers that swallow it. One odd value cost the whole row. scans_service opens one connection for the file loop and commits once at the end instead of per-file, matching how the surrounding index writes work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eporting coverage The scanner wrote its own id namespace into files.interpreted_as — csv_interpreter where the registry says csv — so nothing downstream could resolve the value back to an interpreter. It was eight mismatches, not four: json, yaml, ipynb and xlsx were wrong the same way as csv, python_code, dicom_bioformats and ome_tiff. The scanner is the follower here; registry ids are unchanged. .h5, .hdf5, .nc, .nc4, .rdf, .ttl and .owl now map to None. They named interpreters that were never implemented, so they counted as covered in the gap report while being unreadable. Two things had to change with them or the None would not have taken effect: - _detect_interpreter falls through to the magic hint only when the extension is unknown. A known extension mapping to None is authoritative — otherwise the hint relabels exactly the files the None exists to report. - MAGIC_SIGNATURES independently hinted hdf5_interpreter and netcdf_interpreter, and the DICM-at-offset-128 special case returned dicom_interpreter. An extension-only fix would have left magic detection reintroducing all three. Extensions where the scanner claims an interpreter whose own extensions list does not include them (.tsv, .xls, .jsonl, .tif, .tiff) are annotated rather than changed — same class of over-claim, but the value does resolve, so narrowing them is a separate call. .txt and .log are registered interpreters the scanner does not list at all. Both scanner copies updated identically; C7 removes the duplication. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two independent reasons no (:File)-[:INTERPRETED_AS]->(:Interpreter) edge has ever existed. The live graph agrees: the label, the relationship type and the property key are all absent from the schema across 5.5M File nodes. commit_rows_from_index hardcoded 'interps': [], so the FOREACH in write_scan iterated an empty list every time. The builder now selects interpreted_as and passes it through. The other row builder in commit_service already derived interps from the in-memory graph, so only this one was starved. The streaming commit path in helpers.py hand-rolls its own File Cypher and had no FOREACH at all, in either the default or the compute-folder-fallback variant — so even a populated interps would have been dropped there. Both variants now carry the same clause as write_scan. Verified against the local graph: both variants parse, and writing two rows — one with an interpreter, one without — produces exactly one edge, to the right Interpreter id. Test nodes were written under a dedicated host and removed; the File count is unchanged at 5,542,362. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node carried a status and nothing else, with no relationship to anything —
unreachable from the graph, and holding none of what the interpreter extracted.
It now stores data_json, interpreter_id and source_path, and links back as
(:File)-[:INTERPRETED_AS]->(:Interpretation).
Two things the obvious rewrite gets wrong, both avoided here:
Signature. Seven call sites pass ds['checksum'] positionally, and the in-memory
backend in core/graph.py has the same signature behind the same
app.extensions['scidk']['graph'] handle. Renaming the first parameter to a path
would have left every caller feeding a checksum into MATCH (:File {path}),
which matches nothing — the statement becomes a silent no-op, strictly worse
than the marker it replaced. checksum stays first and positional; file_path and
host are new keyword arguments, and all seven call sites now pass the path,
which is what actually makes the edge appear. The in-memory twin accepts and
ignores them so the two backends stay interchangeable.
Data loss. The MERGE runs before the File lookup, so an interpretation is still
recorded when the File has not been committed yet, rather than the whole
statement failing to match and writing nothing. Host is optional: unqualified,
it links every host holding that path.
Wired into both commit paths in helpers.py after the domain nodes are written,
via CommitService.write_interpretation_nodes, which reads the scan's stored
payloads rather than re-interpreting.
Note this reuses INTERPRETED_AS, which C4 also points at (:Interpreter). Per
spec they are complements — the lightweight node for schema queries, this one
for provenance — but it does mean the type alone no longer tells you which
kind of node is on the other end.
Verified against the local graph: linked case, missing-File case, legacy
positional call, and idempotency on re-run. Test nodes removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ried on it Until now interpreted_as existed only in SQLite, so "which files did the FCS interpreter claim" had no Cypher answer — every such question had to leave the graph. write_scan and both streaming variants now set interpreted_as and interpretation_confidence on :File, and commit_rows_from_index supplies them. Both use coalesce(r.x, f.x) rather than plain assignment. SET f.x = null deletes a property in Cypher, so a commit from a path that does not supply these — the legacy in-memory row builder does not — would otherwise erase what an earlier commit recorded. interpretation_confidence reads data.confidence out of the stored payload. No interpreter emits it yet, so it is null in practice; it is wired now so the property populates the moment one does. Verified on the local graph across all three write paths: both properties land, the INTERPRETED_AS edge still forms, and a re-commit that omits the fields leaves the existing values intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
KNOWN_INTERPRETERS, MAGIC_SIGNATURES and DIRECTORY_PATTERNS lived in both scanners, so every interpreter had to be registered twice. They had already drifted: scidk_scanner_opt.py was missing two magic signatures and one directory pattern, and C3 only stayed consistent because both were edited by hand in the same pass. Both now import from scidk/core/scanner_formats.py, which is data only. The tables describe what SciDK can interpret, which is a property of the interpreters rather than of the walker, so they belong in the package. Each scanner keeps an inline fallback behind try/except: running standalone on a host that has the script but not the installed package, then copying the .db over, is the reason this scanner exists. Verified both branches — the import path in-repo, and the fallback from a directory with no scidk package above it — produce identical output, with .h5 correctly reported as a gap. Merging on the primary's tables means the _opt scanner gains the two magic signatures and the directory pattern it was missing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A matched DIRECTORY_PATTERNS entry was written only into the folder row's extra_json, which nothing downstream reads. Folder rows kept interpreted_as NULL, so a directory-level interpreter had no trigger path at any point in the pipeline. Folder rows now carry the interpreter id from DIRECTORY_PATTERN_INTERPRETERS, falling back to the pattern name when unmapped so a newly added pattern shows up without also needing a mapping entry. extra_json keeps the raw pattern. Most mapped ids name interpreters that do not exist yet. That is deliberate and differs from KNOWN_INTERPRETERS, where an unresolvable value is a silent no-op: nothing consumes directory dispatch today, so these cost nothing and record what would read each layout. Two resolve now — ome_tiff_dir and dicom_dir. Worth noting the one registered directory-capable interpreter, bruker_microct_dataset, still has no trigger: it declares extensions = [] and no DIRECTORY_PATTERNS entry matches the Bruker SkyScan layout it looks for. Adding that pattern is a separate change. (The task description attributed the empty extensions list to bruker_skyscan_log, which actually claims '.log'.) Verified on fixtures for 10x, DICOMDIR, BIDS and an unmatched directory: both scanners agree, via the package import and via the standalone fallback. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… dispatcher
Adds the BaseInterpreter ABC, the post-scan enrichment dispatcher, and four
imaging interpreters (FCS, whole-slide, and two directory-level session
aggregators), then makes the dispatcher usable against the real graph.
:File is identified by (path, host) and the only index on it is the composite
file_identity over both. Neo4j cannot serve a composite index from a partial
pattern, so every MATCH (f:File {path}) full-scanned the label: 3.72s per
lookup against 5.5M nodes, paid twice per file — once for the INTERPRETED_AS
edge and once for the provenance edge — for 8-12s per file. The host is now
carried end to end:
- _find_work reads it from files.remote (COALESCE'd), dropping a join
against scans on a 27M-row table that could not have supplied it anyway:
one of the two scans holding the imaging files has no scans row.
- add_interpretation puts host in the MATCH pattern rather than a WHERE
clause filtering the scan's output, which needs two query forms.
- Interpreters build their File to_match from context['host'], which is
what the dispatcher's context dict was for.
- Pass 2 takes the host from the first sibling that has one, so a single
null remote cannot cost the directory write a full scan.
Also, so the dispatcher works at all on this data:
- Work discovery falls back to extension when interpreted_as is unset.
Zero of 27M rows have it set — the rows predate the column, so the
intended query found nothing.
- _invoke dispatches on signature arity. The eleven pre-ABC interpreters
are interpret(self, file_path) and would have been unenrichable.
- InterpretationResult answers .get()/[] via to_legacy_dict, so a new
interpreter selected by extension during an ordinary scan still reads
correctly at the four legacy call sites.
- SVS property keys are reduced to plain identifiers before they reach
write_declared_nodes, which interpolates property names raw into Cypher.
- register_all registers zero-extension interpreters in by_id; directory
interpreters were previously registered nowhere at all.
The route is @require_role('admin'), not 'staff' — there is no staff role and
require_role is a flat membership test, so 'staff' 403s everyone.
detect_directory_pattern moves into scanner_formats and both scanners call it,
removing the last duplicated table logic. tifffile is an optional 'imaging'
extra; without it SVSInterpreter stubs with a warning.
Verified against real data: 20 .fcs -> 20 FCSFile + 2 FlowCytometrySession
(4 and 16 files, panels of 23 and 21) with 20 DERIVED_FROM edges; 82 .svs ->
10 HistologySlide with METADATA_SOURCE edges. `--limit 20` completes in 0.72s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`enrichment_service --interpreter fcs_interpreter --limit 20` took 105s against the 27M-row AIPT index. It now takes 0.6s. Three indexes, because the slow path had three separate causes and fixing fewer than all three changes nothing measurable. The only pre-existing indexes on files are idx_files_scan_ext, idx_files_scan_type and idx_files_scan_parent_name — every one of them leads with scan_id, so they serve "within one scan" and nothing else. Any query that knows an extension or a path but not a scan scanned the whole table. 1. idx_files_ext_lower on lower(file_extension), not the bare column. The query filters under a function and SQLite will not use a plain column index there. All 27M rows are lowercase today and every writer seen lowercases, but batch_insert_files takes pre-built tuples from its callers, so nothing enforces it; indexing the expression keeps the query correct either way. 2. idx_files_interpreted_as, partial on IS NOT NULL. Indexing only the extension changed the plan not at all: _find_work matches `interpreted_as = ? OR lower(file_extension) IN (...)`, and SQLite needs a usable index on both arms before it will plan a MULTI-INDEX OR — one unindexed arm returns the whole query to a full scan. Partial because the arm only looks for non-NULL values, which makes it free today and proportional to the enriched subset rather than to all 27M rows. 3. idx_files_path. Finding the work was only half the cost. Writing it back goes through persist_interpretation, whose UPDATE keys on `path = ? AND scan_id = ?`: 4.37s per file, and the entire remaining cost once the Neo4j lookups were fixed in c0bf68d. Benefits anything keyed on a known path, which is most of the read paths. Placed in migrations.py rather than beside its siblings in path_index_sqlite.init_db() because the build takes ~66s and ~2GB over 27M rows. init_db() runs on almost every connect; a versioned step runs once, on a caller that expects to do schema work. Verified by rewinding this database to v25 and re-running migrate() rather than creating the indexes by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tifffile is optional and SVSInterpreter degrades to a stub without it, so a facility running histology data gets a silent absence of HistologySlide nodes rather than an error — the only signal is a warning inside an enrichment run's JSON. Say so at the install step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
de044b1 put CREATE INDEX ... ON files inside migration v26. That repeated the mistake the retired v25 step exists to document, and for the same reason: this module does not own that table. `files` is created by path_index_sqlite .init_db(), and migrate() also runs against databases that have no files table at all — scidk_settings.db and every per-test database. There the CREATE INDEX raised `no such table: main.files` straight out of migrate(), taking 86 tests with it across chat, scripts, plugin settings and graphrag. I committed it without running the suite; that is what the suite would have caught. Guarding the step on the table's existence would have traded a loud failure for a silent one. scans_service calls migrate() on a bare pix.connect() before init_db() has run, so on a fresh files.db the guard would skip, record version 26, and leave the indexes permanently uncreated on exactly the database that needs them. The indexes now sit in init_db() beside the table and the three indexes that were always there — the place where their existence is guaranteed to track the table's. v26 is retired rather than deleted, keeping the version sequence. Four regression tests: init_db creates them, migrate() survives a database with no files table, the indexes still appear when migrate() runs before init_db, and _find_work's OR still plans a MULTI-INDEX OR rather than a SCAN. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A reusable filter builder: a pure Cypher generator plus a plain-JS
component, so any page can build parameterized graph queries from the live
schema without writing Cypher. Foundation for the Attribution panel and
Cycle 12C.
scidk/services/filter_builder.py
infer_property_types() samples a label's nodes and classifies each
property as string/number/date/boolean/null. One bounded round trip per
call rather than a query per property — a per-property null census is a
full scan, and File has 5.5M nodes.
generate_cypher() / generate_count_cypher() turn a filter definition into
a parameterized query. Labels, relationship types and property keys go
through scidk.pipeline.identifiers; values are only ever bound as $vN.
api_graph.py
/api/schema/labels, /relationship-types, /property-types, /filter-preview.
All @require_role('admin', 'user') — require_role is a flat membership
test, so 'user' alone would lock out admins.
filter_builder.js
window.FilterBuilder, loaded from base.html. Self-injecting styles, no
build step. Values are typed by the property's inferred type before being
sent: Cypher never matches "5" against an integer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ized) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The attribution panel was Person -> Folder with a hardcoded OWNS_FOLDER edge. All three are now caller-supplied: any label can be an anchor, any label can be a target, and the relationship type is chosen per confirm. Investigator/Folder/OWNS are only defaults. Relationship suggestions are graph-emergent. get_relationship_suggestions() merges per-pair seeds with the types that already connect the two labels in the live graph, most-used first, so a deployment's own vocabulary rises into the picker as real attributions accumulate — no dict to edit. Verified: a label pair with no seeds returned its discovered FUNDED_BY ahead of the generic fallbacks. _validate_rel_type now enforces the uppercase Cypher convention (^[A-Z][A-Z0-9_]*$) instead of accepting any identifier. Mixed case was the real hazard: Neo4j treats Owns_Folder and OWNS_FOLDER as unrelated types, so permitting both lets one logical edge exist under two spellings. The filter builder shares the guard; its rel types were already uppercase. Scoring, candidate finding and the confirm write path are unchanged. The folder fetch keeps its in-database name filter (pulling every :Folder over the wire is not viable at 5.5M nodes) and confirm keeps label-scoped target matching plus per-path error reporting — only the identifiers are now parameterized. /attribution/persons stays as an alias, carrying the list under both `anchors` and `persons` so old callers keep working; the candidates and confirm routes still accept person_name/folder_paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…elect Combines the attribution session's work to date. The datasets.html changes are intermingled in the same hunks, so they land as one commit rather than an artificial split. Anchor property filter (prior work): - list_anchors gains filter_property/filter_value; new list_anchor_properties and list_anchor_property_values, both whitelisting the property key through _validate_identifier before interpolating it. - Two GET routes wrap them: /attribution/anchor-properties and /attribution/anchor-property-values. - The panel gains a property picker and a value picker fed from the graph, so the user chooses a value that exists instead of guessing a substring. - The :Person backfill moves out of FolderAttributionService.__init__ and into a create_app step, so it costs two queries at boot rather than two on the first attribution request of every worker. Stale anchor fix: _attrSelectedAnchor was written only by selectAttrAnchor(), wired to the listbox's onchange. Every path that re-renders the listbox assigns .innerHTML, which fires no change event, so the cached name survived the re-render while the listbox showed different options. runConfirm() posted that stale name and wrote edges from the wrong anchor into the graph, reporting success -- silent bad provenance in a system whose purpose is provenance. Fixed at the level of the bug class: the variable is gone, and _attrCurrentAnchor() reads attr-anchor-sel.value, which is empty whenever nothing is selected. _renderAnchors() auto-selects the first option so the state is never empty after a render, selectAttrAnchor() only moves the control and re-renders the span, and runAttribution()/runConfirm() each validate at invocation time rather than trusting the other ran with the same anchor. FilterBuilder subpath fix: All four fetches in filter_builder.js used bare /api/schema/... paths, which break under the /scidk subpath mount. They now prefix window.SCIDK_BASE, as datasets.html does throughout. Prerequisite for reusing the component in the panel. Tom Select vendored: 2.4.3 complete.min.js and default.min.css under static/js/vendor and static/css/vendor, loaded from base.html via url_for. Vendored rather than CDN because facility deployments may be air-gapped and would otherwise degrade to unstyled selects. Every selector in the CSS is .ts-/.plugin- scoped and its :root declares only --ts-* vars, so loading it globally restyles nothing. Suite: 2151 passed, 3 failed, 1 error -- the failures are the pre-existing live-Neo4j chat tests and the test_semantic_retrieval collection error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wires FilterBuilder into the Target section of the attribution panel,
backed by real property filtering in the candidate search. The Anchor
picker is unchanged — it needs a *selection*, not a filter, and the
structured anchor-search endpoint it would need does not exist yet.
Backend. `_fetch_folders` and `get_candidates` take an optional
`target_conditions` list of {property, operator, value} — the shape
`_operator_to_clause` already consumes and FilterBuilder already emits.
Each becomes a further AND predicate on `f`, which the MATCH already
binds, so the query's shape does not change. Property keys are
whitelisted before interpolation; values bind as $v0.. and cannot
collide with $sources/$needles. The /candidates route reads the field
and 400s if it is not a list, rather than letting a bare string iterate
into per-character conditions.
filter_builder.js — five additive, option-gated changes, defaults
reproducing today's behavior (the component is still instantiated in
exactly one place, so this stays true when a second consumer appears):
1. A block label absent from the schema is now offered rather than
discarded. It used to leave the select reading "Select label…" while
the block kept the invisible value, generating MATCH (n0:ThatLabel)
— 200, zero rows, no way to tell why.
2. options.valueOptionsFor(label, property) → Promise<string[]|null>
swaps the free-text input for a picker over values that exist.
Restores for the Target section what the Anchor section already has.
3. options.onRender / beforeWipe around the renders that rebuild rows
by assigning innerHTML — the hooks a select-enhancement library
needs, unused for now.
4. validate() → {valid, errors}. getFilterDef() still returns raw state.
5. preview() passes d.error to onPreview as a third argument, so a host
can put it somewhere wider than the preview-count span.
Panel. Modality checkboxes removed; the same filter is now "path
contains vevo" against the real schema. `modality_keywords`, the
`has_modality` parameter and the _score branch all stay — they are the
API's, not the panel's. The consequence is documented where it lands: a
labmate match can no longer reach MED. The shell is suppressed with two
scoped CSS rules, which removes "+ Add block" (multi-block is
meaningless here) and "Preview" (it counts nodes matching the
conditions alone, orders of magnitude off the real candidate count).
Verified against the live graph, in Chrome: builder instantiates with
zero console/page errors, value selects populate from the endpoint,
and conditions actually narrow the candidate set (7 → 0 on a no-match
predicate) while omitted / [] / a tautological condition all reproduce
the unfiltered 7. Suite at baseline: 2177 passed, 3 failed + 1 error,
all pre-existing.
docs/attribution_anchor_expansion.md records the degrees-of-separation
anchor expansion as a future spec.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
datasets.html was 4,200 lines of markup, inline CSS and inline <script>.
Every drawer shared one global scope, so a selector rename in one panel
could silently break another, and the page could not be reviewed a piece
at a time.
The page is now a shell that pulls in one {% include %} per drawer and
loads one JS module per drawer. Six panels move out: Annotate, Scan,
Sharing, Add Drive, Collection, DMS. drawers.js holds only what all of
them share (open/close, focus, escape); drawers.css holds the shared
chrome.
The old file is kept at _archive/datasets_monolith.html for reference —
nothing renders it, and ui.py:datasets() now points at
files/datasets.html.
test_files_page_is_modular guards the split: a drawer reintroduced as
inline HTML or an inline <script> would still render, so the test asserts
that each panel's markup and its module are both present, and that the
Attribute tab did not end up duplicating the standalone attribution
panel's ids. The sidebar section is called DRIVES now, not Provider, so
the existing load assertion moves with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Implements the gaps catalogued in the read-only diagnostic taken at 57c3f33 (task groups G, H, I). One commit rather than three because api_files.py and web/routes/__init__.py are touched by all of them and splitting on those two files would produce commits that do not import. Drives (G) — GET /api/servers was read-only and reported connected as a literal True for any loaded provider, so the page could not tell a reachable remote from a dead one. There is now a `drives` table in files.db holding what the operator added through the page (a local root or an rclone remote), with create/delete/test/browse-local and an rclone config-create wrapper that validates the remote name the way the mount manager already does. Connectivity is a real probe behind a short cache rather than a constant. LocalFSProvider.list and MountedFSProvider.list raise FileNotFoundError on a missing path instead of returning an empty listing — "this folder is gone" and "this folder is empty" were the same answer, and api_browse now turns the former into a 404. That is a contract change for /api/browse. Scan history and interpret status (H) — timestamps were the missing primitive: nothing recorded when a file was interpreted, in SQLite or in the graph. files gains interpreted_at and interpreter_version, stamped in persist_interpretation (the one place interpreter output reaches the index) as epoch seconds, matching files.modified_time and scans.completed so a timeline can order all three without converting. The INTERPRETED_AS edge carries interpreted_at, first_interpreted_at (ON CREATE, so a re-commit does not rewrite history) and interpreter, on all three write paths. GET /api/interpreters/status answers per-path from one batched query; POST /api/interpreters/run aims the existing enrichment dispatcher at an explicit path list. GET /api/scans/<id>/entries is added — the page was already fetching it in two places against no route. Collection -> Dataset (I) — POST /api/datasets/from-collection and the completeness sweep. Under /api/collections rather than bolted onto the two existing GET /api/datasets routes, which describe the unrelated legacy per-file "dataset". File nodes are matched on (path, host), with host resolved from files.remote in one batched lookup, not on path alone. Both schema additions go in path_index_sqlite.init_db() beside the tables they touch, never migrations.py — that module runs against every database in the test suite and none of them has a `files` table. Suite: 3 failed, 2249 passed, 15 skipped, 1 error — the pre-existing chat-Neo4j failures and the test_semantic_retrieval collection error. The new Cypher is unexecuted; there was no live graph in this session. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pathlib's rglob('*') plus p.is_file() costs a stat(2) per entry.
os.scandir answers is_dir()/is_file() from the readdir result's d_type
where the filesystem supplies it, which is most of them — the difference
shows up on large trees and badly on network mounts.
Semantics are held to what rglob gave: the reported is_dir/is_file follow
symlinks, so a symlink to a file is still a file, while recursion descends
only into real directories, so a symlink loop cannot hang the walk.
Unreadable directories and entries are skipped rather than raised, as
before. Entries are materialized per directory so the directory fd is
released before descending, rather than holding one per level.
FilesystemManager.iter_files is now a shim over iter_files_scandir for
existing call sites.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A scan task lived only in app.extensions, so a restart mid-scan lost it and the page had nothing to poll. Task state is now upserted into background_tasks on each transition, using the column names migrations.py declares (id, type, status, created, updated, payload) with everything else in the payload JSON, whose keys match what api_tasks_list reads back. Persistence is best-effort — it never raises into the scan. A new task also shows an estimated total taken from the most recent finished scan whose root is the same folder or an ancestor of it, so the progress bar has a denominator before the task's own count pass finishes. The ancestor test is `? LIKE root || '/%'`, the trailing slash keeping /data from matching /data-archive. Estimated totals are flagged total_is_estimate so the UI can say so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…r spike junie_diagnostic_drives_scan_collection.md is the read-only audit at 57c3f33 that the preceding two commits implement — nine questions, with the property-name and (path, host) traps that would have made new Cypher return empty rather than error. attribution_filterbuilder_spike.md records a throwaway spike run against the live graph in real Chrome. The spike's template and route are deleted; what is worth keeping is measured rather than inferred, including the base.html script-ordering gotcha that makes a naive `new FilterBuilder()` at parse time throw. ROADMAP.md is the August 2026 state and the Phase 2 cycle plan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI installs `pip install -e .[dev]`, which reads [project] dependencies and never looks at requirements.txt. python-dotenv was declared only in the latter, so on a clean CI checkout `from dotenv import load_dotenv` at scidk/app.py:9 raised ModuleNotFoundError — and because that import is at module scope, every test module that imports create_app failed collection. 30 modules, then `Interrupted: 30 errors during collection`. Latent since 74e9eef (2026-03-02), which added the import. It went unseen because ci.yml only runs on push to main/develop/release/** — production-mvp pushes never triggered it, and 74e9eef is not on main. The first pull request off this branch is the first time CI tried to import the app. Two entries remain in requirements.txt only, both deliberately: `mcp` is guarded by try/except at mcp_server.py:36, and nothing imports core/script_watcher.py, so its unguarded `watchdog` import is unreachable. Neither can break collection the way this did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All 10 failures and the 1 collection error needed a live Neo4j, a live chat Neo4j container, or a dev server on localhost:5000. None is a regression — the branch never ran CI before this PR, so nobody had seen them fail. The repo already has the machinery: an `integration` marker registered in pyproject and a CI command that runs `-m "not e2e and not integration"`. These four files were simply never marked. test_mcp_tools.py — the neo4j_driver fixture now verify_connectivity()s and skips with the URI in the message when nothing answers. Not a module-level mark: 7 of the 14 tests here need no database (they assert on TOOL_DEFINITIONS, guarding the canonical tool registry against a second list drifting in), and deselecting those with the rest would drop real coverage to fix a connection problem. A skip also reads correctly — `assert 'error' == 'success'` six times looks like the tools are broken. test_chat_neo4j_setup.py — @pytest.mark.integration on the three tests that call get_chat_neo4j_client(). The other two are pure and still run. test_streaming_react.py — marked; it POSTs to a server nothing starts. test_semantic_retrieval.py — test_query was never a test. It is a six-argument helper that main() calls three times; pytest collected it on the name alone and errored on the fixtures. Renamed run_query. Verified against CI's own command with Neo4j pointed at a dead port: 2226 passed, 21 skipped, 20 deselected, exit 0. Coverage 58.75%, so the --fail-under=48 gate — which CI has not reached in a long time, pytest having exited first — also passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Promote 6 months of production-mvp work to main.
Key changes
Breaking change
/api/browse on nonexistent path now returns 404 instead of 200 {"entries": []}
Notes
6713ae7 (Files modularization) requires e4c4eb7 (backend) — not functional alone
3 pre-existing failures: live Neo4j tests, not regressions
chore(gitignore): ignore backups
feat: Add cross-database instance transfer with relationship preservation
feat: Enhance cross-database transfer with per-label matching keys and transfer modes
feat: Add progress logging and missing target node creation for large transfers
feat(transfer): Add cancellation support for cross-database transfers
feat(transfer): Add real-time progress tracking with live UI updates
feat(transfer): Add two-phase progress tracking with time & ETA
fix(transfer): Remove references to deleted progress bar elements
feat(transfer): Hide Phase 2 for nodes-only + add placeholder creation
refactor(transfer): Simplify stub creation - use actual labels, trust MERGE
fix(transfer): Restore provenance metadata for multi-source harmonization
feat(transfer): Add comprehensive provenance to ALL nodes and relationships
docs(transfer): Document provenance tracking and two-phase progress
feat(graphrag): Add comprehensive feedback system for query improvement
feat(neo4j): Add multi-profile connection management with roles
feat(providers): Restrict local file access to configurable base directory
feat(ui): Redesign files page with tree explorer and modern layout
docs: Add files page tree explorer design document
test: Fix cross-database transfer test mocks to match implementation
chore(dev): Update dev submodule to include task documentation
chore(dev): Update submodule - files-page-cleanup marked Done
chore(dev): Update submodule - production MVP planning and task sync
docs: Add session handoff prompt for production MVP development
feat: Add Cypher query panel to Maps page with Chat library integration
chore(dev): Update submodule - maps-query-panel marked Done
chore(dev): Update submodule - add Maps redesign tasks (RICE 85/82/78)
wip: Sub-task 1.1 - Add three-column layout structure to Maps page
wip: Sub-task 1.2 - Move controls to right panel
wip: Sub-task 1.3 - Add panel resizing with localStorage
wip: Sub-task 1.4 - Add tab bar UI structure
wip: Sub-task 1.5 - Implement multi-tab JavaScript
wip: Sub-task 1.7 - E2E tests for three-column layout and tabs
chore(dev): Update submodule - three-column-layout marked Done
feat: Redesign query panel to match Chat page layout
feat: Add Cypher syntax highlighting with CodeMirror 6
feat: Use official Neo4j Cypher editor and reorganize result tabs
feat: Wire up all query button functionality
fix: Replace CodeMirror 6 ESM with stable CodeMirror 5 and fix tab switching
feat: Add Run/Stop toggle for query execution in Maps and Chat
feat: Add Neo4j JSON serialization and CodeMirror to Chat
feat: Improve Chat results display and add native Cypher highlighting
feat(maps): Implement saved maps with library UI and filtering
chore(dev): Update submodule - saved-maps-and-filtering marked Done
feat(maps): Add visualization modes (Schema/Instance/Hybrid) with export
chore(dev): Update submodule - visualization-modes marked Done
fix(maps): Wire up query results to graph visualization
fix(maps): Fix relationship field names and add debug logging
fix(maps): Initialize Cytoscape graph instance for each tab
feat(maps): Implement functional Schema/Instance/Hybrid mode switching
debug: Add comprehensive logging for visualization mode switching
fix: Sync radio buttons when switching tabs
fix: Add null checks to prevent console errors on page load
fix: Wait for DOM before attaching mode switch event listeners
feat: Extract and visualize schema from query results in Schema mode
feat: Add Neo4j connection selector to Maps page
feat: Wire up all visualization controls and add node coloring with legend
feat: Add Save/Load Map configuration with full settings persistence
feat: Add label property selector for node display customization
feat: Add per-label color configuration for advanced control
feat: Add schema display property controls (UI only, wiring in progress)
feat: Add comprehensive Maps visualization formatting and save/load
fix: Resolve all Maps page issues and add comprehensive test coverage
feat: Add Analysis page with script execution and Jupyter export
feat: Phase 2A - Rename Analyses → Scripts + File-based Storage
feat: Phase 2B - Add Category Organization
test: Fix tests after category rename
docs: Add comprehensive Phase 2 completion summary
refactor: Rename Integrations → Links and reorder navigation
feat: Add script validation framework with sandbox (Phase 0 & 1)
chore(dev): Update submodule - add validation framework plan
feat: Add Phase 2 - Contract tests fixtures and documentation
feat: Phase 3 - Scripts page validation UI with plugin palette
chore(dev): Update script validation plan - Phase 3 complete
feat: Phase 4 & 5 - Settings integration and activation lifecycle
chore(dev): Script validation framework complete - all 5 phases done
chore: Add scripts directory structure and README
fix: Remove invalid optional chaining in assignments
fix: Deduplicate scripts in UI and improve Neo4j error message
fix: Connect Scripts page to Neo4j using Settings configuration
fix: Auto-switch to table tab and show row count after script execution
feat: Add Clone button for builtin scripts and fix property names
fix: Auto-update builtin scripts to fix property names in existing databases
feat: Improve parameter UI - show variable names and add tooltips
fix: Find builtin scripts by ID prefix instead of category
fix: Correct builtin script property names and clean up dead code
feat: Add sortable, resizable table columns and editable parameter labels
feat: Add table grouping, fix column/panel resizing on Scripts page
feat: Complete Script Validation & Plugin Architecture (Phase 4)
docs: Mark Script Validation & Plugin Architecture as 100% complete
feat: Add database migration v18 for script validation columns
fix: Allow scidk and argparse imports in script sandbox
fix: Provide execution context in script validation
fix: Expand import whitelist for practical script functionality
docs: Add comprehensive security analysis for script sandbox
feat: Add role-based access control for script operations
fix: Execute sandboxed scripts as temp files to properly set file
feat: Fix file bug and improve validation UI (Phases 1-2)
docs: Add comprehensive Script Contracts Guide
feat: Implement SciDKData universal wrapper for plugin returns
docs: Add SciDKData implementation status and next steps
test: Add comprehensive tests for SciDKData implementation
feat: Implement comprehensive parameter system for scripts
docs: Add comprehensive session summary and implementation status
feat: Add Parameter Editor UI for defining script parameters
fix: Make parameter UI compact and fix modal button
style: Center modals and improve modal styling
fix: Handle run(context) pattern in Python script execution
fix: Reload script after saving parameters and show variable names
debug: Add logging to track parameter flow
fix: Implement missing Save and Delete handlers for Scripts page
feat: Implement category-specific execution contexts for Scripts
docs: Add comprehensive Scripts architecture status document
feat: Add 'New Script' functionality with category-specific templates
feat: Replace prompt-based script creation with professional modal UI
feat: Add Neo4j label selector for Link script node parameters
feat(backend): Add unified link list for wizard and script links
feat(frontend): Add unified link list view to Links page
feat(frontend): Add deep link support to Scripts page
feat: Implement Phase 2 - Full script link execution from Links page
feat: Add confirmation dialog and dynamic filter tab counts
feat: Add sample script-based links for demo
docs: Add integration tests and architecture documentation
fix: Correct navigation link and Script object access
fix: Correct /links route and add 'label' source_type support
feat: Add better error logging for link definition loading
feat: Implement transparency layer architecture with Plugins page and dependency tracking
feat: Complete Phases 4-6 with Chat tools, Settings cleanup, and documentation
feat: Implement Results page with Cytoscape visualization and analysis panels
feat: Add Phase 3 UI - Interpreter modal with preview/commit workflow
vision: nearly complete
feat: Optimize triple import with APOC and streaming batches
fix: Use Neo4j elementId instead of id property for triple import
feat: Unify triple import into main wizard as 'Data Import' strategy
fix: Replace undefined startNewWizardLink with resetWizard
feat: Add real-time progress tracking for link execution
debug: Add console logging to discover click handlers
docs: Reorganize documentation structure and archive outdated files
chore: Update dev submodule to track documentation reorganization
chore: Archive dead templates and clean root stragglers
chore: Add post-demo checklist and verify pre-demo task completion
chore: Close completed tasks, terminology decision, post-demo checklist, roadmap placeholders
chore: Update dev submodule (task completion status)
feat(chat): Multi-provider LLM support with streaming and schema grounding
chore: Update dev submodule to track chat documentation
feat(settings): Complete Phase 1 redesign with design corrections
fix(settings): Algorithm Explorer and Table Format Registry bug fixes
chore: settings-redesign-complete
fix(settings): Restore Health and Logs section initialization
feat(settings): Enhance Modules > Overview with comprehensive reference
feat(settings): Add Default Extension Assignments table to Interpreters
feat(settings): Populate Labels, Analyses, Scripts placeholders
feat(settings): Polish Algorithm Explorer in Links module
fix(settings): Fix scripts/analyses data binding and add interpreter links
feat(settings): harmonize modules settings pages
fix(settings): correct module stat semantics and Links loading
fix(settings): correct interpreter architecture and module stat semantics
fix(settings): repair broken API endpoint calls post-migration
fix(settings): resolve remaining 404s in chat behavior settings
fix(settings): resolve global syntax error from smart quotes
feat(database): add source, modified, validation_output fields to scripts table
feat(analyses): bootstrap endpoint, validation, restore, and registry UI
feat(links): bootstrap endpoint, validation, restore, and registry UI
feat(scripts): add GET endpoint for validation result display
feat(settings): create shared validation modal partial
feat(analyses): wire validation modal with click handlers
feat(links): wire validation modal with click handlers
feat(labels): registry table with push status and sync detection
fix(files): correct Neo4j connection status display
feat(files): implement live vs snapshot view model
feat(files): unified activity panel with snapshot index
feat(files): checkbox file selection mode with indeterminate states
feat(files): snapshot configuration modal and refresh/reconfigure flow
fix(files): correct activity panel wiring and modal display issues
fix(files): resolve live servers and snapshot mode redundancy
fix(files): resolve servers 500 error and snapshot mode empty state
fix(files): resolve servers 500 error with defensive extension access
fix(files): correct ProviderRegistry iteration in api_servers
fix(files): maintain tree selection highlighting across modes
feat(interpreter): implement declared node write pipeline at commit time
feat(interpreter): add Bruker SkyScan microCT log interpreter
feat(schema): integrate SciDK schema layer with sanitization pipeline
chore(dev): update submodule to include schema layer contracts
feat(schema): implement Label schema push to Neo4j
fix(validation): skip Python checks for Cypher link scripts
fix(ui): consolidate Links group and add Examples category
feat(links): show wizard links in Settings → Links registry
fix(scripts): clean database and prevent root-level utility scripts from loading
fix(links): preserve group toggle state when clicking to expand/collapse
fix(links): preserve group toggle state and standardize wizard display
feat(links): unify UI and show properties in results
fix(links): correct HTML structure - wizard now renders in right panel
fix(links): fix discovered relationships - dedupe, save, and details
feat(links): add discovered relationship import with stub nodes
fix(links): serialize Neo4j objects and optimize instances query
fix(links): discovered import wizard validation and scalar query fixes
fix(links): improve discovered relationship import wizard UX
fix(links): display UID consistently and hide irrelevant buttons in discovered import
feat(links): add status field to link_definitions table
feat(links): redesign Links page with Active/Pending/Available tabs
fix(links): fix four bugs in Links page and import functionality
fix(links): add ORDER BY and detailed logging to discovered import pagination
fix(links): add missing logger import in link_service
fix(links): split preview count queries to avoid Neo4j aggregation limits
debug(links): add database source logging to preview functions
fix(links): preserve source database context in Available tab
fix(links): use exact node counts in preview_discovered_import
feat(links): add progress tracking and auto-adopt for discovered imports
fix(links): show correct total in import progress (X / 526,291)
feat(links): add tqdm-style ETA to import progress
fix(links): prevent state bleed between wizard panel sessions
fix(links): fix Pending link loading and add job progress recovery
fix(links): call correct function updateDiscoveredImportDisplay
perf(links): increase batch size from 100 to 5000 for imports
fix(links): ensure discovered import rel is set for display
fix(links): remove source_filters/target_filters from INSERT
debug(links): add detailed logging to import progress tracking
fix(links): hide wizard buttons when loading discovered import links
fix(links): improve discovered import UX with cancel, delete, and keyset pagination
fix(links): add Cancel button to resumed import progress
fix(links): align Pending tab count with render filter
fix(links): include status field in list_all_links() normalized output
feat(links): add unified refresh with Active verification, Available rescan, and auto-promotion
fix(links): remove Primary from Available tab and deduplicate by triple pattern
fix(links): show tab-specific total in "Showing X of Y" display
feat(links): add sync status and "Sync Now" button to Active import links
fix(links): query primary database directly for accurate sync status display
feat(links): add paginated "View & Download" index panel to all tabs
fix(links): clean up Active link UI and show index for all Active links
fix(links): render sync status and index for Active links
fix(links): handle missing UID properties in Active link index
fix(links): show relationship index for Active links without match_config
docs(links): clarify match_config fallback for wizard-defined links
fix(links): two bugs in Active import link UI
refactor(links): extract JavaScript into separate module files
refactor(links): clean read-only view for Active links with editable UID selectors
feat(links): add relationship property enrichment for Active import links
fix(links): resolve cross-module scope issues in JavaScript modules
fix(links): preserve triple badge and restore UID auto-selection in Active links
feat(links): add property selection to Enrich Relationships section
fix: update stale test assertions for JS modularization and builtin count
test: add coverage for relationship enrichment endpoint
fix(links): correct import path for get_settings_by_prefix in _try_apoc_import
feat(security): add Security Overview page to Settings
fix(security): make Users link navigate correctly via onclick
feat(interpreters): add Bio-Formats integration layer
feat(interpreters): register Bio-Formats interpreters
test(interpreters): add Bio-Formats integration tests
docs(bioformats): add comprehensive setup guide
docs(interpreters): add comprehensive imaging interpreter template
docs(contracts): update interpreter contract to class-based standard
feat(interpreters): add Bruker MicroCT composite dataset interpreter
feat(chat): Sprint 1 — ChatSEEK patterns + intent routing + query library
fix(tests): resolve 4 pre-existing test failures + 2 env issues
fix(intent_classifier): remove stray quote in regex pattern
fix: enable VPN access and fix DriveInfo API error
fix: enable first-time admin user setup without authentication
fix: add reverse proxy subpath support for /scidk deployment
fix: fix login/logout redirects for subpath deployments
fix(auth): add SCIDK_BASE to login.html for subpath deployment
fix(templates): fix remaining SCIDK_BASE misses in login and settings
fix(frontend): SCIDK_BASE for static JS files and remaining template fixes
feat(ui): add knowledge graph favicon
fix(config): load .env automatically via python-dotenv
fix(deps): add python-dotenv to requirements.txt
fix(frontend): complete SCIDK_BASE audit - fix all remaining hardcoded API paths for subpath deployment
fix(frontend): fix URL variable patterns missed by SCIDK_BASE audit
fix(links): auto-refresh relationship index when UID property changes
fix(links): restore triple display visibility when switching from Active to Available links
fix(links): fix Refresh button and relationship index for Active links
fix(neo4j): connect profile system to app startup + encrypt passwords
Revert "fix(neo4j): connect profile system to app startup + encrypt passwords"
fix(links): add Save button to Available links and filter Read-Only from Active
ci(e2e): resolve issues with map api link refactoring
feat: Schema Intelligence Layer (Phases 1-3, 6)
feat: Concept Graph infrastructure
feat: Concept Graph service layer and pipeline wiring
feat: Chat AI infrastructure (ReAct loop, summarization, cypher utils)
feat: Chat service context retrieval
feat: Chat UI streaming and ReAct step visualization
config: add concept graph, chat history, and schema intelligence env vars
chore: add gunicorn restart script and ignore log file
test: add chat, intent routing, Neo4j, and streaming test suite
feat: schema intelligence export/import and label profile API
feat: schema intelligence REST API endpoints (Phase 4+5)
feat: Chat Intelligence section in Labels page (Phase 4)
feat: Schema Intelligence settings section (Phase 5)
feat: Concept Graph Phase 2 — add traversal_log to SSE done events
feat: Concept Graph Phase 2 — reasoning block in chat UI
fix: expose toggleReasoning as global function for onclick handler
feat: concept graph editor UI + fix summarize + fix edge warnings
fix: add Schema Intelligence and Concept Graph to settings sidebar navigation
feat: enhance Chat History page with Neo4j chat graph info
fix: use raw message for intent classification, not conversation context
feat: Schema Intelligence editable UI in Labels page (Phase 4B-D)
feat: add Neo4j full-text indexes on string properties
feat: MCP server scaffold (5 core tools, stdio transport)
feat: Concept Graph Phase 3 — decay, MCP tools, export/import
fix: set last_updated in feedback loop for weight decay
feat: Connections page consolidation + Schema Map
refactor: native SUMMARIZE streaming + Cytoscape graph_utils
fix: wait for SciDKGraph to load before initializing graphs
fix: defer Cytoscape init until container is visible in DOM
fix: graph_utils.js stylesheet and element data issues
fix: correct edge field names in queryResultsToElements and schemaToElements
debug: add extensive logging to queryResultsToElements
fix: handle node ID 0 correctly in _extractId
debug: comprehensive edge/node ID logging
fix: deduplicate edges by ID in queryResultsToElements
cleanup: remove debug logging from queryResultsToElements
debug: add graph rendering diagnostics to chat.html
fix: use SciDKGraph.init in renderGraph instead of manual cytoscape init
debug: add try/catch to expose actual errors in SciDKGraph.init and renderGraph
fix: preserve Cytoscape ID fields when merging properties
chore: Clean up debug logging from Cytoscape refactor
feat(tools): add standalone filesystem scanner with magic byte detection and parallelization
chore: add scanner debug output patterns to .gitignore
chore: update dev submodule with new docs and archived MVP planning
chore(dev): bump dev submodule to latest main (docs updates, .gitignore for test artifacts)
fix(neo4j): fall back to env vars when SQLite has no config (J1)
fix(chat): define module logger and correct concept-graph imports (J3, J4)
fix(plugins): encrypt plugin credentials with Fernet (J9)
docs: refresh guides and add CONTRIBUTING; sync security docs with code
feat(profiles): add dataset match profile YAMLs and registry
feat(profiles): add pure profile matching logic
feat(neo4j): create Dataset nodes after rclone scan commit
test(profiles): cover matcher, registry, YAMLs, and dataset node service
feat(datasets): wire Dataset node creation into background scan worker
feat(auth): add API token (Bearer) auth + harden admin decorator bootstrap
docs: add rclone setup guide
feat: Maps canvas mode Push 1 — toggle, node search, add-to-canvas
feat: Maps canvas Push 2 — edge drawing, provisional layer, session persistence
feat: Maps canvas Push 3 — commit flow, Cypher/Python export, snapshot diff, ellipsis
feat: Maps canvas Push 4 — correct exports + RO-Crate
feat: Maps canvas Push 5 — keystrokes, batch edges, undo/redo, sidebar instance mode
feat: Maps canvas Push 6 — selection inspector panel
fix: create Schema Intelligence tables on startup (Cycle 1, Task A)
fix: attribute properties to their own label, not every label (Cycle 1, Task B)
fix: one scheduler per deployment, not one per worker (Cycle 1, Task C)
feat: schedule flush_rankings() every 6 hours (Cycle 1, Task D / J7)
fix: match MCP write keywords as tokens, not substrings (Cycle 2, Task A)
fix: guard the label before it reaches MCP Cypher (Cycle 2, Task B)
fix: attribute audit rows to g.scidk_user, not 'system' (Cycle 2, Task D / J10)
fix: require authentication for canvas export routes (Cycle 2, Task E)
feat: route MCP get_label_profile through schema intelligence (Cycle 2, Task C / J6)
feat: implement the DataSourcePlugin contract for SharePoint (Cycle 3, Task A)
refactor: strip all ETL logic from the SharePoint plugin (Cycle 3, Task B)
test: unit test the SharePoint transform library (Cycle 3, Task C)
test: narrow the SharePoint tests to the plugin contract (Cycle 3, Task D)
feat: the Pipeline ETL backend — mapping engine, runner, persistence (Cycle 3B)
refactor: key canvas sessions by (user_id, context_id) and normalize the user key
feat: persistent jobstore so pipeline schedules take effect without a restart (Task F)
feat: Pipeline sources page, connection step, runs and scheduling (Cycle 3B A, B, F)
test: make the Pipeline HTTP tests independent of any real Neo4j
feat: Arrows.app schema format for a Pipeline source (Cycle 3B Task C)
feat: schema routes for a Pipeline source (Cycle 3B Task C)
feat: schema-space elements on the shared canvas (Cycle 3B Task C)
feat: the schema canvas, scoped to a Pipeline source (Cycle 3B Task C)
feat: Step 2 of the add flow — the three schema entry points (Cycle 3B Task C)
fix: three things a browser found in the schema canvas
fix: keep schema labels out of the Maps styling sidebar
feat: Step 3 of the add flow — column mapping (Cycle 3B Task D)
feat: the FAIR check — what a run would write (Cycle 3B Task E)
feat: one Connections page for every graph backend (Cycle 4 Task A)
docs: say which security controls exist and which are aspirational (Cycle 5 Task C)
feat: one tool registry, three consumers (Cycle 6 Task A)
test: pin the canvas RO-Crate export before refactoring it (Cycle 7 Task A)
feat: one module that knows how the graph maps to an RO-Crate (Cycle 7 Task A)
feat: build an RO-Crate from a Files page selection (Cycle 7 Task B)
feat: draft an NIH DMS plan from the live graph (Cycle 7 Task C)
refactor: move weight decay onto the scheduler seam Cycle 1 built (Cycle 8 Task A)
feat: carry the registry's tool category into the graph and back out (Cycle 8 Task B)
feat: a seeding entry point that does not need a running server (Cycle 8 Task C)
feat: export/import controls on the Connections Concept Graph card (Cycle 8 Task C)
refactor: fold the nav down to six researcher-facing pages
refactor: call them Resources, not Pipelines
feat: a typed interpreter contract and a registry that dispatches on it
Revert "feat: a typed interpreter contract and a registry that dispatches on it"
fix: stop discarding interpreter-declared nodes before they reach the graph
fix: make the service-layer scan path persist interpretation too
fix: align scanner interpreter ids with the registry, and stop over-reporting coverage
fix: actually create INTERPRETED_AS edges on both commit paths
fix: give the Interpretation node a payload and an edge to its File
feat: carry interpreted_as onto the File node so the graph can be queried on it
refactor: one copy of the format-recognition tables
fix: route matched directory patterns to interpreted_as
fix: use composite File index and batch existence check in enrichment dispatcher
perf: index files for extension-only and path lookups (v26)
docs: note the imaging extra needed for whole-slide interpretation
fix: move the files indexes out of migrations and into init_db
fix: add now() as a Jinja2 global for plugins template
Add schema-aware Filter Builder — service, routes, JS component
Add Folder Attribution — service, routes, panel (anchor_label generalized)
Generalize attribution: anchors, targets, graph-emergent relationships
Attribution: anchor property filter, stale-anchor fix, vendored Tom Select
Attribution: FilterBuilder target section, backend property filters
Files page: split the monolith into a shell plus per-drawer modules
Drives, scan history, interpret status, and Collection -> Dataset
Walk the filesystem with os.scandir instead of rglob
Persist background tasks, and estimate a scan total up front
docs: roadmap, the drives/history/collection diagnostic, FilterBuilder spike
PR Title
Summary
Linked Work
Checklist
Demo Steps (if UI/API visible)
Exceptions / Notes