diff --git a/.env.example b/.env.example
index e1fcbd72..ccd6d8ac 100644
--- a/.env.example
+++ b/.env.example
@@ -76,31 +76,3 @@ SCIDK_CHAT_OPENAI_API_KEY=
# Anthropic key for entity extraction (optional, falls back to pattern matching)
SCIDK_ANTHROPIC_API_KEY=
-
-# ------------------------------
-# Chat Neo4j (for chat intelligence & history)
-# ------------------------------
-# Separate Neo4j instance for chat metadata (not research data)
-CHAT_NEO4J_URI=bolt://localhost:7688
-CHAT_NEO4J_AUTH=neo4j/your-password-here
-
-# ------------------------------
-# Concept Graph Neo4j (for meta-reasoning)
-# ------------------------------
-# Separate Neo4j instance for concept graph (intents, tools, schema abstractions)
-# This layer enables semantic intent classification and graph-based planning
-SCIDK_CONCEPT_NEO4J_URI=bolt://localhost:7689
-SCIDK_CONCEPT_NEO4J_AUTH=neo4j/concept-graph-password
-SCIDK_CONCEPT_GRAPH_ENABLED=1
-
-# Chat Intelligence Settings
-SCIDK_CHAT_CONTEXT_RETRIEVAL_TOP_K=3 # messages per strategy (semantic + node overlap)
-SCIDK_CHAT_CONTEXT_MAX_TOTAL=6 # max total messages in context
-SCIDK_CHAT_REACT_MAX_STEPS=4 # ReAct loop step limit
-SCIDK_CHAT_REACT_TOKEN_BUDGET=3000 # tokens per step
-
-# Background Summarization (fast model for compression)
-SCIDK_CHAT_SUMMARIZATION_MODEL=qwen2.5:7b
-
-# Embedding Model (for semantic retrieval)
-SCIDK_CHAT_EMBEDDING_MODEL=nomic-embed-text
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 891f0afe..19d2f50c 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -20,9 +20,9 @@ jobs:
run: |
python -m pip install --upgrade pip
pip install -e .[dev]
- - name: Run pytest with coverage (exclude E2E and integration tests)
+ - name: Run pytest with coverage (exclude E2E)
run: |
- python -m coverage run -m pytest -q -m "not e2e and not integration"
+ python -m coverage run -m pytest -q -m "not e2e"
python -m coverage report
python -m coverage xml
- name: Upload coverage to Codecov
@@ -30,9 +30,9 @@ jobs:
with:
file: ./coverage.xml
fail_ci_if_error: false
- - name: Check coverage threshold (48%)
+ - name: Check coverage threshold (50%)
run: |
- python -m coverage report --fail-under=48
+ python -m coverage report --fail-under=50
# E2E tests temporarily disabled in CI (Feb 2026)
# The test suite has stability issues (auth conflicts, timing, cleanup) that need dedicated attention.
diff --git a/.gitignore b/.gitignore
index 95a221d0..a544053a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -75,13 +75,3 @@ backups/
dev/test-runs/tmp/
dev/code-imports/
docs/archive/
-gunicorn.log
-
-# Scanner and debug outputs
-/ambig_paths.txt
-/ambiguous_paths.txt
-/big_dir_tree.txt
-/magic_results.txt
-/out.txt
-/out*.txt
-/out_*.txt
diff --git a/ARCHITECTURE_HANDOFF.md b/ARCHITECTURE_HANDOFF.md
deleted file mode 100644
index 5bd1ada7..00000000
--- a/ARCHITECTURE_HANDOFF.md
+++ /dev/null
@@ -1,122 +0,0 @@
-# SciDK — Architecture & Status Handoff
-
-_Snapshot date: 2026-06-22 · Branch: `production-mvp` @ `23ba225` (the de-facto trunk)_
-
-> **Read this, not the root `README.md`.** The README has accreted ~2 years of per-cycle
-> notes and is partly stale (e.g. it still claims "in-memory graph, Neo4j not wired" — false;
-> Neo4j, a Concept Graph, and an MCP server are all wired in). This doc reflects the actual code.
-
----
-
-## 1. What SciDK is
-A Flask web app for **scientific data management on top of a knowledge graph**. It scans
-filesystems (local, mounted, or rclone remotes), interprets files, builds a property graph in
-Neo4j, and exposes an LLM chat interface (ReAct + GraphRAG) over that graph, plus an MCP server
-so external agents (Claude Desktop, etc.) can query it.
-
-## 2. Branch / repo topology (important)
-- **`production-mvp`** is the real mainline — it contains all of `main` plus **320 commits** on top.
-- **`main`** is a **stale snapshot frozen at 2026-03-02** (last PR `#52`). The PR→main review flow
- (PRs #32–#52, per `dev/README-planning.md`) stopped there. All March work (AI/Chat stack,
- scanner, Cytoscape fixes) was committed **directly to `production-mvp` and never PR'd into main**.
-- **`dev/`** is a **git submodule** = the planning/docs repo (stories, phases, tasks, features;
- see `dev/README-planning.md`). It is NOT app runtime code.
-
-## 3. Tech stack
-- **Backend:** Python 3.12, Flask (app-factory + blueprints), Swagger/flasgger at `/api/docs`,
- `ProxyFix` for reverse-proxy/subpath deploys, gunicorn in prod (`restart_gunicorn.sh`).
-- **Graph DB:** Neo4j 5 (`docker-compose.neo4j.yml`; Bolt 7687 / HTTP 7474). In-memory graph
- backend exists as a fallback (`SCIDK_STATE_BACKEND=memory`).
-- **Relational/state:** SQLite with WAL — multiple DBs: `scidk.db`, `scidk_settings.db`,
- `scidk_path_index.db`. Auto-migrations run on boot (`scidk/core/migrations*`), reported via `/api/health`.
-- **LLM:** pluggable providers — Ollama / Anthropic / OpenAI (`scidk/ai/provider_factory.py`,
- `llm_providers.py`). GraphRAG/embeddings stack is **optional** (`requirements-graphrag.txt`,
- PyTorch+CUDA, gated by `SCIDK_GRAPHRAG_ENABLED`).
-- **Frontend:** server-rendered Jinja templates + vanilla JS; **Cytoscape.js** for graph viz
- (`scidk/ui/static/js/graph_utils.js`, `SciDKGraph`).
-- **Tests:** pytest, 3 tiers via markers (`unit` / `integration` / `e2e`); Playwright for E2E.
-
-## 4. App initialization — `scidk/app.py::create_app()`
-Single application factory. Wires, in order:
-1. Logging, channel defaults, ProxyFix, Swagger.
-2. SQLite auto-migration; state-backend toggle (`sqlite` default).
-3. **Graph backend** (`core/neo4j_config.create_graph_backend` → Neo4j or in-memory).
-4. **Concept Graph driver** (optional, `SCIDK_CONCEPT_GRAPH_ENABLED=1`; graceful fallback to
- hard-coded intent classifier if unavailable).
-5. Interpreter registry; FilesystemManager; FS providers (`local_fs`, `mounted_fs`, `rclone`).
-6. Everything hung off `app.extensions['scidk']` (graph, concept_driver, registry, fs, providers,
- in-session registries for scans/tasks/directories, neo4j_config/state, rclone_mounts, settings).
-7. Hydrate persisted state from SQLite (last scan, rclone mounts/settings, Neo4j creds).
-8. Register **26 blueprints** (`web/routes/register_blueprints`); init auth middleware (RBAC).
-9. **Plugin system**: label-endpoint registry, plugin-template registry, plugin-instance manager,
- plugin loader (discovers `plugins/`).
-10. Backup scheduler (also runs Concept Graph weight-decay job).
-
-## 5. Package layout (`scidk/`)
-- **`ai/`** — LLM layer. `react_loop.py` (ReAct agent), `chat_graph.py`, `summarization.py`,
- `cypher_utils.py`, `mcp_tools.py`, `schema_context.py`, `provider_factory.py` / `llm_providers.py`.
-- **`services/`** — business logic. Key ones:
- - `concept_graph_service.py` (~1k lines) — Concept Graph: semantic schema layer with weighted
- edges + decay, export/import, intent routing.
- - `schema_intelligence.py` (~675 lines) — editable schema intelligence (labels/profiles, embeddings).
- - `chat_neo4j_client.py`, `chat_service.py` — DB-persisted chat with permissions.
- - `graphrag/` (incl. `intent_classifier.py`) — LOOKUP vs REASONING routing, text-to-Cypher.
- - `neo4j_client.py`, `query_service.py`, `link_service.py` / `link_service_v2.py`,
- `label_service.py`, `fs_index_service.py`, `scan_index_service.py`, `commit_service.py`,
- `saved_maps_service.py`, `metrics.py`.
-- **`web/routes/`** — 26 Flask blueprints (see §6). Pattern: `_get_ext()` to reach
- `app.extensions['scidk']`; background work uses threads with captured `app.app_context()`.
-- **`core/`** — infra: migrations, neo4j/channel/rclone config, plugin loader + registries,
- backup manager/scheduler, alert manager, settings, logging.
-- **`concept_graph/`** — `intents.yaml`, `schema.cypher` (Concept Graph seed/schema).
-- **`interpreters/`, `labels/`, `schema/`, `scripts/`, `export/`** — file interpreters, label
- definitions, schema generation, script registry, exporters.
-- **`mcp_server.py`** — standalone MCP server (`python -m scidk.mcp_server`), 5 read-only tools:
- `query_knowledge_graph`, `get_schema`, `summarize_dataset`, `get_label_profile`, `list_labels`.
-
-## 6. HTTP surface — 26 blueprints (`scidk/web/routes/`)
-UI (`ui.py`: `/`, `/datasets`, `/map`, `/chat`, `/settings`, …) + API blueprints:
-`api_files`, `api_graph`, `api_maps`, `api_scripts`, `api_tasks`, `api_chat`, `api_queries`,
-`api_neo4j`, `api_admin`, `api_interpreters`, `api_providers`, `api_annotations`, `api_labels`,
-`api_integrations`, `api_links` (legacy) + `api_links_v2` (LinkRegistry), `api_settings`,
-`api_auth`, `api_users`, `api_audit`, `api_alerts`, `api_logs`, `api_plugins`, `api_system`
-(chat self-awareness), `api_results`.
-_(The `web/routes/README.md` count of "9 blueprints / 91 routes" is outdated — it's 26 now.)_
-
-## 7. Major subsystems delivered (newest → older)
-- **AI / Chat stack (Mar 2026, production-mvp only):** Concept Graph (Phases 1–3: weighted edges,
- decay, MCP tools, export/import), Schema Intelligence editable UI, ReAct streaming chat with
- reasoning-block visualization, MCP server, Neo4j full-text indexes. Seed scripts at repo root
- (`seed_concept_graph*.py`, `seed_schema_embeddings.py`, `apply_concept_schema.py`).
-- **Standalone filesystem scanner (Mar 2026):** `tools/scidk_scanner.py` + `_opt.py` —
- magic-byte detection + parallel enumeration; spec in `tools/SCIDK_SCANNER_SPEC.md`.
-- **Cross-database transfer V2 (#51):** provenance, progress tracking, cancellation.
-- **Security (#40):** multi-user auth + RBAC + DB-persisted chat with permissions.
-- **Config backup/restore (#41) + auto-lock on inactivity (#44).**
-- **Plugins:** `plugins/{example_plugin, example_ilab, table_loader, ilab_table_loader}` —
- UI-instantiable plugin templates + instances; the recent demo focused on the **iLab plugin +
- concept-graph seeding** (see the `dev` submodule branch `feature/ilab-plugin-and-demo-seeding`).
-- **Providers:** rclone is core (feature flag removed, #31); FUSE mount manager under `./data/mounts/`.
-
-## 8. How to run
-```bash
-pip install -e .[dev] # or requirements.txt (lite) / requirements-graphrag.txt (AI)
-docker compose -f docker-compose.neo4j.yml up -d # Neo4j (default creds neo4j/neo4jiscool)
-scidk-serve # or: python -m scidk.app → http://127.0.0.1:5000
-python -m scidk.mcp_server # MCP server for external agents
-make unit | make integration | make e2e | make check # tests
-```
-Key env: `SCIDK_STATE_BACKEND` (sqlite|memory), `SCIDK_PROVIDERS`, `NEO4J_URI`/`NEO4J_AUTH`,
-`SCIDK_CONCEPT_GRAPH_ENABLED`, `SCIDK_GRAPHRAG_ENABLED`, `SCIDK_CHANNEL` (stable|dev), `SCIDK_BASE`
-(subpath deploys). `.env` is auto-loaded via python-dotenv.
-
-## 9. Open items / gotchas for the next agent
-1. **main is stale.** Decide whether to back-merge the March AI/Chat + scanner work from
- `production-mvp` into `main`, or formally retire `main` in favor of `production-mvp`.
-2. **`dev` submodule diverges** (47 local-only vs 8 pinned-only commits) on
- `feature/ilab-plugin-and-demo-seeding`. Reconcile before committing the superproject's `dev` pin.
-3. **Stale root `README.md`** — large sections describe an earlier MVP; trust this doc + code instead.
-4. **GraphRAG/AI features are optional and heavy** (PyTorch/CUDA) and degrade gracefully when off.
-5. Repo root holds loose scripts/artifacts (`seed_*.py`, `*.db`, `htmlcov/`, `pytest_fully_output.txt`)
- that could use a cleanup pass.
-
diff --git a/README.md b/README.md
index d8fc9bbf..96ba959f 100644
--- a/README.md
+++ b/README.md
@@ -43,10 +43,9 @@ source scripts/init_env.fish --write-dotenv
3) Run the server:
```
scidk-serve
-# or the equivalent module form:
+# or
python3 -m scidk.app
```
-> **Canonical launch command:** `scidk-serve` (entry point `scidk.app:main`). `python3 -m scidk.app` is exactly equivalent. Other docs reference this section rather than restating launch commands.
4) Open the UI in your browser:
- http://127.0.0.1:5000/
@@ -205,8 +204,8 @@ Details:
- Add tests alongside new features in future cycles; see dev/cycles.md for cycle protocol.
## Notes
-- The graph backend is selectable via `SCIDK_GRAPH_BACKEND` (`memory` by default, `neo4j` to persist). Neo4j is fully wired: scans commit File/Folder/Scan nodes and relationships with post-commit verification (see `scidk/services/neo4j_client.py`).
-- With the in-memory backend, graph data resets on restart; use `SCIDK_GRAPH_BACKEND=neo4j` for persistence. Neo4j deployment docs reside in dev/ops/deployment-neo4j.md.
+- This MVP uses an in-memory graph; data resets on restart.
+- Neo4j deployment docs reside in dev/ops/deployment-neo4j.md, but Neo4j is not yet wired in the MVP code.
## Documentation
- Delivery cycles and planning protocol: dev/cycles.md
@@ -215,19 +214,11 @@ Details:
## Architecture
SciDK uses a modular Flask blueprint architecture for web routes:
-- **~27 blueprints** organize 300+ routes by functional area (UI, files, tasks, graph, Neo4j, providers, chat, labels, links, admin, interpreters, and more)
-- **Application factory**: `create_app()` lives in `scidk/app.py` (~314 lines); blueprints are registered via `register_blueprints()` in `scidk/web/routes/__init__.py`
+- **9 blueprints** organize 91 routes by functional area (UI, files, tasks, graph, Neo4j, providers, chat, admin, interpreters)
+- **Reduced footprint**: app.py reduced from 5,781 to 645 lines (89% reduction)
- **Clean separation**: Each blueprint is self-contained with proper import scoping
- See `scidk/web/routes/README.md` for detailed blueprint documentation
-## MCP Server (Model Context Protocol)
-SciDK exposes core functionality via an MCP server for external AI agents (Claude Desktop, etc.):
-- **5 core tools**: `query_knowledge_graph`, `get_schema`, `summarize_dataset`, `get_label_profile`, `list_labels`
-- **Read-only safety**: All queries are validated to block write operations
-- **Stdio transport**: Uses standard MCP protocol over stdin/stdout
-- **Setup guide**: See `docs/mcp-setup.md` for Claude Desktop configuration
-- **Run**: `python3 -m scidk.mcp_server`
-
## Scanning progress and background tasks (MVP)
- Current options:
- Synchronous: POST /api/scan runs immediately and returns when complete.
@@ -244,11 +235,13 @@ SciDK exposes core functionality via an MCP server for external AI agents (Claud
- Preview and download instances for File, Folder, and Scan labels as CSV (XLSX if openpyxl is installed).
## Neo4j integration
-- Status: Neo4j is fully wired. The app ships with docker-compose.neo4j.yml to run a local Neo4j; set `SCIDK_GRAPH_BACKEND=neo4j` (plus `NEO4J_URI`/`NEO4J_AUTH`) to persist to it. With the default in-memory backend, graph data is not persisted across restarts.
-- How it works:
- - The graph backend is selected at startup in `create_app()` via `SCIDK_GRAPH_BACKEND` (`memory` default, `neo4j` to persist); invalid Neo4j params fall back to in-memory.
- - Committing a scan writes `(:File)`, `(:Folder)`, `(:Scan)` nodes with `(Folder)-[:CONTAINS]->(File|Folder)`, `(File|Folder)-[:SCANNED_IN]->(Scan)`, and `INTERPRETED_AS` relationships, via `scidk/services/neo4j_client.py`.
- - Schema triples for `/api/graph/schema` are computed with Cypher (APOC variants available at `/api/graph/schema.apoc`).
+- Status: The app ships with docker-compose.neo4j.yml to run a local Neo4j, but the Flask app currently uses an in-memory graph.
+- Next steps to enable Neo4j writes/reads:
+ 1) Add a GraphAdapter interface and a Neo4jAdapter implementing upsert_dataset, add_interpretation, commit_scan, schema_triples.
+ 2) Add config/feature flag (e.g., SCIDK_GRAPH_BACKEND=neo4j) to switch adapters.
+ 3) Map current in-memory structures to Neo4j schema: (:File), (:Folder), (:Scan) nodes and CONTAINS, INTERPRETED_AS, SCANNED_IN relationships.
+ 4) Use Cypher or APOC to compute schema triples for /api/graph/schema.
+- Until then, data is not persisted to Neo4j. Use the CSV exports or the in-memory map for the demo.
## New in this cycle: Optional Neo4j schema endpoints and extra Instance exports
@@ -450,17 +443,19 @@ Local commands:
- make e2e → pytest -m e2e tests/e2e -q
- make check → runs unit, integration, and e2e sequentially
-CI is defined in `.github/workflows/ci.yml` and runs pytest with `-m "not e2e"`. **E2E tests are disabled in CI as of Feb 2026** (the E2E job is commented out) — run them locally with `npm run e2e` or `pytest -m e2e`.
+See .github/workflows/tests.yml for the CI matrix that runs each tier.
## Verify CI and Record Demo Artifacts
Follow these steps to verify the full test suite and automatically capture screenshots/JSON for the demo.
1) Verify CI on GitHub
-- Navigate to GitHub → Actions → the CI workflow (defined in `.github/workflows/ci.yml`).
-- Confirm the pytest job is green (runs `pytest -m "not e2e"`).
-- E2E (Playwright) is not run in CI as of Feb 2026; verify E2E locally with `npm run e2e`.
-- Click into the latest run to see logs if the job is red.
+- Navigate to GitHub → Actions → "Tests" workflow (defined in `.github/workflows/tests.yml`).
+- Confirm that all three matrix jobs are green:
+ - tier=unit
+ - tier=integration
+ - tier=e2e (installs Playwright browsers automatically)
+- Click into the latest run to see logs if any job is red.
2) Run all tests locally (mirrors CI)
```
diff --git a/SciDK_NCI_Demo.pdf b/SciDK_NCI_Demo.pdf
deleted file mode 100644
index 83c30959..00000000
Binary files a/SciDK_NCI_Demo.pdf and /dev/null differ
diff --git a/apply_concept_schema.py b/apply_concept_schema.py
deleted file mode 100644
index bfc8ab35..00000000
--- a/apply_concept_schema.py
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/usr/bin/env python3
-"""Apply schema constraints to the Concept Graph."""
-from neo4j import GraphDatabase
-
-uri = "bolt://localhost:7689"
-auth = ("neo4j", "concept-graph-password")
-
-driver = GraphDatabase.driver(uri, auth=auth)
-
-with open('scidk/concept_graph/schema.cypher', 'r') as f:
- schema_cypher = f.read()
-
-# Split by semicolon and execute each statement
-statements = [s.strip() for s in schema_cypher.split(';') if s.strip() and not s.strip().startswith('//')]
-
-with driver.session() as session:
- for stmt in statements:
- if stmt:
- try:
- print(f"Executing: {stmt[:60]}...")
- session.run(stmt)
- print(" ✓ Success")
- except Exception as e:
- print(f" ✗ Error: {e}")
-
-driver.close()
-print("\n✓ Schema constraints applied to Concept Graph")
diff --git a/backups/scidk-backup-20260210_031853-85217c23.zip b/backups/scidk-backup-20260210_031853-85217c23.zip
new file mode 100644
index 00000000..159e4e05
Binary files /dev/null and b/backups/scidk-backup-20260210_031853-85217c23.zip differ
diff --git a/backups/scidk-backup-20260210_070000-17137b43.zip b/backups/scidk-backup-20260210_070000-17137b43.zip
new file mode 100644
index 00000000..0ed2742e
Binary files /dev/null and b/backups/scidk-backup-20260210_070000-17137b43.zip differ
diff --git a/backups/scidk-backup-20260210_070000-40a80893.zip b/backups/scidk-backup-20260210_070000-40a80893.zip
new file mode 100644
index 00000000..1afa66a8
Binary files /dev/null and b/backups/scidk-backup-20260210_070000-40a80893.zip differ
diff --git a/dev b/dev
index 7e804faf..2ace2601 160000
--- a/dev
+++ b/dev
@@ -1 +1 @@
-Subproject commit 7e804fafb00eb03347703a17dec8a1b01df1965c
+Subproject commit 2ace2601c0ac8590aa27e66a81976574df1a957a
diff --git a/docker-compose.concept-graph.yml b/docker-compose.concept-graph.yml
deleted file mode 100644
index 423b0cfa..00000000
--- a/docker-compose.concept-graph.yml
+++ /dev/null
@@ -1,38 +0,0 @@
-version: '3.8'
-
-services:
- neo4j-concept:
- image: neo4j:5.13.0
- container_name: scidk-concept-graph
- ports:
- - "7689:7687" # Bolt protocol
- - "7476:7474" # HTTP (browser access)
- environment:
- - NEO4J_AUTH=neo4j/concept-graph-password
- - NEO4J_server_memory_heap_initial__size=512m
- - NEO4J_server_memory_heap_max__size=2G
- - NEO4J_server_memory_pagecache_size=512m
- - NEO4J_dbms_security_procedures_unrestricted=apoc.*
- - NEO4J_dbms_security_procedures_allowlist=apoc.*
- volumes:
- - concept-graph-data:/data
- - concept-graph-logs:/logs
- networks:
- - scidk-network
- restart: unless-stopped
- healthcheck:
- test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider localhost:7474 || exit 1"]
- interval: 10s
- timeout: 5s
- retries: 5
- start_period: 30s
-
-volumes:
- concept-graph-data:
- driver: local
- concept-graph-logs:
- driver: local
-
-networks:
- scidk-network:
- driver: bridge
diff --git a/docker-compose.neo4j.yml b/docker-compose.neo4j.yml
index 4f23fd1c..90c3962b 100644
--- a/docker-compose.neo4j.yml
+++ b/docker-compose.neo4j.yml
@@ -28,26 +28,3 @@ services:
retries: 10
start_period: 20s
- neo4j-chat:
- image: neo4j:2025.10.1
- container_name: scidk-neo4j-chat
- restart: unless-stopped
- environment:
- - NEO4J_AUTH=${CHAT_NEO4J_AUTH:-neo4j/changeme}
- - NEO4J_server_memory_heap_initial__size=512m
- - NEO4J_server_memory_heap_max__size=512m
- - NEO4J_server_memory_pagecache_size=256m
- - NEO4J_dbms_security_auth__enabled=true
- ports:
- - "7688:7687" # Bolt (chat Neo4j on different port)
- - "7475:7474" # HTTP (chat Neo4j Browser)
- volumes:
- - ${CHAT_NEO4J_HOST_DATA_DIR:-./data/neo4j-chat/data}:/data
- - ${CHAT_NEO4J_HOST_LOGS_DIR:-./data/neo4j-chat/logs}:/logs
- healthcheck:
- test: ["CMD-SHELL", "PASS=$${NEO4J_AUTH#neo4j/}; cypher-shell -u neo4j -p \"$${PASS}\" -a bolt://localhost:7687 \"RETURN 1;\""]
- interval: 15s
- timeout: 5s
- retries: 10
- start_period: 20s
-
diff --git a/docs/API.md b/docs/API.md
index 62c77069..3f37934f 100644
--- a/docs/API.md
+++ b/docs/API.md
@@ -70,13 +70,10 @@ curl -H "Authorization: Bearer abc123..." \
### No Authentication (Development)
-Authentication is **off by default**. It is toggled at runtime via the settings API (there is no `SCIDK_AUTH_DISABLED` environment variable). To disable auth (e.g. for development or E2E tests):
+For development or testing, authentication can be disabled (not recommended for production):
```bash
-curl -X POST http://localhost:5000/api/settings/security/auth \
- -H "Content-Type: application/json" \
- -d '{"enabled": false}'
+export SCIDK_AUTH_DISABLED=true
```
-Set `{"enabled": true}` to require login. Current status: `GET /api/settings/security/auth`.
## Common API Operations
@@ -122,8 +119,7 @@ curl http://localhost:5000/api/health/graph
},
"relationships": {
"CONTAINS": 1334,
- "SCANNED_IN": 1245,
- "DERIVED_FROM": 27
+ "SCANNED_IN": 1245
}
}
```
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 57e6cd19..ab566f20 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -79,7 +79,7 @@ SciDK is a scientific data knowledge management system that bridges filesystem d
- Not ideal for high-concurrency writes (mitigated with WAL mode)
- No built-in graph queries (use Neo4j for this)
-**Graph Database**: Neo4j 5.x (Optional but active in production)
+**Graph Database**: Neo4j 5.x (Optional)
- **Why Neo4j**:
- Industry-leading graph database
- Cypher query language
@@ -119,43 +119,26 @@ SciDK is a scientific data knowledge management system that bridges filesystem d
### Web Layer
-**Blueprint Structure** (~27 blueprints, 300+ routes). Blueprints are registered via `register_blueprints()` in `scidk/web/routes/__init__.py`:
+**Blueprint Structure** (9 blueprints, 91+ routes):
```python
scidk/web/routes/
-├── ui.py # User interface (HTML) routes
-├── api_files.py # File, scan, and dataset operations
-├── api_graph.py # Graph schema, instances, RO-Crate export
-├── api_maps.py # Map/visualization data
+├── ui.py # User interface routes
+├── api_files.py # File and dataset operations
+├── api_graph.py # Graph queries and visualization
├── api_labels.py # Schema/label management
├── api_links.py # Link definitions and execution
-├── api_links_v2.py # Link registry (v2)
├── api_integrations.py # External API integrations
-├── api_neo4j.py # Neo4j connection and operations
-├── api_providers.py # Filesystem/rclone providers and mounts
-├── api_tasks.py # Background task management
-├── api_scripts.py # Analysis scripts
-├── api_results.py # Analysis result panels
-├── api_queries.py # Saved Cypher query library
-├── api_chat.py # Chat / GraphRAG interface
-├── api_annotations.py # Annotations
-├── api_plugins.py # Plugin management and instances
-├── api_interpreters.py # Interpreter configuration
├── api_settings.py # Settings and configuration
├── api_auth.py # Authentication endpoints
-├── api_users.py # User management
-├── api_audit.py # Audit log access
-├── api_alerts.py # Alert configuration
-├── api_logs.py # Application logs
-├── api_admin.py # Health / metrics / admin
-└── api_system.py # System info
+└── api_chat.py # Chat interface
```
**Advantages**:
- Clean separation of concerns
- Easy to add new features
- Improved testability
-- Lean application factory: `create_app()` lives in `scidk/app.py` (~314 lines); route definitions live in the per-area blueprint modules above
+- Reduced file size (app.py reduced from 5,781 to 645 lines)
### Core Services
@@ -336,74 +319,68 @@ User Pushes to Neo4j
### SQLite Tables
-**files** (see `scidk/core/path_index_sqlite.py`):
+**files**:
```sql
-CREATE TABLE IF NOT EXISTS files (
- path TEXT NOT NULL,
- parent_path TEXT,
- name TEXT NOT NULL,
- depth INTEGER NOT NULL,
- type TEXT NOT NULL,
- size INTEGER NOT NULL,
- modified_time REAL,
- file_extension TEXT,
- mime_type TEXT,
- etag TEXT,
- hash TEXT,
- remote TEXT,
+CREATE TABLE files (
+ id TEXT PRIMARY KEY,
scan_id TEXT,
- extra_json TEXT
+ path TEXT NOT NULL,
+ name TEXT,
+ size INTEGER,
+ modified REAL,
+ extension TEXT,
+ provider_id TEXT,
+ checksum TEXT,
+ FOREIGN KEY (scan_id) REFERENCES scans(id)
);
--- interpreted_as TEXT and interpretation_json TEXT are added via ALTER TABLE on init.
--- Node identity is the composite (path, host); there is no surrogate `id` column.
+CREATE INDEX idx_files_scan ON files(scan_id);
+CREATE INDEX idx_files_path ON files(path);
+CREATE INDEX idx_files_extension ON files(extension);
```
-**scans** (see `scidk/core/migrations.py`):
+**scans**:
```sql
-CREATE TABLE IF NOT EXISTS scans (
+CREATE TABLE scans (
id TEXT PRIMARY KEY,
- root TEXT,
- started REAL,
- completed REAL,
+ path TEXT NOT NULL,
+ recursive INTEGER,
+ timestamp REAL,
status TEXT,
- extra_json TEXT
+ file_count INTEGER,
+ provider_id TEXT
);
--- Per-scan detail (recursive flag, counts, provider, etc.) is stored in extra_json
--- and in companion tables (scan_items, scan_progress).
```
-**auth_users** (see `scidk/core/auth.py`):
+**users**:
```sql
-CREATE TABLE IF NOT EXISTS auth_users (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
+CREATE TABLE users (
+ id INTEGER PRIMARY KEY,
username TEXT UNIQUE NOT NULL,
password_hash TEXT NOT NULL,
- role TEXT NOT NULL CHECK (role IN ('admin', 'user')),
- enabled INTEGER DEFAULT 1,
- created_at REAL NOT NULL,
- updated_at REAL NOT NULL,
- created_by TEXT,
+ role TEXT NOT NULL,
+ created_at REAL,
last_login REAL
);
```
-**settings** (see `scidk/core/migrations.py`):
+**settings**:
```sql
-CREATE TABLE IF NOT EXISTS settings (
+CREATE TABLE settings (
key TEXT PRIMARY KEY,
- value TEXT
+ value TEXT,
+ updated_at TEXT
);
```
-**auth_audit_log** (see `scidk/core/auth.py`):
+**audit_log**:
```sql
-CREATE TABLE IF NOT EXISTS auth_audit_log (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
+CREATE TABLE audit_log (
+ id INTEGER PRIMARY KEY,
timestamp REAL NOT NULL,
- username TEXT NOT NULL,
- action TEXT NOT NULL,
- details TEXT,
- ip_address TEXT
+ event_type TEXT NOT NULL,
+ user TEXT,
+ ip_address TEXT,
+ details TEXT
);
```
@@ -415,13 +392,12 @@ CREATE TABLE IF NOT EXISTS auth_audit_log (
- **Scan**: Scan session metadata (timestamp, path, recursive)
- **Custom Labels**: User-defined via Labels page
-**Relationships** (see `scidk/services/neo4j_client.py`):
+**Relationships**:
- **(File)-[:SCANNED_IN]->(Scan)**: Files belong to scans
- **(Folder)-[:SCANNED_IN]->(Scan)**: Folders belong to scans
-- **(Folder)-[:CONTAINS]->(File)**: File hierarchy
-- **(Folder)-[:CONTAINS]->(Folder)**: Folder hierarchy
-- **(File)-[:INTERPRETED_AS]->(...)**: Interpretation results
-- **Custom Relationships**: User-defined via Links page (e.g. `DERIVED_FROM`)
+- **(File)-[:CONTAINED_IN]->(Folder)**: File hierarchy
+- **(Folder)-[:CONTAINED_IN]->(Folder)**: Folder hierarchy
+- **Custom Relationships**: User-defined via Links page
## Scalability Considerations
@@ -588,7 +564,7 @@ app.register_blueprint(custom_bp)
- High concurrent write load (>100 writes/sec)
- Distributed deployment required
-### Why Neo4j (Optional but active in production)?
+### Why Neo4j (Optional)?
**Advantages**:
- Native graph queries (relationships are first-class)
diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md
deleted file mode 100644
index 5472bb3d..00000000
--- a/docs/CONTRIBUTING.md
+++ /dev/null
@@ -1,85 +0,0 @@
-# Contributing to SciDK
-
-Conventions for common changes. For setup and how to run the app, see the README ("Run the server" — `scidk-serve` is canonical).
-
-## Add a new route
-
-Routes live in per-area blueprints under `scidk/web/routes/`. Reuse an existing module when one fits; otherwise add a new file following the same pattern:
-
-```python
-# scidk/web/routes/api_widgets.py
-from flask import Blueprint, jsonify
-bp = Blueprint("api_widgets", __name__, url_prefix="/api")
-
-@bp.get("/widgets")
-def list_widgets():
- return jsonify([])
-```
-
-Then register it in `scidk/web/routes/__init__.py` inside `register_blueprints()`:
-
-```python
-from . import api_widgets
-app.register_blueprint(api_widgets.bp)
-```
-
-Add `data-testid` attributes to any new interactive UI elements.
-
-## Persist settings
-
-Use the helpers in `scidk/core/settings.py` — do not write the `settings` table directly:
-
-```python
-from scidk.core.settings import get_setting, set_setting
-set_setting("my_key", "value")
-val = get_setting("my_key", default=None)
-```
-
-## Query Neo4j
-
-Always go through `scidk/services/neo4j_client.py`; never instantiate `neo4j.GraphDatabase` drivers directly:
-
-```python
-from scidk.services.neo4j_client import get_neo4j_client
-client = get_neo4j_client()
-rows = client.execute_read("MATCH (n:File) RETURN count(n) AS c")
-client.execute_write("MERGE (:Tag {name: $name})", {"name": "demo"})
-```
-
-This centralizes connection params, auth modes, and profiles.
-
-## Add a migration
-
-Schema lives in `scidk/core/migrations.py` and auto-runs on boot. Append a new versioned block at the end of `migrate()` — never edit past blocks:
-
-```python
-if version < 26:
- cur.execute("CREATE TABLE IF NOT EXISTS my_table (...);")
- conn.commit()
- _set_version(conn, 26)
- version = 26
-```
-
-`files`/index schema lives in `scidk/core/path_index_sqlite.py`.
-
-## Logging
-
-Use a module-level logger; do not use `print()`:
-
-```python
-import logging
-logger = logging.getLogger(__name__)
-logger.info("scan started: %s", path)
-```
-
-## Run tests
-
-```bash
-pytest tests/ -q # unit + integration (what CI runs)
-```
-
-E2E (Playwright) is **local only** — disabled in CI as of Feb 2026:
-
-```bash
-npm run e2e # or: pytest -m e2e
-```
diff --git a/docs/DEMO_SETUP.md b/docs/DEMO_SETUP.md
index 873bdbf1..13996b1b 100644
--- a/docs/DEMO_SETUP.md
+++ b/docs/DEMO_SETUP.md
@@ -119,8 +119,8 @@ Use this when setting up a new demo instance:
# Clean everything and start fresh
python scripts/seed_demo_data.py --reset --neo4j
-# Start SciDK (canonical launch command — see README "Run the server")
-scidk-serve
+# Start SciDK
+python start.sh
# Login as admin / demo123
```
@@ -326,10 +326,10 @@ Demo files follow a consistent structure:
## See Also
-- [Security & Authentication](SECURITY.md)
-- [Plugin System](plugins.md)
+- [Authentication Documentation](AUTHENTICATION.md)
+- [Plugin System](plugins/README.md)
- [iLab Importer Plugin](plugins/ILAB_IMPORTER.md)
-- [Architecture (graph & Neo4j integration)](ARCHITECTURE.md)
+- [Neo4j Integration](GRAPH_INTEGRATION.md)
## Support
diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md
index c0f9938b..ac047982 100644
--- a/docs/DEPLOYMENT.md
+++ b/docs/DEPLOYMENT.md
@@ -7,14 +7,14 @@ This guide covers production deployment of SciDK, including installation, config
### System Requirements
- **OS**: Linux (Ubuntu 20.04+, RHEL 8+, or compatible), macOS 11+, or Windows 10+ with WSL2
-- **Python**: 3.12 or higher (required; see `pyproject.toml` `requires-python = ">=3.12"`)
+- **Python**: 3.10 or higher
- **Memory**: Minimum 2GB RAM, 4GB+ recommended for large datasets
- **Disk**: 10GB+ free space for application and data storage
- **Neo4j** (optional): 5.x or higher for graph database functionality
### Required Software
-1. **Python 3.12+** with pip and venv
+1. **Python 3.10+** with pip and venv
2. **Neo4j** (optional but recommended): For persistent graph storage
3. **rclone** (optional): For cloud storage provider integration
4. **ncdu or gdu** (optional): For faster filesystem scanning
diff --git a/docs/SECURITY.md b/docs/SECURITY.md
index 561fc1ca..62bb18e5 100644
--- a/docs/SECURITY.md
+++ b/docs/SECURITY.md
@@ -2,12 +2,6 @@
This guide covers the security architecture, best practices, compliance considerations, and incident response procedures for SciDK deployments.
-> **⚠️ Status: implemented vs. recommended.** This guide documents both controls that are **implemented today** and controls that are **recommended / not yet implemented**. Treat unmarked best-practice snippets (nginx, OS, compliance, monitoring) as deployment *recommendations*, not descriptions of current behavior. For the current security posture of the running app — including what is safe for single-user development vs. multi-user production — see [SECURITY_HARDENING.md](SECURITY_HARDENING.md).
->
-> **Implemented today:** session-based login with bcrypt password hashing (`scidk/core/auth.py`), role-based access control with `@require_role`/`@require_admin` (`scidk/web/decorators.py`), the `auth_users` / `auth_audit_log` tables, audit logging, and Fernet-encrypted credentials in AlertManager/ConfigManager/API-endpoint registry and plugin settings (`scidk/core/plugin_settings.py`).
->
-> **Recommended / not yet implemented:** `SESSION_COOKIE_SECURE` / `SESSION_COOKIE_HTTPONLY` / `SESSION_COOKIE_SAMESITE` config, and CSRF protection (see inline notes below).
-
## Security Architecture Overview
SciDK implements defense-in-depth security with multiple layers of protection:
@@ -32,11 +26,11 @@ SciDK supports session-based authentication with the following features:
- Secure password reset mechanisms
**Session Management**:
-- Session-based authentication
+- Session-based authentication using secure cookies
- Configurable session timeout (default: 30 minutes)
- Auto-lock after inactivity
- Session invalidation on logout
-- CSRF protection and secure-cookie flags (`SESSION_COOKIE_SECURE`/`HTTPONLY`/`SAMESITE`) ⚠️ *recommended / not yet implemented*
+- CSRF protection enabled
**Example: Enabling Authentication**:
```python
@@ -293,7 +287,7 @@ chmod 600 .env
```
**Credential Storage**:
-- SciDK stores encrypted credentials in SQLite. Fernet (symmetric) encryption is used for SMTP/alert credentials, config-manager secrets, API-endpoint auth tokens, and plugin settings (`scidk/core/plugin_settings.py`).
+- SciDK stores encrypted credentials in SQLite
- Encryption key should be stored separately
- Consider using external secret managers (HashiCorp Vault, AWS Secrets Manager)
@@ -323,9 +317,9 @@ SciDK implements input validation to prevent:
### Session Security
-**Configuration** ⚠️ *Recommended / not yet implemented* — the app does not currently set these cookie flags or enable CSRF protection; the snippet below is the recommended hardening to apply before multi-user production use:
+**Configuration**:
```python
-# Flask session configuration (RECOMMENDED — not currently set in code)
+# Flask session configuration
app.config.update(
SESSION_COOKIE_SECURE=True, # HTTPS only
SESSION_COOKIE_HTTPONLY=True, # No JavaScript access
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index 64fca27c..49a63305 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -612,7 +612,7 @@ htop
hashed = bcrypt.hashpw(password, bcrypt.gensalt())
conn = sqlite3.connect('/path/to/files.db')
- conn.execute("UPDATE auth_users SET password_hash=? WHERE username='admin'", (hashed,))
+ conn.execute("UPDATE users SET password_hash=? WHERE username='admin'", (hashed,))
conn.commit()
```
diff --git a/docs/branching-and-ci.md b/docs/branching-and-ci.md
index 38b1758e..c483c6e0 100644
--- a/docs/branching-and-ci.md
+++ b/docs/branching-and-ci.md
@@ -28,8 +28,8 @@ Goal: Keep the development flow simple and reliable by working on one active bra
- [ ] Justification added if working on multiple branches concurrently.
## CI Expectations
-- Unit/integration tests run on every PR via `.github/workflows/ci.yml` (`pytest -m "not e2e"`).
-- **E2E is disabled in CI as of Feb 2026** (the E2E job is commented out). Run E2E locally with `npm run e2e` or `pytest -m e2e`; do not let E2E block PRs.
+- Unit tests and smoke checks run on every PR.
+- E2E smoke (where applicable) runs within a few minutes (<5s/spec target).
- Required checks must be green before merge.
- Dev submodule freshness: PRs to main must keep dev/ submodule at the latest commit of its configured branch (see .gitmodules). A CI check enforces this, and main auto-syncs dev/ after merge.
diff --git a/docs/mcp-setup.md b/docs/mcp-setup.md
deleted file mode 100644
index 80e3bc85..00000000
--- a/docs/mcp-setup.md
+++ /dev/null
@@ -1,272 +0,0 @@
-# SciDK MCP Server Setup
-
-The SciDK MCP (Model Context Protocol) server exposes core SciDK functionality to external AI agents like Claude Desktop.
-
-## Features
-
-The MCP server provides 5 core tools:
-
-1. **`query_knowledge_graph`** — Execute safe read-only Cypher queries against the Neo4j knowledge graph
-2. **`get_schema`** — Retrieve the current database schema (labels, relationships, properties)
-3. **`summarize_dataset`** — Generate statistical summaries of labels/relationships
-4. **`get_label_profile`** — Get detailed Schema Intelligence profiles for specific labels
-5. **`list_labels`** — List all labels with node counts
-
-All tools return structured JSON responses and enforce read-only safety (no CREATE/MERGE/DELETE operations).
-
-## Installation
-
-### 1. Install MCP SDK
-
-```bash
-pip install mcp>=1.0.0
-```
-
-This is already included in `requirements.txt`.
-
-### 2. Configure Claude Desktop
-
-Add the following to your Claude Desktop configuration file:
-
-**macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
-**Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
-**Linux**: `~/.config/Claude/claude_desktop_config.json`
-
-```json
-{
- "mcpServers": {
- "scidk": {
- "command": "python3",
- "args": ["-m", "scidk.mcp_server"],
- "cwd": "/path/to/scidk",
- "env": {
- "NEO4J_URI": "bolt://localhost:7687",
- "NEO4J_USER": "neo4j",
- "NEO4J_PASSWORD": "neo4jiscool",
- "NEO4J_DATABASE": "neo4j"
- }
- }
- }
-}
-```
-
-**Important**: Replace `/path/to/scidk` with the absolute path to your SciDK installation.
-
-### 3. Restart Claude Desktop
-
-After updating the config, restart Claude Desktop completely. The MCP server will start automatically when you open a new conversation.
-
-## Testing
-
-### Manual Test
-
-You can test the MCP server directly from the command line:
-
-```bash
-# Start the server (it uses stdio transport)
-python3 -m scidk.mcp_server
-```
-
-The server will connect to Neo4j and wait for MCP protocol messages on stdin.
-
-### Test in Claude Desktop
-
-Once configured, open Claude Desktop and try these queries:
-
-1. **List available labels:**
- ```
- Use the list_labels tool to show me all labels in the database
- ```
-
-2. **Get schema:**
- ```
- Use get_schema to show me the database structure
- ```
-
-3. **Query the graph:**
- ```
- Use query_knowledge_graph to run: MATCH (f:File) RETURN f.name LIMIT 5
- ```
-
-4. **Get label profile:**
- ```
- Use get_label_profile to analyze the File label
- ```
-
-## Tool Usage Examples
-
-### 1. query_knowledge_graph
-
-Execute a read-only Cypher query:
-
-```json
-{
- "cypher": "MATCH (f:File) WHERE f.mime_type CONTAINS 'image' RETURN f.name, f.path LIMIT 10",
- "limit": 10
-}
-```
-
-Returns:
-```json
-{
- "status": "success",
- "rows": [
- {"f.name": "image1.png", "f.path": "/data/images/image1.png"},
- ...
- ],
- "row_count": 10,
- "error": null
-}
-```
-
-### 2. get_schema
-
-Retrieve database schema:
-
-```json
-{
- "max_labels": 50,
- "max_props_per_label": 5
-}
-```
-
-Returns:
-```json
-{
- "status": "success",
- "schema": {
- "labels": ["File", "Folder", "Scan", "Sample", "SampleType"],
- "relationships": ["CONTAINS", "DERIVED_FROM", "SCANNED_IN"],
- "properties": {
- "File": ["name", "path", "mime_type", "size", "created"],
- "Folder": ["name", "path", "host", "host_id"]
- },
- "label_counts": {
- "File": 1523,
- "Folder": 342,
- ...
- }
- },
- "error": null
-}
-```
-
-### 3. list_labels
-
-Get all labels with counts:
-
-```json
-{}
-```
-
-Returns:
-```json
-{
- "status": "success",
- "labels": [
- {"name": "File", "count": 1523},
- {"name": "Folder", "count": 342},
- {"name": "Scan", "count": 89},
- {"name": "Sample", "count": 45},
- {"name": "SampleType", "count": 12}
- ],
- "error": null
-}
-```
-
-### 4. get_label_profile
-
-Get detailed profile for a label:
-
-```json
-{
- "label": "File"
-}
-```
-
-Returns:
-```json
-{
- "status": "success",
- "profile": {
- "label": "File",
- "node_count": 1523,
- "properties": [
- {"name": "name", "frequency": 1523},
- {"name": "path", "frequency": 1523},
- {"name": "mime_type", "frequency": 1521},
- {"name": "size", "frequency": 1520},
- {"name": "created", "frequency": 1518}
- ],
- "relationships": [
- {"type": "CONTAINS", "target": "Folder", "frequency": 1523},
- {"type": "DERIVED_FROM", "target": "Sample", "frequency": 342}
- ]
- },
- "error": null
-}
-```
-
-### 5. summarize_dataset
-
-Generate statistical summary:
-
-```json
-{
- "label": "File"
-}
-```
-
-Returns:
-```json
-{
- "status": "success",
- "summary": "Dataset summary: {'node_count': 1523, 'label': 'File'}",
- "error": null
-}
-```
-
-## Safety Features
-
-The MCP server enforces strict safety rules:
-
-- **Read-only queries**: Blocks CREATE, MERGE, DELETE, SET, DROP, DETACH keywords
-- **Automatic LIMIT**: Adds `LIMIT 50` to queries without explicit limits
-- **Error handling**: Returns structured error messages instead of crashing
-- **Connection pooling**: Uses Neo4j driver connection pooling for efficiency
-
-## Troubleshooting
-
-### MCP server not appearing in Claude Desktop
-
-1. Check the Claude Desktop config file path
-2. Verify JSON syntax is valid (use a JSON validator)
-3. Check that the `cwd` path is correct
-4. Restart Claude Desktop completely (quit and reopen)
-
-### Connection errors
-
-1. Verify Neo4j is running: `docker ps | grep neo4j`
-2. Test connection manually: `python3 -m scidk.mcp_server`
-3. Check environment variables in config
-4. Verify Neo4j credentials are correct
-
-### Tools not working
-
-1. Check the Claude Desktop logs (varies by platform)
-2. Verify the MCP server can import scidk modules
-3. Test tools directly via Python:
- ```python
- from scidk.ai import mcp_tools
- from neo4j import GraphDatabase
-
- driver = GraphDatabase.driver("bolt://localhost:7687", auth=("neo4j", "neo4jiscool"))
- result = mcp_tools.list_labels(driver, "neo4j")
- print(result)
- ```
-
-## Next Steps
-
-Once the MCP server is working, these tools can be integrated into the Concept Graph Phase 3 as `:Concept_Tool` nodes. The Concept Graph will then route user intents to MCP tools alongside native SciDK tools, creating a unified planning layer.
-
-See the [SciDK Architecture Vision](SciDK_Architecture_Vision.md) for the full integration plan.
diff --git a/docs/setup_rclone.md b/docs/setup_rclone.md
deleted file mode 100644
index 49de94ac..00000000
--- a/docs/setup_rclone.md
+++ /dev/null
@@ -1,143 +0,0 @@
-# Rclone Setup
-
-Ops reference for configuring rclone remotes used by a SciDK instance. For UI/API usage of the rclone provider see [`docs/rclone/quickstart.md`](rclone/quickstart.md); for FUSE mounts see [`docs/rclone/mount-examples.md`](rclone/mount-examples.md).
-
-## Install rclone
-
-Install from the official package (do not vendor the binary). See .
-
-```bash
-# Linux
-curl https://rclone.org/install.sh | sudo bash
-# macOS
-brew install rclone
-
-rclone version
-```
-
-SciDK shells out to whatever `rclone` is on `PATH` (`shutil.which('rclone')`). It must be installed on the same host as the SciDK process — not just inside a container the app cannot reach.
-
-## How SciDK uses rclone
-
-`RcloneProvider` (`scidk/core/providers.py`) drives the `rclone` CLI directly:
-
-- **Discover remotes:** `rclone listremotes` → populates the provider roots (each line like `dropbox:`).
-- **Browse / scan:** `rclone lsjson ` → JSON entries (`Name`, `Path`, `Size`, `IsDir`) consumed by the scan loop.
-- **Fetch content:** `rclone cat ` via `provider.cat()` / `provider.open()`.
-
-Any remote you create with `rclone config` is therefore visible to SciDK automatically — there is no separate SciDK-side remote registry. Enable the provider with:
-
-```bash
-export SCIDK_PROVIDERS=local_fs,mounted_fs,rclone
-```
-
-Verify a remote is reachable before scanning:
-
-```bash
-rclone listremotes # SciDK sees exactly these
-rclone lsjson : # what a scan will enumerate
-```
-
-## Configuring remotes
-
-Run `rclone config` and follow the interactive prompts. Below are the three remote types used in this deployment. OAuth backends (SharePoint, Dropbox) need a browser for the initial token grant — if configuring on a headless server, run `rclone authorize` on a machine with a browser and paste the token, or use `rclone config` over an SSH tunnel.
-
-### SharePoint (Microsoft OneDrive provider)
-
-SharePoint document libraries are accessed through rclone's `onedrive` backend.
-
-```bash
-rclone config
-# n) New remote
-# name> sharepoint
-# Storage> onedrive
-# Leave client_id / client_secret blank to use rclone's defaults
-# region> 1 (Microsoft Cloud Global)
-# Use auto config? > y (opens browser for OAuth)
-# Then choose the site type:
-# - "SharePoint site" / "Search for a SharePoint site"
-# - enter the site name or URL, pick the document library (drive)
-```
-
-Confirm:
-
-```bash
-rclone lsd sharepoint:
-```
-
-### Dropbox
-
-```bash
-rclone config
-# n) New remote
-# name> dropbox
-# Storage> dropbox
-# Leave client_id / client_secret blank
-# Use auto config? > y (opens browser for OAuth)
-```
-
-Confirm:
-
-```bash
-rclone lsd dropbox:
-```
-
-### SFTP or mounted path (e.g. `/mnt/server`)
-
-For a local mount (NFS/CIFS already mounted by the OS, like the BMC-Lab6 archive at `/mnt/server`), you do **not** need an rclone remote at all — use SciDK's `mounted_fs` provider and point a scan at the path directly.
-
-To reach a server over SSH instead, configure an SFTP remote:
-
-```bash
-rclone config
-# n) New remote
-# name> server
-# Storage> sftp
-# host> server.example.mit.edu
-# user>
-# port> 22
-# Auth: key_file> /home/scidk/.ssh/id_ed25519 (preferred)
-# or set a password
-```
-
-Confirm:
-
-```bash
-rclone lsd server:/path/to/data
-```
-
-## Reconnecting an expired OAuth token
-
-OAuth backends (SharePoint, Dropbox) hold refresh tokens that can expire or be revoked. Scans then fail with auth errors. Re-grant access without recreating the remote:
-
-```bash
-rclone config reconnect sharepoint:
-rclone config reconnect dropbox:
-```
-
-This reopens the browser OAuth flow and rewrites the token in place. Verify afterward with `rclone lsd :`.
-
-## Troubleshooting
-
-Diagnose any failure by reproducing the exact call SciDK makes, with verbose logging:
-
-```bash
-rclone lsjson : -vv
-```
-
-| Symptom | Likely cause | Fix |
-|---|---|---|
-| `rclone not installed or not on PATH` from SciDK | `rclone` not visible to the app's user/environment | Install rclone for the SciDK service user; confirm `which rclone` |
-| Remote missing from SciDK provider roots | Not in `rclone listremotes`, or `rclone` is a different config/user | Check `rclone config file` location; run `rclone listremotes` as the SciDK user |
-| `401`/`403` / `token expired` / `invalid_grant` | OAuth token expired or revoked | `rclone config reconnect :` |
-| `directory not found` / empty `lsjson` | Wrong path, or backend has no folder placeholders | Verify with `rclone lsd :`; check casing and the document-library name |
-| Hang or timeout on large dirs | Backend slow to enumerate recursively | Scan with a bounded depth, or try `--fast-list` (provider retries without it on failure) |
-| `couldn't connect SSH` | SFTP host/key/firewall | Test `ssh user@host`; confirm `key_file` path and permissions (`600`) |
-
-Useful checks:
-
-```bash
-rclone config file # which config is in effect
-rclone config show # inspect a remote's settings (redacts secrets)
-rclone about : # quota / connectivity sanity check
-```
diff --git a/docs/testing.md b/docs/testing.md
index ddf9ca39..a81e9dd9 100644
--- a/docs/testing.md
+++ b/docs/testing.md
@@ -166,7 +166,7 @@ Many features integrate with tools/services such as rclone and Neo4j. The test s
- `SCIDK_PROVIDERS`: Feature-flag providers set (e.g., `local_fs,mounted_fs,rclone`)
- `NEO4J_URI` / `NEO4J_USER` / `NEO4J_PASSWORD` / `NEO4J_AUTH`: Used to steer code paths; tests often set `NEO4J_AUTH=none` with a fake neo4j module
- `SCIDK_RCLONE_MOUNTS` or `SCIDK_FEATURE_RCLONE_MOUNTS`: Enables rclone mount manager endpoints (tests mock subprocess)
-- `SCIDK_E2E`: Set to `1` to enable E2E tests in local runs (E2E is disabled in CI as of Feb 2026)
+- `SCIDK_E2E`: Set to `1` to enable E2E tests in local runs (automatically set in CI)
## Running Subsets and Debugging
@@ -228,16 +228,20 @@ A GitHub Actions workflow is provided at `.github/workflows/ci.yml`
- Fast feedback on API/unit/contract tests
**E2E smoke (Playwright):**
-- ⚠️ **Disabled in CI as of Feb 2026.** The E2E job is commented out in `.github/workflows/ci.yml`; CI runs only `pytest -m "not e2e"`. Continue writing E2E specs and run them **locally** — don't let E2E block PRs.
-- When re-enabled, the job sets up Python 3.12 + Node 18, installs Playwright browsers (`npx playwright install --with-deps`), runs `npm run e2e` with `SCIDK_PROVIDERS=local_fs`.
+- Sets up Python 3.12 (for Flask app)
+- Sets up Node 18
+- Installs deps and Playwright browsers (`npx playwright install --with-deps`)
+- Runs `npm run e2e`
+- Environment: `SCIDK_PROVIDERS=local_fs` to avoid external dependencies
+- Strict (no continue-on-error) now that smoke and core flows are stable
### Running Locally (CI-equivalent)
```bash
-# Python tests (this is what CI runs)
+# Python tests
python -m pytest -q -m "not e2e"
-# E2E tests (local only — not run in CI)
+# E2E tests
npm install
npx playwright install --with-deps
npm run e2e
@@ -320,10 +324,10 @@ npm run e2e:headed # optional, debug mode
### CI Integration
-⚠️ **E2E tests are disabled in CI as of Feb 2026** — run them locally with `npm run e2e` or `pytest -m e2e`. The E2E job in `.github/workflows/ci.yml` is commented out; CI runs only `pytest -m "not e2e"`. When the job is re-enabled it will:
+E2E tests run automatically in GitHub Actions on every push and PR. See `.github/workflows/ci.yml`:
-- **Job: `e2e`**: Run Playwright tests with `SCIDK_PROVIDERS=local_fs`
-- **On failure**: Upload Playwright report and traces as artifacts
+- **Job: `e2e`**: Runs Playwright tests with `SCIDK_PROVIDERS=local_fs`
+- **On failure**: Uploads Playwright report and traces as artifacts
- **Access artifacts**: Go to Actions → failed run → download `playwright-report`
To view traces locally:
diff --git a/pyproject.toml b/pyproject.toml
index 9d1132b0..01d99338 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -24,9 +24,6 @@ dependencies = [
"bcrypt>=4.0",
"APScheduler>=3.10",
"flasgger>=0.9.7",
- "watchdog>=3.0",
- "mcp>=1.0.0",
- "python-dotenv>=1.0.0",
]
[project.optional-dependencies]
diff --git a/pytest_fully_output.txt b/pytest_fully_output.txt
deleted file mode 100644
index f0c4c306..00000000
--- a/pytest_fully_output.txt
+++ /dev/null
@@ -1,433 +0,0 @@
-============================= test session starts ==============================
-platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.6.0
-rootdir: /home/patch/PycharmProjects/scidk
-configfile: pyproject.toml
-plugins: cov-7.0.0, playwright-0.4.3, base-url-2.1.0
-collected 927 items
-
-tests/test_alert_manager.py .............. [ 1%]
-tests/test_alerts_api.py ............ [ 2%]
-tests/test_annotations_and_selections.py .. [ 3%]
-tests/test_annotations_relationships_syncqueue.py . [ 3%]
-tests/test_annotations_rest_endpoints.py ........... [ 4%]
-tests/test_api_endpoint_registry.py ................... [ 6%]
-tests/test_app_api.py .... [ 6%]
-tests/test_arrows_import_export.py .......... [ 7%]
-tests/test_auth.py ...................... [ 10%]
-tests/test_auth_multiuser.py ........................... [ 13%]
-tests/test_auto_lock.py ............ [ 14%]
-tests/test_backup_automation.py ............. [ 15%]
-tests/test_bioformats_converter.py ........... [ 17%]
-tests/test_bioformats_interpreters.py ................ [ 18%]
-tests/test_bruker_microct_dataset.py .... [ 19%]
-tests/test_bruker_skyscan_interpreter.py .. [ 19%]
-tests/test_chat_api.py ....................................... [ 23%]
-tests/test_commit_index_chain.py . [ 23%]
-tests/test_commit_neo4j_attempt_flag.py . [ 23%]
-tests/test_commit_verbose_and_health.py .. [ 24%]
-tests/test_config_export_import.py ................... [ 26%]
-tests/test_cross_database_transfer.py ............... [ 27%]
-tests/test_csv_interpreter.py .. [ 27%]
-tests/test_directories_api.py . [ 28%]
-tests/test_eda_interpreter.py ........... [ 29%]
-tests/test_error_and_timeout.py ... [ 29%]
-tests/test_files_page_e2e.py .s....Fsss. [ 30%]
-tests/test_filesystem.py ... [ 31%]
-tests/test_folder_config_precedence.py . [ 31%]
-tests/test_folder_hierarchy.py .... [ 31%]
-tests/test_fs_list_api.py . [ 31%]
-tests/test_fuzzy_matching.py ................. [ 33%]
-tests/test_graph.py .. [ 33%]
-tests/test_graph_combined_schema_api.py ......... [ 34%]
-tests/test_graph_query_api.py ..s..... [ 35%]
-tests/test_graph_schema_api.py .. [ 35%]
-tests/test_graph_subschema_api.py .. [ 36%]
-tests/test_graphrag_endpoints.py ... [ 36%]
-tests/test_graphrag_entity_extractor.py ..... [ 36%]
-tests/test_graphrag_errors.py .. [ 37%]
-tests/test_graphrag_feedback.py ............. [ 38%]
-tests/test_graphrag_observability.py . [ 38%]
-tests/test_graphrag_query_engine.py ...... [ 39%]
-tests/test_graphrag_utilities.py .. [ 39%]
-tests/test_health_comprehensive.py ............... [ 41%]
-tests/test_health_sqlite.py . [ 41%]
-tests/test_helpers_example.py .... [ 41%]
-tests/test_ilab_plugin.py ........... [ 42%]
-tests/test_instances_api.py .. [ 43%]
-tests/test_interpreters_effective_and_scan_config.py .. [ 43%]
-tests/test_interpreters_page.py . [ 43%]
-tests/test_interpreters_registry_api.py ..... [ 43%]
-tests/test_ipynb_interpreter.py ...... [ 44%]
-tests/test_ipynb_ui_render.py . [ 44%]
-tests/test_label_endpoint_registry.py ............ [ 45%]
-tests/test_label_instance_count.py ..F [ 46%]
-tests/test_labels_api.py ................................... [ 50%]
-tests/test_link_execution_progress.py ... [ 50%]
-tests/test_links_api.py .........................FF... [ 53%]
-tests/test_links_integration.py .s.....s. [ 54%]
-tests/test_links_page.py ..... [ 55%]
-tests/test_links_triple_import.py .............. [ 56%]
-tests/test_logs_api.py .......... [ 57%]
-tests/test_logs_endpoint.py ..... [ 58%]
-tests/test_map_route.py . [ 58%]
-tests/test_maps_features.py ................... [ 60%]
-tests/test_metrics_endpoint.py . [ 60%]
-tests/test_neo4j_adapter.py .... [ 60%]
-tests/test_neo4j_commit.py . [ 61%]
-tests/test_neo4j_commit_flow.py . [ 61%]
-tests/test_neo4j_commit_folders.py . [ 61%]
-tests/test_neo4j_password_save.py .. [ 61%]
-tests/test_neo4j_settings_api.py .. [ 61%]
-tests/test_plugin_endpoint_integration.py ......... [ 62%]
-tests/test_plugin_instance_manager.py .......... [ 63%]
-tests/test_plugin_label_publishing.py ......... [ 64%]
-tests/test_plugin_loader.py ........... [ 65%]
-tests/test_plugin_settings.py .............. [ 67%]
-tests/test_plugin_settings_api.py .......... [ 68%]
-tests/test_plugin_template_registry.py ............ [ 69%]
-tests/test_plugins_api.py ..... [ 70%]
-tests/test_progress_indicators.py ... [ 70%]
-tests/test_property_removal_and_overwrite.py ......... [ 71%]
-tests/test_providers_api.py ......... [ 72%]
-tests/test_python_interpreter.py .. [ 72%]
-tests/test_rclone_provider.py ... [ 73%]
-tests/test_rclone_recursive_hierarchy.py . [ 73%]
-tests/test_rclone_scan.py . [ 73%]
-tests/test_rclone_scan_ingest.py . [ 73%]
-tests/test_registry.py .. [ 73%]
-tests/test_rescan_idempotency.py . [ 73%]
-tests/test_rocrate_export.py .. [ 74%]
-tests/test_rocrate_referenced.py .. [ 74%]
-tests/test_saved_maps_service.py .................. [ 76%]
-tests/test_scan_browse_indexed.py . [ 76%]
-tests/test_scan_commit_delete.py .. [ 76%]
-tests/test_scan_dry_run.py . [ 76%]
-tests/test_scan_fs_auto_enter_base.py . [ 76%]
-tests/test_scan_nonrecursive_includes_folders.py . [ 76%]
-tests/test_scan_sources.py ... [ 77%]
-tests/test_scans_committed_flag.py . [ 77%]
-tests/test_schema.py ............................. [ 80%]
-tests/test_schema_file_folder.py . [ 80%]
-tests/test_script_sandbox.py ............................... [ 83%]
-tests/test_scripts.py ...................... [ 86%]
-tests/test_search_api.py ... [ 86%]
-tests/test_seed_demo_data.py .............. [ 88%]
-tests/test_selective_scan_cache.py . [ 88%]
-tests/test_sqlite_migrations.py . [ 88%]
-tests/test_sqlite_path_index.py .. [ 88%]
-tests/test_state_backend_toggle.py .... [ 88%]
-tests/test_swagger_api_docs.py ........... [ 90%]
-tests/test_table_format_registry.py .................... [ 92%]
-tests/test_table_loader_plugin.py ................... [ 94%]
-tests/test_tasks_limits_cancel.py .. [ 94%]
-tests/test_tasks_scan.py . [ 94%]
-tests/test_transparency_analysis_scripts.py ssssssss [ 95%]
-tests/test_transparency_chat_tools.py ..... [ 96%]
-tests/test_transparency_dependencies.py .............. [ 97%]
-tests/test_transparency_file_interpretation.py ...... [ 98%]
-tests/test_transparency_plugins_page.py ......... [ 99%]
-tests/contracts/test_api_contracts.py ...... [ 99%]
-tests/e2e/test_demo_recording.py s [ 99%]
-tests/e2e/test_persistence.py s [100%]
-
-=================================== FAILURES ===================================
-________________________ test_no_synchronous_scan_in_ui ________________________
-
- def test_no_synchronous_scan_in_ui():
- """Verify that synchronous /api/scan is NOT used by the Files page UI."""
- from scidk.app import create_app
- app = create_app()
- app.config['TESTING'] = True
-
- with authenticate_test_client(app.test_client(), app) as client:
- resp = client.get('/datasets')
- html = resp.data.decode('utf-8')
-
- # Check that the JavaScript does NOT call /api/scan from provider panel
- # (it should only use /api/tasks)
-> assert "'/api/scan'" not in html or html.count("'/api/scan'") <= 1
-E assert ("'/api/scan'" not in '\n