diff --git a/CLAUDE.md b/CLAUDE.md index fde7257055..12de36d589 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,344 +1,74 @@ -# CLAUDE.md - -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. - -## Architecture Overview - -This is a Rust-based cloud storage microservices architecture built as a Cargo workspace with 80+ crates. The system -handles document storage, processing, search, communication, and email functionality. -When making changes, make sure to test the services individually before committing using `cargo test -p {my_service}` -from the repository root. - -### Key Services - -**Core Storage Services:** - -- `document-storage-service`: Main document storage API -- `document-cognition-service`: Document analysis and processing -- `search_service`: Search functionality across documents -- `static_file_service`: Static file serving - -**Processing Services:** - -- `convert_service`: Document format conversion -- `document-text-extractor`: Text extraction from documents -- `search_processing_service`: Search indexing and processing - -**Communication Services:** - -- `email_service`: Email processing and management -- `notification_service`: User notifications - -**Infrastructure Services:** - -- `authentication_service`: User authentication -- `connection_gateway`: WebSocket gateway -- `contacts_service`: Contact management - -### Data Storage - -The system uses multiple databases: - -- **MacroDB**: Main PostgreSQL database for documents, users, projects, Communication data (messages, channels, - participants), Email threads, messages, metadata, and notification preferences/history -- **ContactsDB**: User connections and contacts - -External storage includes S3 for document files, Redis for caching, OpenSearch for search indexing, and DynamoDB for -connection tracking. - -### MacroDB Schema Changes - -DB migration files are located in `crates/macro_db_client`. Use the `/dump-schema` skill to dump the current Postgres schema for reference. -If you are still getting migration errors after running `just setup_macrodb`, you may need to run `just force_drop_db` -in `crates/macro_db_client` to drop the database and re-create it -with `just setup_macrodb` at the repository root. Remember that some database table and column names may be -camelCased rather than snake_cased (use `/dump-schema` or check the migration files for actual column names). -When a column is camelCased, you need to cast it as the snake_cased version when reading from the database. E.g. -`SELECT "userId" as "user_id" FROM "UserInsights"`. -Any time you make changes to the SQL code in rust, you need to run `just prepare_db` to -update the `.sqlx` directory. Always run it inside `nix develop` and **only** from the -repository root (for example, `nix develop --command just prepare_db`) — do not run it from -individual crate directories anymore. The workspace-level recipe handles every crate that -has sqlx queries. - -## Development Commands - -### Alias Commands (IMPORTANT) - -Use `\cd` instead of `cd` to navigate in the repository. - -### Building - -```bash -just build # Build all services -just build_lambdas # Build all Lambda functions -just check # Type check without building -``` - -### Testing - -```bash -just create_networks -just run_dbs -d -just setup_test_envs -just initialize_dbs -cargo test -p {crate} -``` - -`just test` does not exist. Leave `SQLX_OFFLINE` unset when you run `cargo test`. Run `just prepare_db` only if you changed SQL queries. - -Email rendering snapshots (Playwright HTML fixtures, not inbox e2e) live in `apps/web/src/lib/core/email/tests`. Run `just test-email-rendering`. Add a fixture under `fixtures/` then `just test-email-rendering-update`. - -### Pre Commit -```bash -cargo fmt # format -just clippy # extra lints / best practices -``` - -### Database Management - -Use `just setup_macrodb` or `just initialize_dbs` to create and migrate MacroDB. Those recipes are the same. - -Schemas live in `crates/macro_db_client/migrations/`. - -To reset MacroDB, run `just crates/macro_db_client/drop_db -y -f`, then `just setup_macrodb`. - -### Lambda Building - -Individual lambda builds available for: - -- `build_document_text_extractor` -- `build_docx_unzip_handler` -- `build_delete_chat_handler` -- `build_upload_extractor_lambda_handler` -- `build_email_suppression` -- `build_deleted_item_poller` - -## Key Architectural Patterns - -### Service Communication - -Services communicate via: - -- HTTP APIs (internal service clients) -- SQS queues for async processing -- Lambda triggers for event-driven processing -- Redis for caching and session management - -### Database Architecture - -- Each service has its own database client crate (e.g., `macro_db_client`, `comms_db_client`) -- Uses SQLx for database interactions with offline query validation -- Migrations managed per service - -### AWS Integration - -Heavy use of AWS services: - -- S3 for file storage -- Lambda for serverless processing -- SQS for message queuing -- DynamoDB for connection tracking -- OpenSearch for search capabilities - -### -Environment variables are managed in doppler. New env vars should be added to doppler. All environment variables should -_always_ be loaded with the macros in the macro_env_var crate. They should never be loaded with std::env::var. - -## Development Notes - -### Prerequisites - -- Docker (for local databases) -- `sqlx-cli` for database migrations -- `just` for task running -- Pulumi CLI for infrastructure -- AWS CLI for deployment - -### Offline Development - -The project uses `SQLX_OFFLINE=true` for building without database connections. Database queries are pre-validated and -cached. - -### Document Processing Pipeline - -Documents go through: Upload → Text Extraction → Search Indexing → Storage → Retrieval - -- DOCX files are unzipped via Lambda -- PDFs processed with pdfium library -- Text indexed in OpenSearch -- Metadata stored in PostgreSQL - -## Case Study: Implementing Generic Entity Mentions - -This case study documents the process of extending message mentions to support generic entity mentions (e.g., documents -mentioning other documents). - -### Task Understanding & Planning - -1. **Analyzed Requirements**: Extended existing MessageMention functionality to support any entity mentioning any other - entity -2. **Created Todo List**: Used TodoWrite tool to track implementation steps -3. **Examined Existing Code**: Reviewed current message_mentions table structure and usage - -### Implementation Steps - -1. **Data Model Changes** - - Created `EntityMention` struct with generic source/target fields - - Maintained backward compatibility with existing mentions - -2. **Database Migration** - - Renamed `message_mentions` → `entity_mentions` - - Added `source_entity_type` and `source_entity_id` columns - - Migrated existing data (messages) to new structure - - Updated all indexes for performance - -3. **Updated Database Client** - - Created `entity_mentions` module with create/delete functions - - Modified `get_attachment_references` to query new table - - Updated `create_message_mentions` to insert into new table - - Fixed test fixtures to use new table structure - -4. **API Endpoints** - - Created POST/DELETE `/entity-mentions` endpoints - - Used proper Extension extractors for axum handlers - - Added OpenAPI documentation - -### Testing & Debugging - -1. **Compilation Issues** - - Fixed import errors (wrong Context type, missing http import) - - Added Clone trait to structs used in tests - - Updated fixture references from `message_mentions` to `mentions` - -2. **Test Failures** - - Fixed `create_message_mentions` test by updating query logic - - Query now returns all mentioned users, not just newly inserted ones - - Updated fixtures to include entity_mentions data - -3. **SQLX Offline Mode** - - Encountered "no cached data" errors due to schema changes - - Required running migrations before `cargo sqlx prepare` - -### Database Preparation - -1. Run migrations: `just migrate_db` -2. Update SQLX cache: `just prepare_db` -3. Verify with tests: `cargo test` - -### Key Learnings - -1. **Todo Management**: Proactive use of TodoWrite helps track complex multi-step tasks -2. **Incremental Testing**: Run tests frequently to catch issues early -3. **Fixture Management**: Update test fixtures when changing table structures -4. **SQLX Workflow**: Schema changes require migration → prepare → test cycle -5. **Axum Patterns**: Handlers take shared services via `State`, not `Extension` (see docs/STYLE_GUIDE.md CS-30; this case study predates that convention) - -### Index Strategy - -The migration included comprehensive indexes: - -- Composite index on (entity_type, entity_id) for efficient lookups -- Index on source columns for reverse lookups -- Index on created_at for ordering -- Maintained existing performance optimizations - -## Development Best Practices - -### Database Query Management - -- Prefer SQLx compile-time checked macros (`query!`, `query_as!`, `query_scalar!`) for database queries whenever possible instead of dynamic `sqlx::query` calls. -- Never manually create or edit `.sqlx/query-*.json` files. To update SQLx query metadata, run `just prepare_db` from the repository root. -- When creating a new SQLx migration file, run `sqlx migrate add ` from the relevant database crate (or use SQLx's `--source` option) and then edit the generated file. Never manually create migration files, and never invent, copy, or guess timestamp prefixes to fake a migration filename. -- Always run tests between changes that involve changes to db queries -- Never run `cargo test` with `SQLX_OFFLINE=true`. Tests are designed to validate against the live local Postgres; offline mode forces sqlx macros to consult the cached `.sqlx` data and can either surface confusing "type annotations needed" errors when a query was not in the cache or hide regressions where a query no longer matches the schema. If tests fail with sqlx "no cached data" errors, run `just prepare_db` (with `--tests` when the failure is in test code) — do not flip offline mode on. `SQLX_OFFLINE=true` is fine for `cargo check` / `cargo build` / `cargo clippy` only. - -### Rust Error Handling - -- New code uses `rootcause` for error handling — it's preferred over `anyhow` (see docs/STYLE_GUIDE.md CS-46) -- In code still on anyhow: prefer `anyhow::bail!("error message")` over `Err(anyhow::anyhow!("error message"))` for early returns - it's more concise and idiomatic - -### Agent Guide Maintenance - -`docs/AGENT_GUIDE/` documents how agents drive the web app through a browser (routes, UI affordances, interaction patterns, completion signals). When you change how a part of the app works or how users/agents interact with it — routes, creation flows, editor behavior, AI surfaces, composer semantics — update the corresponding guide file in the same change. - -### Documentation Requirements - -- Add `#![deny(missing_docs)]` to `lib.rs` in new crates to enforce documentation on all public items -- This ensures all public functions, structs, enums, and modules have documentation comments -- Do not use `ignore` to except code blocks from doc tests unless explicitely directed - - -### Test File Organization - -Place tests in a separate `test.rs` file within the same module directory, rather than inline with `#[cfg(test)]` blocks in the implementation file. - -**Pattern:** -- Implementation: `foo/mod.rs` or `foo.rs` -- Tests: `foo/test.rs` - -Note: You do NOT need to convert a file module (`foo.rs`) into a directory module (`foo/mod.rs`) to add tests. -Rust supports `foo.rs` alongside a `foo/` directory — just create `foo/test.rs` and it works as a submodule of `foo.rs`. - -**Example structure:** -``` -src/ - user.rs # Contains: mod test; (with #[cfg(test)]) + implementation - user/ - test.rs # Contains: use super::*; and test functions -``` - -**In `user.rs`:** -```rust -#[cfg(test)] -mod test; - -// implementation code... -``` - -**In `test.rs`:** -```rust -use super::*; - -#[tokio::test] -async fn test_something() { - // test code -} -``` - -This keeps implementation files focused and makes tests easier to locate and maintain. - -### Tracing - -- Include `err` when adding the `tracing::instrument` attribute to functions that return `Result`. Do not include `err` on functions that return `Option`, `()`, or other non-`Result` types. Never include `level = "info"`. -- When including an error with a log, include it like so: `tracing::error!(error=?e, "error msg");` -don't inject it directly into the error message. -- Prefer using `.inspect_err` instead of `if let Err(e)` in order to do logging. - -## Development Memories - -### DB Crate Changes - -- When making changes to a db crate you should always update tests, and run prepare - -## Cursor Cloud specific instructions - -These apply to Cursor Cloud only. On a local dev machine the `.cursor/*.sh` scripts prompt for sudo and are the wrong entry point: to verify a frontend change, run `PORT= bun run dev` from `apps/web` against the dev backend (see `apps/web/AGENTS.md`), and only reach for the local stack for backend work. - -`.cursor/install.sh` prepares the durable caches, databases, test dependencies, frontend dependencies, service binaries, and stack init snapshot, then stops dockerd and nix-daemon so the Cloud bake can exit. `.cursor/start.sh` runs at boot and starts nothing beyond the nix daemon, so sessions and subagents are usable immediately. `.cursor/infra.sh` starts Docker, Postgres, and Redis on demand. `.cursor/stack.sh` starts the on-demand product: backend containers behind the proxy (8090) plus the hot-reloading frontend dev server — the app is at http://localhost:3000/app and frontend edits apply on save (idempotent; a healthy stack is left alone). `.cursor/rebuild.sh` remounts new backend binaries after Rust edits. These scripts are the only supported entry points; each re-enters the pinned nix shell itself, so run them with plain `bash` from any environment. To run the app and see your edits, follow the `run-app` skill (`.claude/skills/run-app/SKILL.md`). - -Nix is the only host dependency. The pinned dev shell supplies the Docker CLI and daemon, Compose, `fuse-overlayfs`, and OpenSSH. `ssh-keygen` must come from this shell. - -Service binaries come from the private S3 Nix cache. A sibling workflow on push to main (`push_local_stack_binaries.yml`) builds and pushes `.#local-stack-binaries`. It is not part of the deploy pipeline. Cursor Cloud realizes the aggregate with `nix build .#local-stack-binaries`. - -Set `NIX_CACHE_AWS_ACCESS_KEY_ID` and `NIX_CACHE_AWS_SECRET_ACCESS_KEY` as Cursor environment secrets. The IAM credentials need read-only access to the cache bucket. - -Set `DOPPLER_TOKEN` as a Cursor environment **runtime** secret: a Doppler service token scoped to the `local` project's `lcl_preview` config (the same token CI stores as `DOPPLER_PREVIEW_TOKEN` also works). Do not paste the token into chat. When the token is present, `bash .cursor/stack.sh` pulls those secrets instead of passing `--no-doppler`. Install/bake stays on stubs so the snapshot does not embed secrets. Existing running agents do not pick up newly added secrets — start a new agent after adding the token. - -Nothing runs after boot. Before DB-backed `cargo test -p `, run `bash .cursor/infra.sh` once — it brings up Docker, Postgres, and Redis in seconds because install baked the images and volumes. Pure-logic crate tests need nothing. Run `bash .cursor/stack.sh` for a product-ready environment. - -After backend edits, run `bash .cursor/rebuild.sh`. That nix-builds the stack binaries and runs `just stack update --binaries-dir`, which remounts them without wiping volumes. - -No seeding or OTP is needed to log in: passwordless login auto-creates a user for any email, and the stack's auth service is built with `return_passwordless_code`, so the login API returns the code in its response (codes are also visible in Mailpit at http://localhost:8025). `just seed-scenario apply --file seed/scenarios/team-perms.json` is optional, for multi-user team/permission fixtures. The `agent_harness_service` restart loop is expected when AI provider keys are missing; with `DOPPLER_TOKEN` those keys come from `local`/`lcl_preview`. - -Leave `SQLX_OFFLINE` unset for `cargo test`. If SQLx reports missing cached query data, run `just prepare_db` instead of enabling offline mode. Run crate tests from the repository root with `cargo test -p `. +# Repository guide for coding agents + +Macro is a document and collaboration app with a Rust backend and a web frontend. +This file is the shared entry point: `AGENTS.md` symlinks to `CLAUDE.md`. Edit +`CLAUDE.md` and preserve the symlink; do not maintain two copies. + +## Start here + +1. Identify the affected app, service, or crate using the map below. +2. Read any applicable directory-level `AGENTS.md` / `CLAUDE.md` before editing. +3. Read the task-relevant guides below, not every linked document. +4. Follow [the style guide](docs/STYLE_GUIDE.md) for the language you change. + Prefer its current rules over patterns in older code. + +## Repository map + +| Path | Contents | +| --- | --- | +| `apps/web/` | Web frontend; its own agent guide covers Bun, UI patterns, and browser verification. | +| `apps/`, `packages/` | Other apps and shared frontend packages. | +| `services/` | Deployable services and workers; Rust package names come from each `Cargo.toml`. | +| `crates/` | Reusable backend libraries, domain services, and adapters. | +| `crates/macro_db_client/migrations/` | MacroDB PostgreSQL migrations. | +| `infra/` | Deployment definitions and local infrastructure. | +| `tooling/` | Development tools, scripts, and imported `just` recipes. | +| `docs/AGENT_GUIDE/` | How browser agents operate the app, not how to develop the repository. | + +## Read when relevant + +| Task | Guide | +| --- | --- | +| Rust code, builds, or tests | [Rust development](docs/RUST_DEVELOPMENT.md) | +| SQLx queries, migrations, DB tests, or cache errors | [Database development](docs/DATABASE_DEVELOPMENT.md) | +| Web frontend or email-rendering snapshots | [Web agent guide](apps/web/AGENTS.md) | +| Running the frontend or backend on a local machine | [Running locally](docs/RUNNING_LOCALLY.md) | +| Working inside Cursor Cloud | [Cursor Cloud](docs/CURSOR_CLOUD.md) | +| Driving the app through a browser | [App agent guide](docs/AGENT_GUIDE/README.md) | +| Deployment | [Infrastructure guide](infra/README.md) | + +## Essential guardrails + +- Run Rust tests from the repository root with `cargo test -p ` and + **leave `SQLX_OFFLINE` unset**. Offline mode is for checks/builds/lints, not tests. +- Generate migrations with `sqlx migrate add`; never invent timestamped filenames. + Never hand-edit `.sqlx/query-*.json`. Prepare the workspace cache from the root + inside Nix; see the database guide for the workflow and test-query flags. +- Do not reset databases or wipe stack volumes to troubleshoot without explicit + approval. Database and Cloud guides distinguish rebuilds from destructive resets. +- On a local machine, frontend-only work uses `apps/web` against the dev backend; + it does not need a local Rust stack. `.cursor/*.sh` scripts are **Cursor Cloud + only**, not local-machine setup commands. Treat hosted dev data as real data. +- Load Rust configuration through `macro_env_var` / `macro_config`, never + `std::env::var` or hand-rolled wrappers. Register new env vars in Doppler; never + paste secrets into chat or commit them. +- Use `\cd` instead of `cd` to bypass repository shell aliases. + +## Before handing off + +- Test the affected packages/services individually before committing. Use the + relevant guide for setup; there is no root `just test` recipe. +- For code changes, run `just check` (format/lint/code rules scoped to changes). + `just check full` adds TypeScript checking and Rust clippy; `just rust-check` is + the workspace Rust type check. These checks do not replace tests. +- Exercise user-visible changes in a browser. If routes, creation flows, editors, + AI surfaces, or composer behavior change, update the corresponding + [app agent guide](docs/AGENT_GUIDE/README.md) in the same change. +- Report what you tested and any checks blocked by the environment. + +## Keeping these instructions useful + +Keep this file short: shared guardrails and links with explicit reading triggers. +Put workflows in the relevant guide, coding rules in `docs/STYLE_GUIDE.md`, and +subtree-specific instructions near their code. Do not append implementation +journals, duplicate command lists, or environment-specific runbooks here. diff --git a/docs/CLOUD_STORAGE.md b/docs/CLOUD_STORAGE.md index 1d529dc687..e995c06d04 100644 --- a/docs/CLOUD_STORAGE.md +++ b/docs/CLOUD_STORAGE.md @@ -3,29 +3,15 @@ The Rust backend is split across deployable processes in `services/`, reusable libraries in `crates/`, and deployment definitions in `infra/`. -## Prerequisites - -- docker -- sqlx-cli -- just -- pulumi cli -- aws cli - -# Testing - -To run tests locally, run the following commands: - -```bash -just create_networks -just run_dbs -d -just setup_test_envs -just initialize_dbs -cargo test # NB: SQLX_OFFLINE should NOT be set -``` - -## clean up - -To reset the local database, use the repository-root database recipes rather than deleting unrelated containers: `just crates/macro_db_client/drop_db -y -f`, then `just setup_macrodb`. +## Development and testing + +- [Rust development](RUST_DEVELOPMENT.md): toolchain, build/check commands, and + tests for the affected packages. +- [Database development](DATABASE_DEVELOPMENT.md): local test databases, + migrations, SQLx cache preparation, and approved destructive resets. +- [Running locally](RUNNING_LOCALLY.md): frontend against hosted services or a + full local stack. +- [Cursor Cloud](CURSOR_CLOUD.md): Cloud-only setup and rebuild entry points. ## Deployment diff --git a/docs/CURSOR_CLOUD.md b/docs/CURSOR_CLOUD.md new file mode 100644 index 0000000000..912b98ea28 --- /dev/null +++ b/docs/CURSOR_CLOUD.md @@ -0,0 +1,82 @@ +# Cursor Cloud + +**Only for Cursor Cloud VMs.** On a local machine, `.cursor/*.sh` can prompt for +sudo and are the wrong entry points. Use [running locally](RUNNING_LOCALLY.md) +and the [web agent guide](../apps/web/AGENTS.md) instead. + +## Start only what the task needs + +Boot starts the Nix daemon, not Docker, databases, or the product. Pure-logic tests +need no infrastructure. Use the supported scripts from the repository root: + +| Task | Command | +| --- | --- | +| Start Postgres and Redis for DB-backed tests | `bash .cursor/infra.sh` | +| Start the product with a hot-reloading frontend | `bash .cursor/stack.sh` | +| Pick up backend Rust edits without wiping volumes | `bash .cursor/rebuild.sh` | +| Pick up frontend source edits | Save the file; Vite applies them on save. | + +The scripts re-enter the pinned Nix shell themselves; invoke them with plain +`bash`. For other commands, use the pinned shell, e.g. from the root: + +```bash +nix develop --command env -u SQLX_OFFLINE cargo test -p +``` + +Run tests per affected crate, with `SQLX_OFFLINE` unset. Installation baked the +initial database and test envs; apply new migrations as needed. Use +[database development](DATABASE_DEVELOPMENT.md) for SQLx preparation and errors. + +## Run and verify the app + +Follow the [run-app skill](../.claude/skills/run-app/SKILL.md) for the full browser +walkthrough, frontend restarts, and login troubleshooting. + +- App: ; backend proxy: . + These URLs are inside the VM, not reachable from the user's laptop. +- `stack.sh` leaves a healthy backend running. After Rust edits, use `rebuild.sh`: + it builds Nix binaries and remounts them via `just stack update --binaries-dir`, + preserving volumes. `cargo build` alone does not update running containers. +- Do not replace the scripts with `just run_local`, hand-rolled Compose, or + `just stack up`. **`stack.sh --fresh` deliberately wipes and recreates the + stack**; use it only with explicit approval, never to pick up code edits. +- Sign in with any email; passwordless login creates the user on demand. The + auth service is built with `return_passwordless_code`, so the login API returns + the code. Codes also appear in Mailpit at . +- Seeding is optional, not required for login. For multi-user permission fixtures, + use `just seed-scenario apply --file seed/scenarios/team-perms.json` inside Nix. +- An `agent_harness_service` restart loop is expected without AI provider keys; + see the runtime secrets section below rather than debugging it as a code regression. + +## Runtime secrets + +Set `DOPPLER_TOKEN` as a Cursor environment **runtime** secret: a Doppler service +token scoped to project `local`, config `lcl_preview`. The token stored in CI as +`DOPPLER_PREVIEW_TOKEN` also works. Never paste tokens into chat or commit them. + +With the token present, `stack.sh` pulls those secrets, including AI provider +keys. Without it, the stack uses `--no-doppler` stubs and real external +integrations are unavailable. Install/bake always uses stubs so the durable +snapshot does not embed secrets. + +Existing agents do not inherit newly added environment secrets; start a new +agent after adding the token. + +## Environment maintenance and binary cache + +This section is for Cloud environment setup, not ordinary feature work. + +- [`install.sh`](../.cursor/install.sh) prepares durable dependency caches, + databases, test dependencies, frontend dependencies, service binaries, and a + stack-init snapshot. It then stops dockerd and nix-daemon so the bake can exit. +- [`start.sh`](../.cursor/start.sh) runs at boot and only ensures the Nix daemon + and cache links, keeping sessions and subagents cheap to start. +- Nix is the only host dependency. The pinned shell provides Docker CLI/daemon, + Compose, `fuse-overlayfs`, and OpenSSH; use its `ssh-keygen`. +- Service binaries are built/cached as `.#local-stack-binaries`, realized with + `nix build .#local-stack-binaries`. The + [`push_local_stack_binaries.yml`](../.github/workflows/push_local_stack_binaries.yml) + workflow pushes them to the private S3 Nix cache on main; it is separate from + the deploy pipeline. +- Set `NIX_CACHE_AWS_ACCESS_KEY_ID` and `NIX_CACHE_AWS_SECRET_ACCESS_KEY` as Cursor + environment secrets with read-only access to that cache bucket. diff --git a/docs/DATABASE_DEVELOPMENT.md b/docs/DATABASE_DEVELOPMENT.md new file mode 100644 index 0000000000..4085f3038b --- /dev/null +++ b/docs/DATABASE_DEVELOPMENT.md @@ -0,0 +1,122 @@ +# Database development + +Read this for SQLx queries, migrations, database-backed tests, and cache errors. +Commands run from the repository root inside `nix develop` unless shown with an +explicit Nix invocation. See also [Rust development](RUST_DEVELOPMENT.md). + +## Locate the schema and owner + +- MacroDB migrations live in [`crates/macro_db_client/migrations/`](../crates/macro_db_client/migrations/). + Check the owning crate's migrations/recipes before assuming another database + uses the same setup. +- Inspect migrations or use the [dump-schema skill](../.agents/skills/dump-schema/SKILL.md) + against the local database. Do not guess table or column spelling. +- Quote camelCase identifiers and alias selected columns to their Rust field + names: `SELECT "userId" AS user_id FROM "UserInsights"`. +- Use the owning domain's database adapter; follow the style guide's `[db]` rules + (CS-01–09) and ownership rule (CS-27) in [STYLE_GUIDE.md](STYLE_GUIDE.md). + +## Set up local test databases + +On a **local machine**, from the root inside Nix: + +```bash +just run_dbs -d +just setup_test_envs +just setup_macrodb +``` + +`run_dbs` creates the required networks/volumes and starts Postgres and Redis. +`setup_macrodb` creates and migrates MacroDB; `initialize_dbs` is its alias. +The default local URL is defined in [database.just](../tooling/just/database.just). +These recipes target that default database, not a named local-stack instance. + +On **Cursor Cloud**, installation already prepares the database and test envs. +Start the infrastructure with `bash .cursor/infra.sh`; see [Cursor Cloud](CURSOR_CLOUD.md). +Apply any new migrations before testing. + +## Change queries or schema + +1. **If the schema changes, generate a migration with SQLx**, never by hand. + For MacroDB, from the root: + + ```bash + sqlx migrate add --source crates/macro_db_client/migrations + ``` + + Edit the generated file. For another migration directory, use its `--source` + or run `sqlx migrate add` from its owning crate. Never invent/copy timestamp + prefixes. If SQLx is unavailable, use the pinned Nix shell or report the blocker; + do not fabricate a migration filename. + +2. **Use compile-time checked queries** (`query!`, `query_as!`, `query_scalar!`, + `query_file!`) by default. Dynamic SQL is only for genuinely dynamic queries; + bind values and allowlist dynamic identifiers. Review indexes, bounded results, + and access-control predicates with the [SQLx query validator](../.pi/skills/sqlx-query-validator/SKILL.md). + +3. **Apply schema changes** to the local database: + + ```bash + just crates/macro_db_client/migrate_db + ``` + +4. **Update affected tests/fixtures and run package tests** from the root with + `SQLX_OFFLINE` unset. Test between batches of query changes, not just at the end. + +5. **Refresh the workspace SQLx cache** when queries (including test queries) or + schema change, or when required metadata is missing. From the repository root: + + ```bash + nix develop --command just prepare_db + ``` + + Commit generated `.sqlx` changes with the query/schema change. Never manually + create or edit `.sqlx/query-*.json`, and never generate a crate-local cache. + Unrelated Rust-only DB-crate edits still need appropriate tests, but do not + require cache preparation when queries/schema are unchanged and metadata is present. + +6. **Rerun the affected tests** after preparation or fixes. Cache preparation is + not a substitute for `cargo test -p ` against the live local database. + +### Include test-only queries in preparation + +The root `prepare_db` wrapper takes **no flags**. If SQLx needs metadata for +queries compiled only in tests, call the workspace helper from the root. First +set `DATABASE_URL` to your intended local database URL, replacing the placeholders +below with your local connection details: + +```bash +export DATABASE_URL='postgres://:@:/' +nix develop --command just sqlx::prepare_db "$DATABASE_URL" --tests +``` + +The helper in +[sqlx.just](../tooling/just/sqlx.just) forwards `--tests` to Cargo while keeping +workspace scope and the root `.sqlx` directory. Do not run preparation from an +individual crate or try `just prepare_db --tests`. + +## Troubleshooting + +- **Connection or schema errors:** confirm local Postgres is running, test envs + point to the intended local database, and migrations have been applied. +- **Missing cached query data:** use the preparation commands above, including + `--tests` through the helper for test-only queries. Leave `SQLX_OFFLINE` unset + for `cargo test`; enabling it can hide schema drift and produce misleading + type-inference errors. Offline mode is only for builds/checks/lints. +- **Preparation still fails:** fix in-scope query/schema issues; if the blocker + is environmental, report it. Do not hand-edit the cache or wipe databases as + an automatic workaround. + +### Destructive local reset — explicit approval required + +Only after confirming the target is the disposable local database and the user +approves losing its data, run from the repository root inside Nix: + +```bash +just crates/macro_db_client/drop_db -y -f +just setup_macrodb +``` + +Do not use this on hosted dev/production databases or delete unrelated containers. +For a named local stack, follow [its instance-specific commands](RUNNING_LOCALLY.md) +so you do not reset the default database by mistake. diff --git a/docs/RUST_DEVELOPMENT.md b/docs/RUST_DEVELOPMENT.md new file mode 100644 index 0000000000..a2015b9e68 --- /dev/null +++ b/docs/RUST_DEVELOPMENT.md @@ -0,0 +1,84 @@ +# Rust development + +Read this for backend changes in `crates/`, `services/`, or Rust tooling. +Return to the [repository guide](../AGENTS.md) for other task guides. + +## Before editing + +- Read the Rust section of the [style guide](STYLE_GUIDE.md). It owns the coding + rules, including configuration, errors, tracing, public docs, and test layout. +- Follow the [hexagonal architecture skill](../.agents/skills/cloud-storage-hexagonal-architecture/SKILL.md) + before changing backend Rust. Keep domain policy and authorization in domain + services; inbound adapters call services and outbound adapters implement ports. +- Find the owning crate before adding logic. Do not grow catch-all crates or + bypass the owning domain's APIs with direct database access. +- For queries, migrations, or SQLx failures, use [database development](DATABASE_DEVELOPMENT.md). + +## Environment + +Run commands from the repository root inside `nix develop`, unless a guide says +otherwise. Nix supplies Cargo, the Rust toolchain, `just`, SQLx, and the other +build tools; see [local prerequisites](RUNNING_LOCALLY.md#shared-prerequisites). + +Database-backed tests need running local Postgres (and Redis where used). +Crates with compile-time SQLx macros may need Postgres even to compile their +unit tests. Standalone pure-logic crates need no database setup. + +- **Local machine:** use the database guide for test infrastructure, or + [running locally](RUNNING_LOCALLY.md#run-the-local-stack) for the full product. +- **Cursor Cloud:** use [the Cloud entry points](CURSOR_CLOUD.md), not the local + stack TUI. Run `bash .cursor/infra.sh` before DB-backed tests. + +## Build and check + +| Command | Purpose | +| --- | --- | +| `cargo build -p ` | Build the affected Cargo package. | +| `just build` | Build with the workspace's default Cargo selection, using the SQLx cache. | +| `just rust-check` | Type-check the Rust workspace using the SQLx cache. | +| `cargo fmt` | Format Rust code. | +| `just check` | Fast, change-scoped format/lint/code-rule gate; not a type check or test runner. | +| `just check full` | Add TypeScript checking and Rust clippy to the gate. | +| `just clippy` | Run workspace Rust lints using the SQLx cache. | + +`SQLX_OFFLINE=true` is allowed for `cargo check`, `cargo build`, and `cargo clippy` +when the cache is current. Do not export it for a session that will run tests. + +For deployable Lambda artifacts, use `just build_lambdas` or the relevant +service's build recipe, e.g. `just services/document_text_extractor/build`. +Use `just --show ` to check prerequisites; do not assume all Lambdas +have the same build requirements. + +Command definitions live in [rust.just](../tooling/just/rust.just) and +[check.just](../tooling/just/check.just). + +## Test the affected packages + +Use the package name from the affected `Cargo.toml`, not a guessed spelling of +its directory name. Run each affected package's tests before committing: + +```bash +unset SQLX_OFFLINE +cargo test -p +``` + +There is no root `just test` recipe. Tests use the live local schema, not offline +query metadata. If a test reports missing SQLx cache data, follow the +[database troubleshooting workflow](DATABASE_DEVELOPMENT.md#troubleshooting); +do not enable offline mode to work around it. + +Keep tests in a sibling `test.rs` (CS-49). For example, `src/user.rs` declares: + +```rust +#[cfg(test)] +mod test; +``` + +The tests themselves live in `src/user/test.rs` and can use `use super::*;`. +A file module can coexist with its directory; do not convert `user.rs` to +`user/mod.rs` just to add tests. + +Update regression tests for changed behavior, including allow/deny cases for +authorization changes. Run relevant tests between batches of query changes and +refresh SQLx metadata as directed by the database guide. If you cannot run a +check, report the command and blocker rather than treating it as passed. diff --git a/docs/STYLE_GUIDE.md b/docs/STYLE_GUIDE.md index 4ed51b8de5..8e447bb85e 100644 --- a/docs/STYLE_GUIDE.md +++ b/docs/STYLE_GUIDE.md @@ -44,10 +44,10 @@ TypeScript · `[ui]` UI / UX conventions already fixed per durable object / per tenant). (#3961) - **CS-08** `[db]` Use `sqlx::query!` / `query_as!` (compile-time checked) by default; the non-macro form is only for queries that genuinely cannot be statically known. - (#4156 · enforced: clippy `disallowed-methods` · also: CLAUDE.md) -- **CS-09** `[db]` The `.sqlx` cache lives at the workspace root — run `just prepare_db` - from the repository root; never commit a `.sqlx` directory inside an individual - crate. (#4577 · also: CLAUDE.md) + (#4156 · enforced: clippy `disallowed-methods` · workflow: [database development](DATABASE_DEVELOPMENT.md)) +- **CS-09** `[db]` The `.sqlx` cache lives at the workspace root — run + `nix develop --command just prepare_db` from the repository root; never commit a + `.sqlx` directory inside an individual crate. (#4577 · workflow: [database development](DATABASE_DEVELOPMENT.md)) - **CS-10** `[types]` Newtype your identifiers and tokens — wrap raw `String` ids/tokens/model-ids in a validated newtype that checks shape at construction. (#4020, #4077, #4276) @@ -61,25 +61,26 @@ TypeScript · `[ui]` UI / UX conventions `std::env::var`, never hand-rolled wrappers; use `MaybeEnvVar` for optional vars. The same goes for AWS config instantiation (`macro_aws_config`) and tracing subscriber setup (`macro_entrypoint`): use the shared crates. - (#4306, #4334, #4380 · enforced: clippy `disallowed-methods` · also: CLAUDE.md) + (#4306, #4334, #4380 · enforced: clippy `disallowed-methods`) - **CS-15** `[cfg]` Fail fast: validate config at service instantiation, not deep inside request handling — a missing env var should kill startup, not a request. (#4077, #4156) - **CS-16** `[cfg]` Don't add `.context()` to env-var macro errors — the macro error already statically names the missing variable. (#4156) - **CS-17** `[cfg]` Doppler secret key names must exactly match the env var name referenced in code. (#4525) -- **CS-18** `[cfg]` All new environment variables are plain env vars, not - `LocalOrRemote`/doppler-wrapped; non-secret config goes in Doppler as raw values, not - AWS Secrets Manager secrets. (#4305, #4525) +- **CS-18** `[cfg]` Register new environment variables in Doppler. They are plain + env vars, not `LocalOrRemote`/doppler-wrapped; non-secret config goes in Doppler as + raw values, not AWS Secrets Manager secrets. (#4305, #4525) - **CS-19** `[err]` Give third-party errors their own variant — don't collapse e.g. a `jsonwebtoken` failure into a generic internal error. (#4020) - **CS-20** `[err]` Depending on a rate-limited external provider requires a fallback (fallback model, retry story, or documented degradation). (#4296) - **CS-21** `[err]` Wire usage metering on every invocation path — MCP-triggered tool calls count too, not just the primary path. (#4296) -- **CS-22** `[err]` Tracing: `#[instrument(err)]` only on `Result` functions; log errors - as structured fields (`tracing::error!(error=?e, "msg")`); prefer `.inspect_err` over - `if let Err(e)` for logging. (also: CLAUDE.md) +- **CS-22** `[err]` When adding `#[tracing::instrument]`, include `err` on `Result` + functions and omit it on other return types. Never set `level = "info"` on the + attribute. Log errors as structured fields (`tracing::error!(error=?e, "msg")`); + prefer `.inspect_err` over `if let Err(e)` for logging. - **CS-23** `[arch]` Do not grow `macro_db_client` — new domain logic gets a new crate; the catch-all crates must shrink, not accumulate. (#4380) - **CS-24** `[arch]` Keep source files under ~1000 lines — split before a reviewer has @@ -99,7 +100,7 @@ TypeScript · `[ui]` UI / UX conventions - **CS-29** `[arch]` Group proliferating root files (e.g. Dockerfiles) into a dedicated folder. (#4380) - **CS-30** `[api]` Axum handlers take shared services via `State`, not `Extension`. - (#4556 · enforced: ast-grep `rust-no-axum-extension-param`, warning · also: CLAUDE.md) + (#4556 · enforced: ast-grep `rust-no-axum-extension-param`, warning) - **CS-31** `[api]` Attach cross-cutting services to the owning domain service, not ad hoc at the router/handler layer — e.g. `EntityAccessManagementService` hangs off the email/document service itself, the way the documents crate does. (#4572) @@ -132,16 +133,18 @@ TypeScript · `[ui]` UI / UX conventions - **CS-45** `[rust]` CLI binaries use `clap`, not hand-rolled arg parsing. (#3678) - **CS-46** `[rust]` Use `rootcause` for error handling in new code — it's preferred over `anyhow` these days. In code that's still on anyhow, prefer `bail!` for early - error returns. (also: CLAUDE.md) + error returns. - **CS-47** `[perf]` Keep latency-critical services thin: push bytes directly instead of round-tripping through presigned URLs or extra services; dispatch non-blocking background work with `wait_until`. (#3781) - **CS-48** `[perf]` Don't do per-message work on hot websocket paths — accumulate and flush on a timer/alarm. (#3961) - **CS-49** `[test]` Tests live in a sibling `test.rs`, not inline `#[cfg(test)]` blocks - in the implementation file. (#4647 · also: CLAUDE.md) -- **CS-50** `[test]` Update tests and run `just prepare_db` with any db-crate change. - (also: CLAUDE.md) + in the implementation file. (#4647 · example: [Rust development](RUST_DEVELOPMENT.md#test-the-affected-packages)) +- **CS-50** `[test]` Update affected tests for db-crate changes. Refresh SQLx metadata + when queries/schema change or required cache data is missing, not for unrelated + Rust-only edits. Run affected tests with `SQLX_OFFLINE` unset. + (workflow: [database development](DATABASE_DEVELOPMENT.md)) - **CS-51** `[arch]` Domain modules reference no infrastructure or transport: no AWS SDKs, redis, reqwest, opensearch, kafka, axum, or http types under `src/domain/**` — wrap clients in outbound adapters behind ports; response mapping lives in inbound. @@ -156,6 +159,9 @@ TypeScript · `[ui]` UI / UX conventions - **CS-53** `[arch]` Inbound adapters run no database queries — handlers, tools, and listeners call a domain service backed by an outbound repository, never sqlx directly. (enforced: ast-grep `rust-no-sqlx-in-inbound`, warning) +- **CS-54** `[rust]` New crates put `#![deny(missing_docs)]` in `lib.rs` and document + all public items. Do not mark documentation code blocks `ignore` to skip doctests + unless explicitly directed. ## Frontend and shared TypeScript (`apps/web`, `packages/`)