From fae6a388f383c74cb379ee4285a5c458cd8619f0 Mon Sep 17 00:00:00 2001 From: MicrowaveDev Date: Fri, 31 Jul 2026 18:31:39 +0100 Subject: [PATCH 1/3] docs: plan agent-friendly API delivery --- docs/README.md | 2 + .../agent-friendly-api-implementation-plan.md | 402 ++++++++++++++++++ docs/agent-friendly-api-recommendations.md | 4 + docs/agent-map.md | 3 + docs/todo.md | 30 ++ 5 files changed, 441 insertions(+) create mode 100644 docs/agent-friendly-api-implementation-plan.md diff --git a/docs/README.md b/docs/README.md index 237a4449..a046419b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -12,6 +12,8 @@ API reference output with handwritten architecture, operations, and module notes - [Agent docs map](./agent-map.md): task-to-doc routing for agents and maintainers. - [Agent-friendly API recommendations](./agent-friendly-api-recommendations.md): public discovery, immutable asset, auth, error, async, and batch contracts. +- [Agent-friendly API implementation plan](./agent-friendly-api-implementation-plan.md): + current-state gaps, delivery phases, workstreams, rollout, and verification. - [Active TODO](./todo.md): unfinished deterministic implementation sections. - [Implemented work](./implemented.md): delivered foundations and verification history previously mixed into the TODO. diff --git a/docs/agent-friendly-api-implementation-plan.md b/docs/agent-friendly-api-implementation-plan.md new file mode 100644 index 00000000..d2cfcf01 --- /dev/null +++ b/docs/agent-friendly-api-implementation-plan.md @@ -0,0 +1,402 @@ +# Agent-Friendly API Implementation Plan + +## Table Of Contents + +- [Outcome](#outcome) +- [Current-State Analysis](#current-state-analysis) +- [Contract Decisions To Freeze](#contract-decisions-to-freeze) +- [Delivery Sequence](#delivery-sequence) +- [Workstream Specifications](#workstream-specifications) +- [Parallelization And Ownership](#parallelization-and-ownership) +- [Rollout And Compatibility](#rollout-and-compatibility) +- [Verification Matrix](#verification-matrix) +- [Definition Of Done](#definition-of-done) +- [Deferred Follow-Ups](#deferred-follow-ups) + +## Outcome + +An automated client that knows only a GeeSome public origin can discover the +externally reachable API, inspect an accurate contract, validate its +credentials, upload and verify immutable assets idempotently, follow async +work, and resume a multi-file release without source-code knowledge or +deployment-specific path guesses. + +The implementation should extend the existing `api`, `content`, `storage`, +`gateway`, `asyncOperation`, `database`, and `pin` foundations. It should not +create a second HTTP server, a second content store, or an unrelated job system. + +## Current-State Analysis + +The recommendations are valid, but several foundations already exist and +should be hardened rather than replaced. + +| Area | Existing foundation | Gap to close | +| --- | --- | --- | +| Discovery | `GET /v1`, OpenAPI/apiDoc routes, docs headers, and a live route registry exist in `app/modules/api/index.ts`. | Discovery uses relative URLs, does not expose `/.well-known/geesome`, has no capability or compatibility contract, and does not prove that advertised paths work through the public proxy. | +| Reverse proxy | `bash/nginx.conf` and `bash/nodomain-nginx.conf` explicitly proxy conventional OpenAPI paths and `/api/`. | The public prefix is topology-dependent. `/.well-known/` is otherwise served from a filesystem alias, and the Cloudflare template does not expose the same API discovery surface. There is no production-shaped proxy contract test. | +| Errors | Route helpers map some failures to HTTP status codes, and the API wrapper catches unhandled errors. | Responses include bare statuses, empty bodies, `{error, errorCode}`, and `{message, errorCode}`. Storage failures can be reported as `400`; documented routes can still fall through to proxy/HTML errors. There is no common problem schema or request correlation. | +| Uploads | `/v1/user/save-file` accepts multipart data, supports raw storage through `driver`, enforces user limits, and can run synchronously or asynchronously. | The client must know content/driver internals. There is no required digest, durable idempotency contract, stable asset representation, logical-path metadata contract, or deterministic `201`/`202` split. | +| Immutable reads | `/ipfs/*` and `/v1/content-data/*` support `GET`, `HEAD`, ranges, content length/type, CORS, and immutable caching. | Responses lack a standard digest, stable ETag, GeeSome storage-id header, and request ID. Invalid or unavailable identifiers are not expressed as stable problems. | +| Async operations | `asyncOperation` persists user-owned work, progress, errors, cancellation, and queue links. | Uploads return either content or `{asyncOperationId, channel}` under the same success status. Operation reads use legacy POST routes and `inProcess` rather than a small stable state machine. | +| Integration auth | API keys already support permissions, expiry, disable/revocation, ownership checks, and a current-key endpoint. | Permissions are not presented as a documented integration-scope vocabulary. There is no `lastUsedAt`, no stable credential-introspection response, and `403` does not tell a client which scopes are required. | +| Physical metadata | `StorageObject` is the canonical physical-byte registry and pin modules track durable pin attempts/status. | SHA-256 is not canonical metadata, and asset/idempotency/logical-path records do not exist. Batch preflight and manifest completion are absent. | +| OpenAPI | The spec is generated from apiDoc and recognizes bearer auth and multipart file fields. | Server URLs are relative and generic; generated operations mostly declare only `200`. Required fields, status-specific schemas, problem responses, scopes, limits, examples, and compatibility metadata are incomplete. | + +This leads to one sequencing rule: repair the public contract and response +infrastructure before adding asset and batch endpoints. Otherwise the new +routes would inherit the ambiguity they are meant to remove. + +## Contract Decisions To Freeze + +Record these decisions in a short API contract/ADR before implementation. The +ADR is the shared input for all workstreams. + +1. **Canonical discovery:** `GET /.well-known/geesome` is the bootstrap route. + It returns absolute URLs derived from one validated public-origin + configuration, never from an untrusted `Host` header. `GET /v1` remains as a + backward-compatible route index. +2. **External route shape:** discovery advertises the actual public base, such + as `https://host/api/v1`; route annotations continue to describe versioned + application paths. OpenAPI receives the deployed absolute server URL. +3. **Error shape:** all new routes and all cross-cutting auth/parser failures use + RFC 9457-style `application/problem+json`, with a stable GeeSome `code` and + `requestId`. Existing routes migrate incrementally through the same helper. +4. **Asset ownership:** `Content` remains the owner/library record and + `StorageObject` remains physical metadata. New asset records store only the + integration contract: owner, content/storage reference, SHA-256, optional + logical path, idempotency identity, and lifecycle timestamps. +5. **Digest storage:** store lowercase SHA-256 as canonical physical metadata on + `StorageObject`, with an indexed lookup. Treat this as an additive production + schema change and ship a real migration. Do not globally reveal hash + existence; batch preflight only reports reusable assets visible to the + authenticated owner. +6. **Idempotency:** uniqueness is scoped to authenticated owner plus endpoint + namespace plus `Idempotency-Key`. Persist a normalized request fingerprint. + Replaying the same request returns the original result; reusing the key for a + different request returns `409 idempotency_key_conflict`. +7. **Upload verification:** compute byte count and SHA-256 while multipart bytes + are written to the temporary file, compare before calling content/storage, + and always clean temporary files on rejection. A mismatch returns `422` and + must not create Content, StorageObject, file-catalog, or pin rows. +8. **Async shape:** synchronous creation returns `201` plus the asset resource. + Deferred work returns `202`, `Location`, `Retry-After`, and an operation + resource. The public operation state maps existing rows to `pending`, + `running`, `succeeded`, `failed`, or `cancelled`. +9. **Pin semantics:** distinguish `stored` from `pinned`/`confirmed`. Discovery + describes the deployment's immediate-read and pin policy. Asset success + never implies more durability than the recorded pin state proves. +10. **Manifest integrity:** a completed batch manifest uses deterministic + canonical JSON and includes a manifest SHA-256. Signing is added only when a + configured server signing key, key identifier, verification method, and key + rotation policy are defined; hash binding is mandatory in the first slice. + +## Delivery Sequence + +### Phase 0 — Freeze Contracts And Baselines + +- Add the ADR described above and versioned JSON fixtures for discovery, + problems, assets, operations, batches, and manifests. +- Capture baseline behavior for direct-node and nginx-shaped requests, + including the currently deployed `/api/v1` topology. +- Add a deterministic TODO section for this roadmap so later agents can use + `npm run todo:context -- `. +- Decide the public-origin configuration names and precedence across direct, + nginx, Cloudflare, gateway, and no-domain installs. + +Exit gate: fixture schemas and public URL rules are reviewed; no route +implementation begins while response shapes remain unsettled. + +### Phase 1 — Make The Existing API Reliably Discoverable + +- Implement request IDs and the common problem response layer. +- Implement `/.well-known/geesome` and absolute discovery/OpenAPI links. +- Align every nginx template and add `Link` headers at the public entry points. +- Add black-box tests through a production-shaped reverse proxy. + +Exit gate: a client starting at the site origin discovers valid absolute URLs, +and every tested API/proxy failure returns a correlated problem response. + +### Phase 2 — Add The Immutable Asset Façade + +- Add asset persistence and SHA-256 metadata without duplicating stored bytes. +- Implement `POST /v1/assets`, `GET /v1/assets/:storageId`, and the corresponding + externally prefixed routes through discovery. +- Make raw storage the default and previews an explicit `previewPolicy`. +- Add digest/ETag/storage-id/request-id headers to `GET` and `HEAD` reads. + +Exit gate: one idempotent synchronous upload can be byte-verified end to end +using only discovery, OpenAPI, the asset response, and `HEAD`/`GET`. + +### Phase 3 — Normalize Operations And Integration Credentials + +- Add `GET /v1/operations/:id` and cancellation semantics as a façade over the + existing async-operation store. +- Return deterministic `201` or `202` from asset creation. +- Define asset/operation scope aliases over existing core permissions and expose + a safe current-credential introspection resource. +- Record `lastUsedAt` without writing synchronously on every request; use a + throttled update or bounded background flush. + +Exit gate: a minimally scoped key can upload and poll its own operation, cannot +read another user's operation, and receives actionable `403` problems. + +### Phase 4 — Add Resumable Batches And Manifests + +- Implement batch create, per-item upload association, status, and completion. +- Reuse owner-visible hashes and resume interrupted uploads without duplicate + Content, file-catalog, asset, or batch-item rows. +- Emit a deterministic hash-bound manifest and store it through the existing + immutable content/storage path. + +Exit gate: the 50-file interrupted-batch scenario passes under retries and +process restart. + +### Phase 5 — Publish Executable Documentation And Roll Out + +- Complete OpenAPI schemas, status responses, scopes, limits, and examples. +- Add tested curl and Node `fetch` examples that always start with discovery. +- Run the Meat Master publish/hydrate contract through the public proxy. +- Roll out additive routes first, measure legacy route use, then publish + deprecation and sunset dates where replacement is justified. + +Exit gate: CI runs examples through the same topology used in production and +the public deployment passes the consumer contract without prefix heuristics. + +## Workstream Specifications + +### WS-A — Request Context And Problem Responses + +- **Inputs:** contract fixtures; current API response adapter; route error + helpers; nginx error behavior. +- **Outputs:** request-ID middleware; typed problem factory; error-to-status/code + mapping; response adapter support for content type, status, and location; + sanitized structured logging. +- **Write scope:** `app/modules/api/**`, shared API interfaces/helpers, focused + API/error tests, nginx error interception only where required. +- **Dependencies:** Phase 0 fixtures. +- **Completion criteria:** request IDs accept a valid inbound correlation ID or + generate one; the same ID reaches logs and responses; auth, parser, not-found, + rate/size, storage-backend, and unhandled errors use problem JSON; no secrets + or stack traces are returned. +- **Verification:** unit tests for every mapping; malformed multipart and invalid + CID integration tests; black-box proxy tests asserting JSON content type and + request-ID continuity. + +### WS-B — Public Discovery, Proxy, And Capability Contract + +- **Inputs:** public-origin ADR; existing discovery builder; install-time nginx + templates; deployment configuration. +- **Outputs:** `/.well-known/geesome`; absolute discovery fields; validated + capability/backend characteristics; absolute OpenAPI server URL; consistent + `Link` headers; startup/public-origin validation command. +- **Write scope:** `app/modules/api/**`, `app/config.ts`, relevant interfaces, + `bash/*nginx.conf`, install scripts, discovery/proxy checks. +- **Dependencies:** Phase 0 fixtures; WS-A for problem responses. +- **Completion criteria:** all advertised URLs resolve from outside the app + container; direct, `/api`, Cloudflare, gateway, and no-domain topologies have + an explicit tested behavior; file aliases do not shadow GeeSome discovery. +- **Verification:** direct-node tests plus a containerized nginx black-box suite; + negative startup/config tests; security route inventory update/check. + +### WS-C — Asset Persistence, Idempotency, And Upload API + +- **Inputs:** asset fixtures; `Content`/`StorageObject` ownership rules; + `asyncBusboy`; raw content save path; file catalog and quota behavior. +- **Outputs:** asset module/API; additive asset/idempotency models; nullable + `StorageObject.sha256`; streaming hash/byte-count capture; request-fingerprint + validation; stable asset serializer. +- **Write scope:** new `app/modules/asset/**`; focused additions in `content`, + `database`, and app wiring; migrations and migration integrity checks; asset + unit/integration tests. +- **Dependencies:** WS-A and WS-B; Phase 0 data decisions. +- **Completion criteria:** exactly one file is required; SHA-256 and MIME/size + limits are validated; hash mismatch publishes nothing; replay is stable; + conflicting key reuse returns `409`; logical path does not affect CID; + previews are opt-in; legacy `/user/save-file` remains compatible. +- **Verification:** upload success/mismatch/limit/duplicate/conflict tests; + concurrent same-key test; failure cleanup test; permission isolation test; + migration integrity and restored-upgrade rehearsal where available. + +### WS-D — Verifiable Immutable Reads + +- **Inputs:** asset representation; storage metadata; existing `GET`, `HEAD`, and + range implementations. +- **Outputs:** `ETag`, RFC-compliant `Content-Digest` (plus legacy `Digest` + only where a measured compatibility need exists), `X-Geesome-Storage-Id`, + `X-Request-Id`, consistent immutable caching, and stable read errors. +- **Write scope:** `app/modules/content/**`, `app/modules/gateway/**`, focused + header/range/error tests. +- **Dependencies:** WS-A; WS-C for persisted SHA-256. Header plumbing can begin + in parallel with WS-C using fixtures. +- **Completion criteria:** `GET`, `HEAD`, and range reads agree on identity, + length, type, cache policy, and ranges; public assets remain CORS-readable; + absent, malformed, locked/private, and backend-unavailable cases are + distinguishable without leaking private existence. +- **Verification:** byte-for-byte digest tests; conditional request tests; + `HEAD`/`GET` parity tests; range regression suite; API and gateway black-box + coverage. + +### WS-E — Operation Resource And Deterministic Async Semantics + +- **Inputs:** operation fixtures; existing `UserAsyncOperation` and queue + ownership/retry behavior; asset serializer. +- **Outputs:** stable operation serializer and state mapping; authorized GET + status route; cancellation route; `Location`/`Retry-After`; embedded asset or + problem result; retention behavior documented in discovery/OpenAPI. +- **Write scope:** `app/modules/asyncOperation/**`, asset response integration, + async-operation tests. +- **Dependencies:** WS-A and WS-C. +- **Completion criteria:** synchronous and async paths never share a status/body + shape; operation access is owner-scoped; success resolves to the same asset + schema; failures preserve stable problem codes and request lineage; cancelled + work cannot later be reported as successful without an explicit terminal + transition policy. +- **Verification:** state-transition tests, restart recovery, cross-user denial, + cancellation race tests, and `201`/`202` contract tests. + +### WS-F — Integration Scopes And Credential Introspection + +- **Inputs:** current core permissions, API-key expiry/disable flow, asset and + operation authorization requirements. +- **Outputs:** documented integration scope vocabulary; mapping to core + permissions; safe current-key resource containing id/title/scopes/created/ + expiry/last-used state; required-scope problem extension; throttled last-used + persistence. +- **Write scope:** `app/index.ts`, `app/modules/api/**`, API-key model/interface, + required migration/integrity checks, auth tests and docs. +- **Dependencies:** WS-A; endpoint scope mapping from WS-C and WS-E. +- **Completion criteria:** least-privilege keys work; expired, revoked, malformed, + and insufficient-scope credentials have distinct safe responses; secret value + and hash never leave persistence/auth internals. +- **Verification:** scope matrix tests, expiry/revocation tests, redaction tests, + concurrent last-used throttling test, security inventory update/check. + +### WS-G — Batch Upload And Hash-Bound Manifest + +- **Inputs:** batch/manifest fixtures; asset idempotency API; operation resource; + storage and pin state. +- **Outputs:** batch, item, and completion models; create/status/complete routes; + owner-visible hash preflight; deterministic manifest serializer; immutable + manifest storage and digest. +- **Write scope:** `app/modules/assetBatch/**` or an `asset/batch` submodule, + database models/migrations, async producer, batch tests and examples. +- **Dependencies:** WS-C, WS-D, and WS-E; WS-F for scopes. +- **Completion criteria:** item identity is stable; completion is atomic from the + caller's perspective; incomplete or hash-mismatched batches cannot complete; + retries and restarts do not duplicate rows; manifest ordering and digest are + deterministic; global hash existence is not disclosed. +- **Verification:** 50-file interrupted/resumed test; concurrent completion; + duplicate logical ID/hash cases; process restart; manifest reserialization + digest equality; migration integrity and database scalability inventory. + +### WS-H — OpenAPI, Examples, Consumer Contract, And Observability + +- **Inputs:** all frozen fixtures and implemented routes; API doc generator; + production-shaped proxy harness; Meat Master consumer flow. +- **Outputs:** complete OpenAPI components/responses/security; schema validation; + executable curl and Node examples; consumer smoke; metrics for discovery, + upload outcomes, idempotency replay/conflict, operation latency, digest + mismatch, and batch completion. +- **Write scope:** apiDoc annotations, `app/apiDocSpec.ts`, `docs/**`, `check/**`, + package scripts, CI configuration where present. +- **Dependencies:** begins with fixtures in Phase 0, lands final coverage after + WS-B through WS-G. +- **Completion criteria:** generated OpenAPI validates and contains the external + server, binary multipart field, required inputs, scopes, all success/problem + statuses, limits, examples, and deprecation metadata; examples and the Meat + Master flow pass without hardcoded `/api`. +- **Verification:** `npm run generate-docs`; OpenAPI schema/lint test; executable + example suite through nginx; route-doc drift check; consumer contract smoke. + +## Parallelization And Ownership + +Use one integration owner for contract fixtures, shared interfaces, migrations, +and final branch/PR coordination. Suggested execution lanes: + +| Lane | Work | Can run in parallel with | Merge dependency | +| --- | --- | --- | --- | +| 1 | WS-A request context/problems | Early WS-B and WS-H fixture/tooling work | Merge first because every new route consumes it. | +| 2 | WS-B discovery/proxy | WS-A, then WS-D header preparation | Merge after shared public-origin and problem interfaces settle. | +| 3 | WS-C asset core | WS-D header plumbing and WS-F scope vocabulary after fixtures freeze | Merge before operation/batch endpoint integration. | +| 4 | WS-D reads + WS-F auth | Each other, and late WS-C tests | Merge before public end-to-end gate. | +| 5 | WS-E operations | WS-F and WS-H schema work | Requires asset serializer. | +| 6 | WS-G batch | Final WS-H docs/examples | Requires asset, reads, operations, and scopes. | + +Agents must not edit the same shared files concurrently. In particular, +`app/modules/api/index.ts`, `app/apiDocSpec.ts`, database model registration, and +generated docs need single-owner integration windows. + +## Rollout And Compatibility + +1. Ship discovery, problem infrastructure, and proxy routes additively. +2. Keep `/v1/user/save-file` and legacy async-operation routes unchanged while + new clients adopt `/assets` and `/operations`. +3. Backfill SHA-256 lazily for existing storage objects on first verified read or + an explicit bounded maintenance job. Do not block deployment on hashing the + full store. +4. Guard new capabilities with truthful discovery flags. A flag becomes `true` + only after its public black-box test passes for that deployment. +5. Treat asset database success, storage availability, and confirmed pinning as + separate states in responses and metrics. +6. Add deprecation and sunset headers only after usage telemetry and migration + notes exist. Do not remove legacy routes as part of the first delivery. +7. Roll back by disabling advertisement/new route registration while preserving + additive rows and nullable columns. Never require destructive data rollback. + +## Verification Matrix + +| Contract | Required evidence | +| --- | --- | +| Origin-only discovery | Direct and nginx/Cloudflare/no-domain black-box tests; every absolute advertised URL fetched successfully. | +| Stable problems | Status/code/schema matrix, JSON content type, request-ID continuity, proxy fallthrough test, secret-redaction test. | +| Upload integrity | Known-byte SHA-256 fixture, mismatch with zero durable side effects, quota/size/MIME failures, temporary-file cleanup. | +| Idempotency | Serial replay, concurrent replay, request-fingerprint conflict, restart replay, owner isolation. | +| Immutable reads | `HEAD`/`GET` parity, digest and byte count, ETag/conditional request, ranges, immutable cache, CORS. | +| Async operations | `201` final, `202` operation, state transitions, failure problem, cancellation, restart recovery, ownership. | +| Credentials | Scope matrix, current-key metadata, expiry, revocation, last-used throttling, no secret/hash serialization. | +| Batch resume | 50 files, interruption mid-upload, retry after restart, no duplicate rows/catalog entries, deterministic manifest hash. | +| Documentation | Generated OpenAPI validation, route/doc drift checks, executable curl/Node examples through nginx. | +| Consumer proof | Meat Master publish/hydrate smoke starts at discovery and persists/verifies CID plus SHA-256 without prefix heuristics. | + +For every route change, also run the repository-required documentation and +security inventory workflows. Database work must update migration integrity; +large batch/list queries must update the database scalability inventory. Use the +narrow unit/integration suites first, then `npm run test:docker` for the final +cross-module gate. + +## Definition Of Done + +- `/.well-known/geesome` is public and consistent across supported deployment + topologies. +- Discovery contains absolute, externally reachable URLs, version/capabilities, + compatibility links, upload limits, and storage-read/pin characteristics. +- All new endpoints and migrated cross-cutting failures return stable problem + documents with a correlated request ID. +- Asset upload needs no driver knowledge and proves byte count and SHA-256 before + publication. +- Asset idempotency is durable, concurrency-safe, owner-scoped, and conflict + detecting. +- `GET`/`HEAD` allow an agent to verify immutable bytes and cache safely. +- Sync and async paths use distinct status codes and a single asset schema. +- Integration credentials are least-privilege, introspectable, expiring, and + revocable without exposing secrets. +- Interrupted batches resume without duplicate product or storage metadata. +- OpenAPI and executable examples match the public proxy, and the consumer smoke + passes from origin-only discovery. +- `docs/todo.md`, `docs/implemented.md`, module docs, generated API docs, + security inventory, scalability inventory, and migration integrity evidence + accurately reflect delivered scope. + +## Deferred Follow-Ups + +- Generic idempotency middleware for non-asset mutation endpoints, after the + asset contract proves the persistence model. +- Signed manifests, after signing-key ownership, rotation, verification, and + recovery policies are defined. +- Pre-signed/direct-to-storage uploads, only if proxy upload throughput becomes + a measured bottleneck and end-to-end digest verification remains enforced. +- A broader REST normalization of legacy POST-based reads. Do not couple that + migration to the agent-friendly asset delivery. +- Global content deduplication or cross-user hash reuse. This needs an explicit + privacy and quota policy and is not required for resumable owner-scoped + batches. diff --git a/docs/agent-friendly-api-recommendations.md b/docs/agent-friendly-api-recommendations.md index b74696af..fd41cf8d 100644 --- a/docs/agent-friendly-api-recommendations.md +++ b/docs/agent-friendly-api-recommendations.md @@ -12,6 +12,10 @@ This document records recommendations from the Meat Master character-asset integration performed on 2026-07-31. It is an API roadmap, not a claim that the target contracts are already implemented. +See [Agent-Friendly API Implementation Plan](./agent-friendly-api-implementation-plan.md) +for the current-state gap analysis, phased delivery sequence, workstream scopes, +and verification gates. + ## Integration Findings The repository-level API contract was understandable after reading the diff --git a/docs/agent-map.md b/docs/agent-map.md index c2d6d53f..b2d3e340 100644 --- a/docs/agent-map.md +++ b/docs/agent-map.md @@ -20,6 +20,9 @@ Use this map after loading the repo instructions in `AGENTS.md`. - Read [Agent-friendly API recommendations](./agent-friendly-api-recommendations.md) when changing public discovery, reverse-proxy paths, uploads, immutable gateway reads, auth scopes, async responses, errors, or integration examples. +- Use the [Agent-friendly API implementation plan](./agent-friendly-api-implementation-plan.md) + for delivery order, workstream boundaries, rollout constraints, and required + verification evidence. - Run or update the security route inventory when route auth, permissions, or endpoint shape changes. diff --git a/docs/todo.md b/docs/todo.md index ac89cb97..4c16fec0 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -254,6 +254,36 @@ Verification: - Focused authorization and abuse tests for every changed route. + +## Agent-Friendly Public API + +Goal: let an integration discover, authenticate, upload, verify, and resume +immutable asset publication when it starts with only the public site origin. + +Source of truth: + +- [Recommendations](./agent-friendly-api-recommendations.md) +- [Implementation plan](./agent-friendly-api-implementation-plan.md) + +Delivery order: + +1. Freeze versioned response fixtures and public-origin rules. +2. Standardize request IDs/problems and expose absolute discovery through the + production-shaped proxy. +3. Add the idempotent SHA-256-verified asset façade and verifiable reads. +4. Normalize operation resources and least-privilege integration credentials. +5. Add resumable batches, deterministic hash-bound manifests, executable + examples, and the Meat Master consumer contract smoke. + +Verification: + +- Use the per-workstream gates and verification matrix in the implementation + plan. +- Run API docs generation and route/security inventory checks for route changes. +- Run migration integrity/scalability checks for asset and batch persistence. +- Run the final cross-module suite with `npm run test:docker`. + + ## Static Sites: Settings And Delivery From 158efd5e3169c94e7dc9bba9c2b07808d685d524 Mon Sep 17 00:00:00 2001 From: MicrowaveDev Date: Fri, 31 Jul 2026 19:44:09 +0100 Subject: [PATCH 2/3] feat: add agent-friendly asset API Closes #1312 --- app/apiDocSpec.ts | 98 ++++- app/config.ts | 13 +- app/index.ts | 23 +- app/interface.ts | 3 + app/modules/api/api.ts | 18 +- app/modules/api/index.ts | 143 +++++-- app/modules/api/integrationScopes.ts | 82 ++++ app/modules/api/interface.ts | 3 + app/modules/api/problem.ts | 154 +++++++ app/modules/api/publicUrls.ts | 28 ++ app/modules/api/routeErrorHelpers.ts | 7 +- app/modules/asset/api.ts | 185 ++++++++ app/modules/asset/docs/README.md | 12 + app/modules/asset/index.ts | 396 ++++++++++++++++++ app/modules/asset/interface.ts | 19 + app/modules/asset/models.ts | 78 ++++ app/modules/asyncOperation/api.ts | 86 ++++ app/modules/asyncOperation/index.ts | 15 +- app/modules/asyncOperation/interface.ts | 1 + app/modules/asyncOperation/models.ts | 3 + app/modules/content/asyncBusboy.ts | 21 +- app/modules/content/index.ts | 17 +- app/modules/database/index.ts | 4 + app/modules/database/interface.ts | 5 + .../20260731000000-add-agent-api-metadata.cjs | 68 +++ app/modules/database/models/storageObject.ts | 26 +- app/modules/database/models/userApiKey.ts | 6 + app/modules/gateway/index.ts | 5 + bash/cf-nginx.conf | 13 +- bash/docker-test | 2 +- bash/nginx.conf | 6 +- bash/nodomain-nginx.conf | 1 + bash/uncert-nginx.conf | 8 +- check/databaseMigrationIntegrity.ts | 19 + check/databaseScalabilityInventory.ts | 23 + docker-compose.yml | 5 + docs/README.md | 6 +- .../agent-friendly-api-implementation-plan.md | 7 + docs/agent-friendly-api.md | 45 ++ docs/api-problems.md | 25 ++ docs/database-scalability-inventory.md | 4 +- docs/decisions/agent-friendly-api-contract.md | 23 + docs/implemented.md | 36 +- docs/security-route-inventory.md | 14 +- docs/todo.md | 30 -- examples/agent-friendly-assets.mjs | 41 ++ package.json | 1 + test/agent-api-nginx.conf | 17 + test/agentFriendlyApiIntegration.test.ts | 247 +++++++++++ test/agentFriendlyApiUnit.test.ts | 92 ++++ test/apiCallbackHandling.test.ts | 3 +- test/apiHeaders.test.ts | 31 +- test/asyncOperationSecurityUnit.test.ts | 4 +- test/authRouteErrors.test.ts | 6 +- test/contentHeaders.test.ts | 15 +- test/contentRouteErrors.test.ts | 17 +- test/docker-compose.yml | 9 + test/storageRouteErrors.test.ts | 3 +- 58 files changed, 2137 insertions(+), 135 deletions(-) create mode 100644 app/modules/api/integrationScopes.ts create mode 100644 app/modules/api/problem.ts create mode 100644 app/modules/api/publicUrls.ts create mode 100644 app/modules/asset/api.ts create mode 100644 app/modules/asset/docs/README.md create mode 100644 app/modules/asset/index.ts create mode 100644 app/modules/asset/interface.ts create mode 100644 app/modules/asset/models.ts create mode 100644 app/modules/database/migrations/20260731000000-add-agent-api-metadata.cjs create mode 100644 docs/agent-friendly-api.md create mode 100644 docs/api-problems.md create mode 100644 docs/decisions/agent-friendly-api-contract.md create mode 100644 examples/agent-friendly-assets.mjs create mode 100644 test/agent-api-nginx.conf create mode 100644 test/agentFriendlyApiIntegration.test.ts create mode 100644 test/agentFriendlyApiUnit.test.ts diff --git a/app/apiDocSpec.ts b/app/apiDocSpec.ts index 7926e324..54f6ab33 100644 --- a/app/apiDocSpec.ts +++ b/app/apiDocSpec.ts @@ -52,19 +52,52 @@ function fieldSchema(field: any): any { return {type: 'string', format: 'binary'}; } const mapped = TYPE_MAP[(field.type || '').toLowerCase()]; - const base: any = mapped ? {type: mapped} : {}; + const base: any = mapped ? {type: mapped} : {}; + if (field.allowedValues?.length) { + base.enum = field.allowedValues.map((value: string) => value.replace(/^\"|\"$/g, '')); + } + if (field.defaultValue !== undefined) { + base.default = field.defaultValue; + } if (field.isArray) { return {type: 'array', items: base}; } return base; } +function objectSchema(fields: any[]): any { + const properties: any = {}; + const required: string[] = []; + for (const field of fields) { + const schema = fieldSchema(field); + const description = stripHtml(field.description); + if (description) { + schema.description = description; + } + properties[field.field] = schema; + if (!field.optional) { + required.push(field.field); + } + } + return {type: 'object', properties, ...(required.length ? {required} : {})}; +} + +const OPERATION_SCOPES: Record = { + AssetCreate: ['assets:write'], + AssetGet: ['assets:read-private'], + AssetBatchCreate: ['asset-batches:write'], + AssetBatchGet: ['asset-batches:write'], + AssetBatchComplete: ['asset-batches:write'], + OperationGet: ['operations:read'], + OperationCancel: ['operations:read'] +}; + const HTTP_METHODS = ['get', 'post', 'put', 'patch', 'delete', 'head']; // Build an OpenAPI 3 document from the parsed apiDoc data: paths, path params, // request bodies (multipart when a file field is present, else JSON), bearer // security when an Authorization header is documented, and summaries/tags. -export function buildOpenApiFromApiDoc(version: string, docsStorageId?: string): any | null { +export function buildOpenApiFromApiDoc(version: string, docsStorageId?: string, publicApiBaseUrl?: string): any | null { const data = getApiDocData(); if (!data) { return null; @@ -91,9 +124,12 @@ export function buildOpenApiFromApiDoc(version: string, docsStorageId?: string): return `{${name}}`; }); - const operation: any = {responses: {'200': {description: 'OK'}}}; + const operation: any = {responses: {}}; if (endpoint.name) { operation.operationId = endpoint.name; + if (OPERATION_SCOPES[endpoint.name]) { + operation['x-required-scopes'] = OPERATION_SCOPES[endpoint.name]; + } } if (endpoint.title) { operation.summary = endpoint.title; @@ -112,21 +148,30 @@ export function buildOpenApiFromApiDoc(version: string, docsStorageId?: string): if (headerFields.some((h: any) => h.field === 'Authorization')) { operation.security = [{bearerAuth: []}]; } + for (const field of headerFields.filter((item: any) => item.field !== 'Authorization')) { + operation.parameters = operation.parameters || []; + operation.parameters.push({name: field.field, in: 'header', required: !field.optional, schema: fieldSchema(field), description: stripHtml(field.description)}); + } const body = endpoint.body || []; if (body.length && method !== 'get' && method !== 'head') { const isMultipart = body.some((b: any) => b.field === 'file' || (b.type || '').toLowerCase() === 'file'); - const properties: any = {}; - for (const field of body) { - const schema = fieldSchema(field); - const fieldDescription = stripHtml(field.description); - if (fieldDescription) { - schema.description = fieldDescription; - } - properties[field.field] = schema; - } const contentType = isMultipart ? 'multipart/form-data' : 'application/json'; - operation.requestBody = {content: {[contentType]: {schema: {type: 'object', properties}}}}; + operation.requestBody = {required: body.some((field: any) => !field.optional), content: {[contentType]: {schema: objectSchema(body)}}}; } + const successGroups = endpoint.success?.fields || {}; + for (const [status, fields] of Object.entries(successGroups)) { + operation.responses[status] = {description: status === '201' ? 'Created' : 'Success', content: {'application/json': {schema: objectSchema(fields as any[])}}}; + } + const errorGroups = endpoint.error?.fields || {}; + for (const status of Object.keys(errorGroups)) { + operation.responses[status] = {description: 'API problem', content: {'application/problem+json': {schema: {$ref: '#/components/schemas/ApiProblem'}}}}; + } + if (!Object.keys(operation.responses).length) { + operation.responses['200'] = {description: 'OK'}; + } + if (operation.security) { + operation.responses['403'] ||= {description: 'Insufficient scope', content: {'application/problem+json': {schema: {$ref: '#/components/schemas/ApiProblem'}}}}; + } paths[specPath] = paths[specPath] || {}; paths[specPath][method] = operation; @@ -139,12 +184,29 @@ export function buildOpenApiFromApiDoc(version: string, docsStorageId?: string): version, description: "Generated from the node's apiDoc annotations. The full human reference is also published to IPFS on each boot (see x-docs-ipfs).", }, - servers: [ - {url: `/${version}`, description: 'Direct node'}, - {url: `/api/${version}`, description: 'Behind the bundled nginx reverse proxy'}, - ], + servers: getOpenApiServers(version, publicApiBaseUrl), 'x-docs-ipfs': docsStorageId ? `/ipfs/${docsStorageId}` : null, - components: {securitySchemes: {bearerAuth: {type: 'http', scheme: 'bearer'}}}, + components: { + securitySchemes: {bearerAuth: {type: 'http', scheme: 'bearer'}}, + schemas: {ApiProblem: { + type: 'object', + required: ['type', 'title', 'status', 'code', 'requestId'], + properties: { + type: {type: 'string'}, title: {type: 'string'}, status: {type: 'integer'}, + code: {type: 'string'}, detail: {type: 'string'}, requestId: {type: 'string'} + } + }} + }, paths, }; } + +function getOpenApiServers(version: string, publicApiBaseUrl?: string) { + if (publicApiBaseUrl) { + return [{url: publicApiBaseUrl, description: 'Advertised public API'}]; + } + return [ + {url: `/${version}`, description: 'Direct node'}, + {url: `/api/${version}`, description: 'Behind the bundled nginx reverse proxy'}, + ]; +} diff --git a/app/config.ts b/app/config.ts index cbfce608..2ff51a3a 100644 --- a/app/config.ts +++ b/app/config.ts @@ -9,7 +9,7 @@ //TODO: move communicator and fileCatalog to improve const modulePacks = { - 'main': ['drivers', 'database', 'api', 'accountStorage', 'communicator', 'storage', 'content', 'staticId', 'asyncOperation', 'privateGroup', 'group', 'chat', 'fileCatalog', 'entityJsonManifest', 'imageComposition', 'remoteGroup'], + 'main': ['drivers', 'database', 'api', 'accountStorage', 'communicator', 'storage', 'content', 'staticId', 'asyncOperation', 'asset', 'privateGroup', 'group', 'chat', 'fileCatalog', 'entityJsonManifest', 'imageComposition', 'remoteGroup'], 'improve': ['groupCategory', 'invite', 'staticSiteGenerator', 'rss', 'activityPub', 'autoActions', 'pin', 'foreignAccounts', 'ethereumAuthorization', 'storageSpace', 'gateway'], 'socNet': ['socNetAccount', 'socNetImport', 'bluesky', 'telegramClient', 'twitterClient', 'tgContentBot'] }; @@ -17,6 +17,12 @@ const modulePacks = { //TODO: refactor modules config export default { domain: process.env.DOMAIN || '', + apiConfig: { + publicUrl: process.env.GEESOME_PUBLIC_URL || getPublicUrlFromDomainEnv(process.env.DOMAIN), + publicBasePath: normalizeApiBasePath(process.env.GEESOME_API_BASE_PATH || '/api/v1'), + deploymentVersion: process.env.GEESOME_VERSION || process.env.npm_package_version || 'unknown', + maxUploadBytes: process.env.GEESOME_MAX_UPLOAD_BYTES || 2000 * 1024 * 1024 + }, databaseModule: 'sql', databaseConfig: { @@ -134,3 +140,8 @@ function getPublicUrlFromDomainEnv(domain): string { return ''; } } + +function normalizeApiBasePath(value): string { + const path = `/${String(value || '').trim().replace(/^\/+|\/+$/g, '')}`; + return path === '/' ? '/api/v1' : path; +} diff --git a/app/index.ts b/app/index.ts index bf5bc913..2279b9c6 100644 --- a/app/index.ts +++ b/app/index.ts @@ -32,6 +32,7 @@ import IGeesomePrivateGroupModule from "./modules/privateGroup/interface.js"; import IGeesomeChatModule from "./modules/chat/interface.js"; import IGeesomeImageCompositionModule from "./modules/imageComposition/interface.js"; import IGeesomeApiModule from "./modules/api/interface.js"; +import IGeesomeAssetModule from "./modules/asset/interface.js"; import {IGeesomeApp, IUserInput} from "./interface.js"; import {GeesomeEmitter} from "./events.js"; import { @@ -50,6 +51,7 @@ import config from './config.js'; import {startMemoryProfiler} from './memoryProfiler.js'; import type {MemoryProfilerHandle} from './memoryProfiler.js'; import {cleanupAndRethrow, cleanupResource} from './resourceCleanup.js'; +import {getScopePermissions, normalizeIntegrationScopes, serializeApiKey} from './modules/api/integrationScopes.js'; const {pick, merge, isUndefined, startsWith, reverse, clone, extend, isString} = _; const log = debug('geesome:app'); const apiKeyListParams: IListParamsOptions = { @@ -137,6 +139,7 @@ function getModule(config, appPass) { drivers: IGeesomeDriversModule, api: IGeesomeApiModule, asyncOperation: IGeesomeAsyncOperationModule, + asset: IGeesomeAssetModule, staticId: IGeesomeStaticIdModule, invite: IGeesomeInviteModule, group: IGeesomeGroupModule, @@ -377,6 +380,18 @@ function getModule(config, appPass) { data.userId = userId; data.valueHash = generated.uuid; + if (data.scopes) { + const scopes = normalizeIntegrationScopes(data.scopes); + const permissions = getScopePermissions(scopes); + for (const permission of permissions) { + if (!await this.isUserCan(userId, permission)) { + throw new Error('not_permitted'); + } + } + data.scopes = JSON.stringify(scopes); + data.permissions = JSON.stringify(permissions); + } + if (!data.permissions) { data.permissions = JSON.stringify(await this.ms.database.getCorePermissions(userId).then(list => list.map(i => i.name))); } else if (Array.isArray(data.permissions)) { @@ -420,6 +435,12 @@ function getModule(config, appPass) { if (!this.isApiKeyActive(keyObj)) { return {user: null, apiKey: null}; } + const now = new Date(); + const lastUsedAt = keyObj.lastUsedAt ? new Date(keyObj.lastUsedAt).getTime() : 0; + if (!lastUsedAt || now.getTime() - lastUsedAt >= 5 * 60 * 1000) { + await this.ms.database.updateApiKey(keyObj.id, {lastUsedAt: now}); + keyObj.lastUsedAt = now; + } return { user: await this.ms.database.getUser(keyObj.userId), apiKey: keyObj, @@ -438,7 +459,7 @@ function getModule(config, appPass) { listParams = helpers.prepareListParams(listParams, apiKeyListParams); await this.checkUserCan(userId, CorePermissionName.UserApiKeyManagement); return { - list: await this.ms.database.getApiKeysByUser(userId, isDisabled, search, listParams), + list: (await this.ms.database.getApiKeysByUser(userId, isDisabled, search, listParams)).map(serializeApiKey), total: await this.ms.database.getApiKeysCountByUser(userId, isDisabled, search) }; } diff --git a/app/interface.ts b/app/interface.ts index c998d0cb..d0e09523 100644 --- a/app/interface.ts +++ b/app/interface.ts @@ -24,6 +24,7 @@ import IGeesomePrivateGroupModule from "./modules/privateGroup/interface.js"; import IGeesomeChatModule from "./modules/chat/interface.js"; import IGeesomeImageCompositionModule from "./modules/imageComposition/interface.js"; import IGeesomeApiModule from "./modules/api/interface.js"; +import IGeesomeAssetModule from "./modules/asset/interface.js"; import {GeesomeEmitter} from "./events.js"; import { CorePermissionName, @@ -47,6 +48,7 @@ export interface IGeesomeApp { api: IGeesomeApiModule; content: IGeesomeContentModule, asyncOperation: IGeesomeAsyncOperationModule; + asset: IGeesomeAssetModule; staticId: IGeesomeStaticIdModule; invite: IGeesomeInviteModule; group: IGeesomeGroupModule; @@ -281,6 +283,7 @@ export interface IUserApiKeyInput { title?: string; type?: string; permissions?: string; + scopes?: string[] | string; expiredOn?: Date | string; isDisabled?: boolean; } diff --git a/app/modules/api/api.ts b/app/modules/api/api.ts index 9a6627cc..573500fe 100644 --- a/app/modules/api/api.ts +++ b/app/modules/api/api.ts @@ -5,6 +5,7 @@ import IGeesomeApiModule from "./interface.js"; import {CorePermissionName} from "../database/interface.js"; import {IGeesomeApp} from "../../interface.js"; import {sendBadGatewayOnStorageRouteError, sendForbiddenOnAuthRouteError} from "./routeErrorHelpers.js"; +import {serializeApiKey} from './integrationScopes.js'; const {isNumber} = _; const log = debug('geesome:api:routes'); @@ -259,7 +260,22 @@ export default (app: IGeesomeApp, module: IGeesomeApiModule) => { * @apiInterface (../database/interface.ts) {IUserApiKey} apiSuccess */ module.onAuthorizedGet('user/api-key/current', async (req, res) => { - res.send(req.apiKey); + res.send(serializeApiKey(req.apiKey)); + }); + + /** + * @api {get} /v1/integrations/credentials/current Inspect current integration credential + * @apiName IntegrationCredentialCurrent + * @apiGroup UserApiKey + * @apiUse ApiKey + * @apiSuccess {Number} id API-key identifier. + * @apiSuccess {String[]} scopes Granted integration scopes. + * @apiSuccess {Date} [expiresAt] Expiration time. + * @apiSuccess {Date} [lastUsedAt] Last recorded use time. + * @apiSuccess {Boolean} revoked Revocation state. + */ + module.onAuthorizedGet('integrations/credentials/current', async (req, res) => { + res.send(serializeApiKey(req.apiKey)); }); /** diff --git a/app/modules/api/index.ts b/app/modules/api/index.ts index 1de51e73..36b109d2 100644 --- a/app/modules/api/index.ts +++ b/app/modules/api/index.ts @@ -10,6 +10,8 @@ import {closeHttpServer} from '../../httpServer.js'; import {buildOpenApiFromApiDoc, getApiDocData} from "../../apiDocSpec.js"; import {IUser} from "../database/interface.js"; import {cleanupAndRethrow} from '../../resourceCleanup.js'; +import {ApiProblemError, getRequestId, sendApiProblem, sendResponse} from './problem.js'; +import {getPublicApiContext} from './publicUrls.js'; import IGeesomeApiModule, { IApiModuleCommonOutput, IApiModuleGetInput, @@ -18,7 +20,7 @@ import IGeesomeApiModule, { const {trimStart} = _; export default async (app: IGeesomeApp, options: any = {}) => { - const module = await getModule(app, 'v1', options.port || process.env.PORT || app.config.port || 2052); + const module = await getModule(app, 'v1', options.port || process.env.PORT || app.config.port || 2052, options.host || '0.0.0.0'); try { await (options.registerRoutes || registerApiRoutes)(app, module); return module; @@ -27,7 +29,7 @@ export default async (app: IGeesomeApp, options: any = {}) => { } } -async function getModule(app: IGeesomeApp, version, port) { +async function getModule(app: IGeesomeApp, version, port, host) { const service = express(); // Registry of routes for the discovery index (GET /v1) and OpenAPI spec. @@ -55,6 +57,10 @@ async function getModule(app: IGeesomeApp, version, port) { service.use(morgan('combined')); } service.use((req, res, next) => { + const requestId = getRequestId(req.headers['x-request-id'] as string); + req.requestId = requestId; + res.locals.requestId = requestId; + res.setHeader('X-Request-Id', requestId); trackRuntimeHttpRequest('api', req, res); next(); }); @@ -87,7 +93,7 @@ async function getModule(app: IGeesomeApp, version, port) { next(); }); - const server = await service.listen(port); + const server = await service.listen(port, host); let stopPromise: Promise | null = null; function setHeaders(res) { @@ -95,12 +101,13 @@ async function getModule(app: IGeesomeApp, version, port) { res.setHeader('Access-Control-Allow-Credentials', 'true'); res.setHeader('Access-Control-Allow-Origin', '*'); res.setHeader('Access-Control-Allow-Methods', "GET, POST, PATCH, PUT, DELETE, OPTIONS, HEAD"); - res.setHeader('Access-Control-Allow-Headers', 'Content-Type, Authorization, Content-Length, X-Requested-With'); + res.setHeader('Access-Control-Allow-Headers', 'Content-Type, Authorization, Content-Length, Idempotency-Key, X-Request-Id, X-Requested-With'); + res.setHeader('Access-Control-Expose-Headers', 'Content-Digest, ETag, Link, Location, Retry-After, X-Geesome-Storage-Id, X-Request-Id'); res.setHeader('Connection', 'close'); //TODO: determine the best solution https://serverfault.com/questions/708319/chrome-requests-get-stuck-pending // Point clients/agents at the API docs. Use the IPFS path (served at /ipfs/ // regardless of any reverse-proxy prefix) so it is unambiguous; the JSON // discovery index is reachable at the API base root (GET /{version}). - const docsLinks = buildDocsDiscoveryLinks(version, app.docsStorageId); + const docsLinks = buildDocsDiscoveryLinks(version, app.docsStorageId, getPublicApiContext(app, version, port)); setDocsHeaders(res, docsLinks); } @@ -119,12 +126,7 @@ async function getModule(app: IGeesomeApp, version, port) { if (isResponseClosed(res)) { return; } - const statusCode = Number.isInteger(callbackError?.code) ? callbackError.code : 500; - const body = {message: callbackError.message || callbackError, errorCode: 3}; - if (typeof res.status === 'function') { - return res.status(statusCode).send(body); - } - return res.send(body, statusCode); + return sendApiProblem(res, callbackError, req.requestId); } } @@ -166,13 +168,13 @@ async function getModule(app: IGeesomeApp, version, port) { async authorizeAndHandleCallback(req: IApiModuleGetInput, res: IApiModuleCommonOutput, callback) { if (!req.token) { - return res.send({error: "Need authorization token", errorCode: 1}, 401); + return sendApiProblem(res, new ApiProblemError(401, 'credentials_required', 'Credentials required', 'Supply a bearer token in the Authorization header.'), req.requestId); } const {user, apiKey} = await app.getUserByApiToken(req.token); req.user = user; req.apiKey = apiKey; if (!req.user || !req.user.id) { - return res.send({error: "Not authorized", errorCode: 2}, 401); + return sendApiProblem(res, new ApiProblemError(401, 'invalid_credentials', 'Invalid credentials', 'The supplied bearer token is invalid, expired, or revoked.'), req.requestId); } return app.runWithApiKey(apiKey, () => this.handleCallback(req, res, callback)); } @@ -271,6 +273,7 @@ async function getModule(app: IGeesomeApp, version, port) { query: req.query, route: req.url.replace(version + '/', ''), fullRoute: req.originalUrl.replace(version + '/', ''), + requestId: req.requestId || req.locals?.requestId, stream: req }; if (req.body) { @@ -286,11 +289,19 @@ async function getModule(app: IGeesomeApp, version, port) { if (res.stream) { return res; } + const sendWithStatus = (data, status?) => { + if (status === undefined && typeof data === 'number' && data >= 400 && data <= 599) { + return res.status(data).send(); + } + return sendResponse(res, data, status); + }; return { - send: res.send.bind(res), + send: sendWithStatus, + sendWithStatus, setHeader: res.setHeader.bind(res), writeHead: res.writeHead.bind(res), stream: res, + requestId: res.locals?.requestId, }; } @@ -356,10 +367,7 @@ async function getModule(app: IGeesomeApp, version, port) { version, description: 'Operation/route map generated from the live node. Full parameter and response details (apiDoc) are published to IPFS on each boot — see x-docs-ipfs and the GET /' + version + ' discovery index.', }, - servers: [ - {url: `/${version}`, description: 'Direct node'}, - {url: `/api/${version}`, description: 'Behind the bundled nginx reverse proxy'}, - ], + servers: [{url: getPublicApiContext(app, version, port).apiBaseUrl, description: 'Advertised public API'}], 'x-docs-ipfs': app.docsStorageId ? `/ipfs/${app.docsStorageId}` : null, components: {securitySchemes: {bearerAuth: {type: 'http', scheme: 'bearer'}}}, paths, @@ -368,7 +376,10 @@ async function getModule(app: IGeesomeApp, version, port) { // Primary spec is generated from the node's apiDoc annotations (full param // schemas); fall back to the route-registry map if apiDoc parsing is // unavailable at runtime. - const openapiHandler = (req, res) => res.send(buildOpenApiFromApiDoc(version, app.docsStorageId) || buildOpenApi()); + const openapiHandler = (req, res) => { + const publicApiBaseUrl = getPublicApiContext(app, version, port).apiBaseUrl; + return res.send(buildOpenApiFromApiDoc(version, app.docsStorageId, publicApiBaseUrl) || buildOpenApi()); + }; apiModule.onGet('openapi.json', openapiHandler); // Raw apiDoc data (native format) for clients that prefer it. apiModule.onGet('apidoc.json', (req, res) => res.send(getApiDocData() || [])); @@ -376,14 +387,40 @@ async function getModule(app: IGeesomeApp, version, port) { // they return the real spec instead of being shadowed by the frontend SPA. ['openapi.json', 'swagger.json', 'api-docs.json', '.well-known/openapi.json'].forEach((p) => apiModule.onUnversionGet(p, openapiHandler)); + /** + * @api {get} /.well-known/geesome Discover GeeSome public API + * @apiName GeesomeDiscovery + * @apiGroup Discovery + * @apiDescription Returns absolute public API, gateway, documentation, health, capability, compatibility, and storage-characteristic links for automated clients. + * @apiSuccess {Number} schemaVersion Discovery schema version. + * @apiSuccess {String} product Product identifier. + * @apiSuccess {String} apiVersion API version. + * @apiSuccess {String} apiBaseUrl Absolute public API base URL. + * @apiSuccess {String} gatewayBaseUrl Absolute public gateway base URL. + * @apiSuccess {String} openapiUrl Absolute OpenAPI URL. + */ + const wellKnownDiscoveryHandler = (req, res) => res.send(buildPublicDiscovery(app, version, port)); + apiModule.onUnversionGet('.well-known/geesome', wellKnownDiscoveryHandler); + + /** + * @api {get} /v1/health Get API health + * @apiName ApiHealth + * @apiGroup Discovery + * @apiSuccess {Boolean} ok Whether the HTTP API is serving requests. + * @apiSuccess {String} apiVersion API version. + */ + apiModule.onGet('health', (req, res) => res.send({ok: true, apiVersion: version})); + // Machine-readable discovery index so an agent with only the node URL can find // the route map and the published API docs. Served at GET /{version} and // /{version}/ (e.g. /api/v1 behind nginx). Fast JSON, never the SPA shell. const discoveryHandler = (req, res) => { - const docsLinks = buildDocsDiscoveryLinks(version, app.docsStorageId); + const publicDiscovery = buildPublicDiscovery(app, version, port); + const docsLinks = buildDocsDiscoveryLinks(version, app.docsStorageId, getPublicApiContext(app, version, port)); return res.send({ name: 'geesome-node', version, + publicDiscovery, docs: { description: 'Full API reference (apiDoc) is generated and published to IPFS on each node boot.', discovery: docsLinks.discovery, @@ -408,30 +445,78 @@ async function getModule(app: IGeesomeApp, version, port) { return apiModule; } -function buildDocsDiscoveryLinks(version: string, docsStorageId?: string) { +function buildDocsDiscoveryLinks(version: string, docsStorageId?: string, publicContext?: any) { const repo = 'https://github.com/galtproject/geesome-node'; const docsRepoRoot = `${repo}/tree/master/docs`; const docsRepoBlob = `${repo}/blob/master/docs`; - const ipfsRoot = docsStorageId ? `/ipfs/${docsStorageId}` : null; + const origin = publicContext?.publicUrl || ''; + const apiBaseUrl = publicContext?.apiBaseUrl || `${origin}/${version}`; + const ipfsRoot = docsStorageId ? `${origin}/ipfs/${docsStorageId}` : null; return { repo, - discovery: `/${version}`, - openapi: `/${version}/openapi.json`, - apidoc: `/${version}/apidoc.json`, + discovery: `${apiBaseUrl}`, + openapi: `${apiBaseUrl}/openapi.json`, + apidoc: `${apiBaseUrl}/apidoc.json`, apiHtml: ipfsRoot || docsRepoRoot, repoDocs: ipfsRoot ? `${ipfsRoot}/README.md` : `${docsRepoBlob}/README.md`, moduleDocs: ipfsRoot ? `${ipfsRoot}/modules.md` : `${docsRepoBlob}/modules.md`, agentMap: ipfsRoot ? `${ipfsRoot}/agent-map.md` : `${docsRepoBlob}/agent-map.md`, ipfsRoot, conventionalOpenapi: { - openapi: '/openapi.json', - swagger: '/swagger.json', - apiDocs: '/api-docs.json', - wellKnown: '/.well-known/openapi.json', + openapi: `${origin}/openapi.json`, + swagger: `${origin}/swagger.json`, + apiDocs: `${origin}/api-docs.json`, + wellKnown: `${origin}/.well-known/openapi.json`, + }, + }; +} + +function buildPublicDiscovery(app: IGeesomeApp, version: string, port: number | string) { + const context = getPublicApiContext(app, version, port); + const docsLinks = buildDocsDiscoveryLinks(version, app.docsStorageId, context); + const maxUploadBytes = parsePositiveNumber(app.config?.apiConfig?.maxUploadBytes); + return { + schemaVersion: 1, + product: 'geesome', + deploymentVersion: app.config?.apiConfig?.deploymentVersion || 'unknown', + apiVersion: version, + apiBaseUrl: context.apiBaseUrl, + gatewayBaseUrl: context.publicUrl, + openapiUrl: docsLinks.openapi, + docsUrl: docsLinks.apiHtml, + healthUrl: `${context.apiBaseUrl}/health`, + capabilities: { + contentUpload: Boolean(app.ms.content), + rawContentUpload: Boolean(app.ms.content), + assetUpload: Boolean(app.ms['asset']), + asyncOperations: Boolean(app.ms.asyncOperation), + batchContentUpload: Boolean(app.ms['asset']?.supportsBatches) + }, + limits: { + maxUploadBytes + }, + storage: { + identity: 'cid', + immediateRead: true, + pinPolicy: app.ms['pin'] ? 'deployment-configured' : 'local-storage', + rangeRequests: true, + contentDigest: 'sha-256' }, + compatibility: { + changelogUrl: 'https://github.com/galtproject/geesome-node/commits/dev', + migrationNotesUrl: `${docsLinks.repoDocs.replace(/README\.md$/, 'implemented.md')}` + } }; } +function parsePositiveNumber(value): number | null { + const parsed = Number(value); + if (!Number.isFinite(parsed) || parsed <= 0) { + return null; + } + return Math.floor(parsed); +} + function setDocsHeaders(res, docsLinks) { res.setHeader('X-Api-Docs', docsLinks.apiHtml); res.setHeader('X-Api-Docs-Openapi', docsLinks.openapi); diff --git a/app/modules/api/integrationScopes.ts b/app/modules/api/integrationScopes.ts new file mode 100644 index 00000000..9955ef5c --- /dev/null +++ b/app/modules/api/integrationScopes.ts @@ -0,0 +1,82 @@ +import {CorePermissionName, IUserApiKey} from '../database/interface.js'; +import {ApiProblemError} from './problem.js'; + +const SCOPE_PERMISSION_MAP = { + 'assets:write': CorePermissionName.UserSaveData, + 'assets:read-private': CorePermissionName.UserSaveData, + 'content:write': CorePermissionName.UserSaveData, + 'operations:read': CorePermissionName.UserSaveData, + 'asset-batches:write': CorePermissionName.UserSaveData +}; + +export function normalizeIntegrationScopes(value: any): string[] { + const scopes = parseStringList(value); + for (const scope of scopes) { + if (!SCOPE_PERMISSION_MAP[scope]) { + throw new ApiProblemError(400, 'integration_scope_invalid', 'Invalid integration scope', `Unknown integration scope: ${scope}`); + } + } + return Array.from(new Set(scopes)).sort(); +} + +export function getScopePermissions(scopes: string[]): string[] { + return Array.from(new Set(scopes.map(scope => SCOPE_PERMISSION_MAP[scope]))); +} + +export function getApiKeyScopes(apiKey: IUserApiKey): string[] { + const explicit = parseStringList(apiKey?.scopes); + if (explicit.length) { + return explicit.sort(); + } + const permissions = parseStringList(apiKey?.permissions); + return Object.entries(SCOPE_PERMISSION_MAP) + .filter(([, permission]) => permissions.includes(permission) || permissions.includes(CorePermissionName.UserAll)) + .map(([scope]) => scope) + .sort(); +} + +export function requireIntegrationScopes(apiKey: IUserApiKey, requiredScopes: string[]) { + const granted = getApiKeyScopes(apiKey); + const missing = requiredScopes.filter(scope => !granted.includes(scope)); + if (missing.length) { + throw new ApiProblemError(403, 'insufficient_scope', 'Insufficient scope', 'The API key does not grant every scope required by this operation.', { + requiredScopes, + grantedScopes: granted + }); + } +} + +export function serializeApiKey(apiKey: any) { + const data = typeof apiKey?.toJSON === 'function' ? apiKey.toJSON() : apiKey || {}; + return { + id: data.id, + title: data.title || null, + type: data.type || null, + scopes: getApiKeyScopes(data), + createdAt: data.createdAt || null, + updatedAt: data.updatedAt || null, + lastUsedAt: data.lastUsedAt || null, + expiresAt: data.expiredOn || null, + revoked: data.isDisabled === true + }; +} + +function parseStringList(value: any): string[] { + if (Array.isArray(value)) { + return value.map(item => String(item).trim()).filter(Boolean); + } + if (!value) { + return []; + } + if (typeof value === 'string') { + try { + const parsed = JSON.parse(value); + if (Array.isArray(parsed)) { + return parsed.map(item => String(item).trim()).filter(Boolean); + } + } catch (e) { + return value.split(/[\s,]+/).map(item => item.trim()).filter(Boolean); + } + } + return []; +} diff --git a/app/modules/api/interface.ts b/app/modules/api/interface.ts index 7e8f8f9b..a5010b62 100644 --- a/app/modules/api/interface.ts +++ b/app/modules/api/interface.ts @@ -37,9 +37,11 @@ export default interface IGeesomeApiModule { export interface IApiModuleCommonOutput { send: (data: any, status?: number) => any; + sendWithStatus?: (data: any, status?: number) => any; setHeader: (name: string, value: string) => any; writeHead: (status: number, data: any) => any; stream: Stream; + requestId?: string; } export interface IApiModuleCommonInput { @@ -52,6 +54,7 @@ export interface IApiModuleCommonInput { apiKey?: IUserApiKey; query?: any; rawBody?: Buffer; + requestId?: string; stream: Stream; } diff --git a/app/modules/api/problem.ts b/app/modules/api/problem.ts new file mode 100644 index 00000000..950943d2 --- /dev/null +++ b/app/modules/api/problem.ts @@ -0,0 +1,154 @@ +import {randomUUID} from 'node:crypto'; + +export const PROBLEM_CONTENT_TYPE = 'application/problem+json'; + +export class ApiProblemError extends Error { + status: number; + code: string; + title: string; + extensions: Record; + + constructor(status: number, code: string, title: string, detail?: string, extensions: Record = {}) { + super(detail || title); + this.name = 'ApiProblemError'; + this.status = status; + this.code = code; + this.title = title; + this.extensions = extensions; + } +} + +export function getRequestId(value?: string): string { + const candidate = String(value || '').trim(); + if (candidate && candidate.length <= 128 && /^[A-Za-z0-9._:-]+$/.test(candidate)) { + return candidate; + } + return `req_${randomUUID()}`; +} + +export function getApiProblem(error: any, requestId?: string) { + const normalized = normalizeApiError(error); + return { + type: `https://github.com/galtproject/geesome-node/blob/dev/docs/api-problems.md#${normalized.code}`, + title: normalized.title, + status: normalized.status, + code: normalized.code, + detail: normalized.detail, + requestId: requestId || null, + ...normalized.extensions + }; +} + +export function sendApiProblem(res: any, error: any, requestId?: string) { + const problem = getApiProblem(error, requestId || res.requestId || res.locals?.requestId); + setResponseHeader(res, 'Content-Type', PROBLEM_CONTENT_TYPE); + setResponseHeader(res, 'X-Request-Id', problem.requestId); + return sendResponse(res, problem, problem.status); +} + +export function sendResponse(res: any, data: any, status?: number) { + if (typeof res.sendWithStatus === 'function') { + return res.sendWithStatus(data, status); + } + if (status && typeof res.status === 'function') { + return res.status(status).send(data); + } + if (typeof res.send === 'function') { + if (status !== undefined) { + return res.send(data, status); + } + return res.send(data); + } +} + +function normalizeApiError(error: any) { + if (error instanceof ApiProblemError) { + return { + status: error.status, + code: error.code, + title: error.title, + detail: error.message, + extensions: error.extensions + }; + } + + const message = getErrorMessage(error); + const explicitStatus = getExplicitStatus(error); + if (explicitStatus) { + return getMappedError(explicitStatus, getStableCode(error, message), message); + } + if (message === 'not_permitted') { + return getMappedError(403, 'forbidden', 'The authenticated principal is not allowed to perform this action.'); + } + if (message === 'not_authorized') { + return getMappedError(401, 'invalid_credentials', 'The supplied credentials are invalid or inactive.'); + } + if (message.includes('limit_reached') || message.includes('files_limit') || message.includes('parts_limit')) { + return getMappedError(413, 'upload_limit_exceeded', 'The request exceeds the configured upload limit.'); + } + if (message.includes('not_found')) { + return getMappedError(404, getStableCode(error, message), 'The requested resource was not found.'); + } + return getMappedError(500, 'internal_error', 'The server could not complete the request.'); +} + +function getMappedError(status: number, code: string, detail: string) { + return { + status, + code, + title: getStatusTitle(status), + detail, + extensions: {} + }; +} + +function getExplicitStatus(error: any): number | null { + const value = Number(error?.status || error?.statusCode || error?.httpStatus || error?.code); + if (Number.isInteger(value) && value >= 400 && value <= 599) { + return value; + } + return null; +} + +function getStableCode(error: any, message: string): string { + const candidate = String(error?.errorCode || error?.problemCode || error?.code || message || '').trim(); + if (/^[a-z][a-z0-9_]{1,79}$/.test(candidate)) { + return candidate; + } + return 'request_failed'; +} + +function getStatusTitle(status: number): string { + const titles = { + 400: 'Bad request', + 401: 'Unauthorized', + 403: 'Forbidden', + 404: 'Not found', + 409: 'Conflict', + 413: 'Content too large', + 422: 'Unprocessable content', + 423: 'Locked', + 429: 'Too many requests', + 500: 'Internal server error', + 502: 'Bad gateway', + 503: 'Service unavailable' + }; + return titles[status] || 'Request failed'; +} + +function getErrorMessage(error: any): string { + if (error?.message) { + return String(error.message); + } + if (typeof error === 'string') { + return error; + } + return 'request_failed'; +} + +function setResponseHeader(res: any, name: string, value: any) { + if (value === undefined || value === null || typeof res.setHeader !== 'function') { + return; + } + res.setHeader(name, String(value)); +} diff --git a/app/modules/api/publicUrls.ts b/app/modules/api/publicUrls.ts new file mode 100644 index 00000000..0330496d --- /dev/null +++ b/app/modules/api/publicUrls.ts @@ -0,0 +1,28 @@ +import {IGeesomeApp} from '../../interface.js'; + +export function getPublicApiContext(app: IGeesomeApp, version = 'v1', port?: number | string) { + const configuredValue = app.config?.apiConfig?.publicUrl; + const configuredUrl = normalizePublicUrl(configuredValue); + if (configuredValue && !configuredUrl) { + throw new Error('GEESOME_PUBLIC_URL must be an absolute http or https URL without credentials, query, or fragment'); + } + const publicUrl = configuredUrl || `http://127.0.0.1:${port || app.ms.api?.port || 2052}`; + const configuredBasePath = String(app.config?.apiConfig?.publicBasePath || `/api/${version}`); + const basePath = configuredBasePath.replace(/\{version\}/g, version).replace(/\/+$/, ''); + return { + publicUrl, + apiBaseUrl: `${publicUrl}${basePath.startsWith('/') ? '' : '/'}${basePath}` + }; +} + +function normalizePublicUrl(value): string | null { + try { + const url = new URL(String(value || '').trim()); + if (!['http:', 'https:'].includes(url.protocol) || url.username || url.password || url.search || url.hash) { + return null; + } + return `${url.protocol}//${url.host}${url.pathname.replace(/\/+$/, '')}`; + } catch (e) { + return null; + } +} diff --git a/app/modules/api/routeErrorHelpers.ts b/app/modules/api/routeErrorHelpers.ts index daa845a5..8198be6b 100644 --- a/app/modules/api/routeErrorHelpers.ts +++ b/app/modules/api/routeErrorHelpers.ts @@ -1,5 +1,6 @@ import helpers from "../../helpers.js"; import {IApiModuleCommonOutput} from "./interface.js"; +import {ApiProblemError, sendApiProblem} from './problem.js'; type DebugLog = { enabled: boolean; @@ -15,7 +16,7 @@ export function sendBadRequestOnContentRouteError(log: DebugLog, res: IApiModule error: getErrorMessage(error) } ]); - res.send(400); + sendApiProblem(res, new ApiProblemError(400, 'content_request_invalid', 'Invalid content request', 'The content path or request options are invalid.')); }; } @@ -28,7 +29,7 @@ export function sendForbiddenOnAuthRouteError(log: DebugLog, res: IApiModuleComm error: getErrorMessage(error) } ]); - res.send(403); + sendApiProblem(res, new ApiProblemError(403, 'forbidden', 'Forbidden', 'The authenticated principal is not allowed to perform this action.')); }; } @@ -41,7 +42,7 @@ export function sendBadGatewayOnStorageRouteError(log: DebugLog, res: IApiModule error: getErrorMessage(error) } ]); - res.send(null, 502); + sendApiProblem(res, new ApiProblemError(502, 'storage_backend_unavailable', 'Storage backend unavailable', 'The storage backend did not complete the request.')); }; } diff --git a/app/modules/asset/api.ts b/app/modules/asset/api.ts new file mode 100644 index 00000000..4e928ab4 --- /dev/null +++ b/app/modules/asset/api.ts @@ -0,0 +1,185 @@ +import {IGeesomeApp} from '../../interface.js'; +import {ApiProblemError} from '../api/problem.js'; +import {requireIntegrationScopes} from '../api/integrationScopes.js'; +import {getPublicApiContext} from '../api/publicUrls.js'; +import asyncBusboy from '../content/asyncBusboy.js'; +import IGeesomeAssetModule from './interface.js'; +import { + getAssetRequestHash, + validateBytes, + validateLogicalPath, + validateMimeType, + validateSha256 +} from './index.js'; + +export default (app: IGeesomeApp, assetModule: IGeesomeAssetModule) => { + /** + * @api {post} /v1/assets Upload immutable asset + * @apiName AssetCreate + * @apiGroup Assets + * @apiUse ApiKey + * @apiHeader {String} Idempotency-Key Owner-scoped retry key. + * @apiBody {File} file Asset bytes. + * @apiBody {String} expectedSha256 Lowercase SHA-256 hex digest. + * @apiBody {String} [logicalPath] Optional release metadata path. + * @apiBody {String="none","standard"} [previewPolicy="none"] Preview generation policy. + * @apiBody {Boolean} [async=false] Return an operation resource while processing. + * @apiSuccess (201) {String} storageId Immutable storage CID. + * @apiSuccess (201) {String} sha256 Verified byte digest. + * @apiSuccess (201) {Object} urls Public content and metadata URLs. + * @apiUse AuthErrors + * @apiUse UploadErrors + */ + app.ms.api.onAuthorizedPost('assets', async (req, res) => { + requireIntegrationScopes(req.apiKey, ['assets:write']); + return handleAssetCreate(app, assetModule, req, res); + }); + + /** + * @api {get} /v1/assets/:storageId Get immutable asset metadata + * @apiName AssetGet + * @apiGroup Assets + * @apiUse ApiKey + * @apiParam {String} storageId Immutable storage CID. + * @apiSuccess {String} storageId Immutable storage CID. + * @apiSuccess {String} sha256 Verified byte digest. + */ + app.ms.api.onAuthorizedGet('assets/:storageId', async (req, res) => { + requireIntegrationScopes(req.apiKey, ['assets:read-private']); + res.send(await assetModule.getAsset(req.user.id, req.params.storageId)); + }); + + /** + * @api {post} /v1/asset-batches Create resumable asset batch + * @apiName AssetBatchCreate + * @apiGroup AssetBatches + * @apiUse ApiKey + * @apiHeader {String} Idempotency-Key Owner-scoped batch retry key. + * @apiBody {Object[]} items Expected manifest items. + * @apiSuccess {Number} batchId Batch identifier. + * @apiSuccess {Object[]} items Batch item upload requirements. + */ + app.ms.api.onAuthorizedPost('asset-batches', async (req, res) => { + requireIntegrationScopes(req.apiKey, ['asset-batches:write']); + const key = String(req.headers['idempotency-key'] || '').trim(); + res.send(await assetModule.createBatch(req.user.id, req.body, key), 201); + }); + + /** + * @api {get} /v1/asset-batches/:batchId Get asset batch + * @apiName AssetBatchGet + * @apiGroup AssetBatches + * @apiUse ApiKey + * @apiParam {Number} batchId Batch identifier. + */ + app.ms.api.onAuthorizedGet('asset-batches/:batchId', async (req, res) => { + requireIntegrationScopes(req.apiKey, ['asset-batches:write']); + res.send(await assetModule.getBatch(req.user.id, Number(req.params.batchId))); + }); + + /** + * @api {post} /v1/asset-batches/:batchId/complete Complete asset batch + * @apiName AssetBatchComplete + * @apiGroup AssetBatches + * @apiUse ApiKey + * @apiParam {Number} batchId Batch identifier. + * @apiSuccess {String="completed"} status Completed status. + * @apiSuccess {Object} manifest Immutable hash-bound manifest. + */ + app.ms.api.onAuthorizedPost('asset-batches/:batchId/complete', async (req, res) => { + requireIntegrationScopes(req.apiKey, ['asset-batches:write']); + res.send(await assetModule.completeBatch(req.user.id, Number(req.params.batchId), req.apiKey.id)); + }); +}; + +async function handleAssetCreate(app: IGeesomeApp, assetModule: IGeesomeAssetModule, req: any, res: any) { + const maxUploadBytes = Number(app.config?.apiConfig?.maxUploadBytes) || 2000 * 1024 * 1024; + const {files, fields} = await asyncBusboy(req.stream, {headers: req.headers, limits: {files: 1, fileSize: maxUploadBytes}}); + if (files.length !== 1) { + throw new ApiProblemError(400, 'asset_file_required', 'One asset file is required'); + } + const file = files[0]; + let handedToAssetModule = false; + try { + const sha256 = validateSha256(fields.expectedSha256); + if (file.sha256 !== sha256) { + throw new ApiProblemError(422, 'asset_digest_mismatch', 'Asset digest mismatch', 'The uploaded bytes do not match expectedSha256.', {expectedSha256: sha256, actualSha256: file.sha256}); + } + const bytes = validateBytes(file.bytes); + const mimeType = validateMimeType(file.mimeType); + const logicalPath = validateLogicalPath(fields.logicalPath); + const previewPolicy = validatePreviewPolicy(fields.previewPolicy); + const batchId = parseOptionalPositiveInteger(fields.batchId, 'asset_batch_id_invalid'); + const logicalId = fields.logicalId ? String(fields.logicalId).trim() : null; + const idempotencyKey = String(req.headers['idempotency-key'] || '').trim(); + const requestHash = getAssetRequestHash({sha256, bytes, mimeType, logicalPath, previewPolicy, batchId, logicalId}); + const prepared = await assetModule.prepareAssetRequest(req.user.id, idempotencyKey, requestHash); + if (prepared.asset) { + return res.send({...serializeExisting(assetModule, prepared.asset), created: false}, 200); + } + if (prepared.operationId) { + return sendOperation(res, prepared.operationId, app); + } + const options = { + userId: req.user.id, + userApiKeyId: req.apiKey.id, + idempotencyRequestId: prepared.request.id, + sha256, + bytes, + mimeType, + logicalPath, + previewPolicy, + batchId, + logicalId, + requestId: req.requestId, + async: isTrue(fields.async) + }; + if (options.async) { + handedToAssetModule = true; + const operation = await app.ms.asyncOperation.asyncOperationWrapper('asset', 'createAssetFromUpload', [req.user.id, file, file.filename, options], options); + await assetModule.linkAssetOperation(prepared.request.id, operation.asyncOperationId); + return sendOperation(res, operation.asyncOperationId, app); + } + handedToAssetModule = true; + const asset = await assetModule.createAssetFromUpload(req.user.id, file, file.filename, options); + return res.send(asset, 201); + } finally { + if (!handedToAssetModule) { + file.emitFinish?.(); + } + } +} + +function validatePreviewPolicy(value): string { + const policy = String(value || 'none').trim(); + if (!['none', 'standard'].includes(policy)) { + throw new ApiProblemError(400, 'asset_preview_policy_invalid', 'Invalid preview policy'); + } + return policy; +} + +function parseOptionalPositiveInteger(value, code: string): number | null { + if (value === undefined || value === null || value === '') { + return null; + } + const parsed = Number(value); + if (!Number.isSafeInteger(parsed) || parsed <= 0) { + throw new ApiProblemError(400, code, 'Invalid identifier'); + } + return parsed; +} + +function isTrue(value): boolean { + return value === true || value === 'true' || value === '1'; +} + +function sendOperation(res, operationId: number, app: IGeesomeApp) { + const statusUrl = `${getPublicApiContext(app).apiBaseUrl}/operations/${operationId}`; + res.setHeader('Location', statusUrl); + res.setHeader('Retry-After', '2'); + return res.send({schemaVersion: 1, operationId: `op_${operationId}`, status: 'pending', statusUrl}, 202); +} + +function serializeExisting(assetModule: any, asset: any) { + return assetModule.serializeAsset(asset, false); +} diff --git a/app/modules/asset/docs/README.md b/app/modules/asset/docs/README.md new file mode 100644 index 00000000..35a9194a --- /dev/null +++ b/app/modules/asset/docs/README.md @@ -0,0 +1,12 @@ +# Asset Module + +The asset module is the stable integration façade over existing content and +storage modules. It verifies SHA-256 and byte length before publication, +persists owner-scoped idempotency, exposes asset metadata, supports resumable +batches, and stores a deterministic immutable batch manifest. + +It does not duplicate bytes or reveal global hash existence. Reuse during batch +preflight is limited to assets owned by the authenticated user. + +Public clients should bootstrap through `GET /.well-known/geesome` and use the +advertised `apiBaseUrl` and `openapiUrl`. diff --git a/app/modules/asset/index.ts b/app/modules/asset/index.ts new file mode 100644 index 00000000..09c8546a --- /dev/null +++ b/app/modules/asset/index.ts @@ -0,0 +1,396 @@ +import {createHash} from 'node:crypto'; +import {Op} from 'sequelize'; +import {IGeesomeApp} from '../../interface.js'; +import {ApiProblemError} from '../api/problem.js'; +import {getPublicApiContext} from '../api/publicUrls.js'; +import IGeesomeAssetModule from './interface.js'; + +export default async (app: IGeesomeApp) => { + app.checkModules(['api', 'database', 'content', 'asyncOperation']); + const database: any = app.ms.database; + const models = await (await import('./models.js')).default(database.sequelize, database.models); + const module = getModule(app, models); + (await import('./api.js')).default(app, module); + return module; +}; + +export function getModule(app: IGeesomeApp, models: any): IGeesomeAssetModule { + class AssetModule implements IGeesomeAssetModule { + supportsBatches = true; + + async flushDatabase() { + await models.AssetBatchItem.destroy({where: {}}); + await models.AssetBatch.destroy({where: {}}); + await models.AssetIdempotencyKey.destroy({where: {}}); + await models.Asset.destroy({where: {}}); + } + + async prepareAssetRequest(userId: number, key: string, requestHash: string) { + validateIdempotencyKey(key); + const [request, created] = await models.AssetIdempotencyKey.findOrCreate({ + where: {userId, namespace: 'assets', key}, + defaults: {userId, namespace: 'assets', key, requestHash, status: 'pending'} + }); + if (created) { + return {request, created: true}; + } + if (request.requestHash !== requestHash) { + throw new ApiProblemError(409, 'idempotency_key_conflict', 'Idempotency key conflict', 'The idempotency key was already used for a different asset request.'); + } + if (request.status === 'succeeded' && request.assetId) { + const asset = await models.Asset.findByPk(request.assetId); + return {request, created: false, asset}; + } + if (request.asyncOperationId) { + return {request, created: false, operationId: request.asyncOperationId}; + } + if (request.status === 'failed') { + await request.update({status: 'pending', errorCode: null}); + return {request, created: true}; + } + throw new ApiProblemError(409, 'idempotency_request_in_progress', 'Request in progress', 'An asset request with this idempotency key is already in progress.'); + } + + async createAssetFromUpload(userId: number, data: any, fileName: string, options: any) { + const request = await models.AssetIdempotencyKey.findOne({where: {id: options.idempotencyRequestId, userId}}); + if (!request) { + throw new ApiProblemError(409, 'idempotency_request_not_found', 'Idempotency request not found'); + } + try { + const content = await app.ms.content.saveData(userId, data, fileName, { + userApiKeyId: options.userApiKeyId, + driver: options.previewPolicy === 'standard' ? undefined : {raw: true}, + skipFileCatalog: true, + mimeType: options.mimeType, + properties: { + assetSha256: options.sha256, + assetLogicalPath: options.logicalPath || null + } + }); + await setStorageObjectSha256(app, content.storageId, options.sha256); + const pinStatus = await getAssetPinStatus(app, content); + const [asset, created] = await models.Asset.findOrCreate({ + where: {userId, storageId: content.storageId}, + defaults: { + userId, + contentId: content.id, + storageId: content.storageId, + sha256: options.sha256, + bytes: options.bytes, + mimeType: options.mimeType || content.mimeType || 'application/octet-stream', + logicalPath: options.logicalPath || null, + pinStatus + } + }); + if (!created) { + await asset.update({sha256: options.sha256, bytes: options.bytes, pinStatus}); + } + await this.attachAssetToBatch(userId, asset, options.batchId, options.logicalId); + await request.update({status: 'succeeded', assetId: asset.id, errorCode: null}); + return {...this.serializeAsset(asset, created), contentId: content.id}; + } catch (error) { + await request.update({status: 'failed', errorCode: getErrorCode(error)}); + throw error; + } finally { + data?.emitFinish?.(); + } + } + + async getAsset(userId: number, storageId: string) { + const asset = await models.Asset.findOne({where: {userId, storageId}}); + if (!asset) { + throw new ApiProblemError(404, 'asset_not_found', 'Asset not found', 'No asset visible to the authenticated user has this storage ID.'); + } + return this.serializeAsset(asset, false); + } + + async linkAssetOperation(requestId: number, operationId: number) { + await models.AssetIdempotencyKey.update({asyncOperationId: operationId}, {where: {id: requestId}}); + } + + async createBatch(userId: number, input: any, idempotencyKey: string) { + validateIdempotencyKey(idempotencyKey); + const items = validateBatchItems(input?.items); + const requestHash = sha256(JSON.stringify(items)); + const existingBatch = await models.AssetBatch.findOne({where: {userId, idempotencyKey}}); + if (existingBatch) { + if (existingBatch.requestHash !== requestHash) { + throw new ApiProblemError(409, 'idempotency_key_conflict', 'Idempotency key conflict', 'The batch idempotency key was already used for different items.'); + } + return this.getBatch(userId, existingBatch.id); + } + const transaction = await models.AssetBatch.sequelize.transaction(); + let createdBatchId: number | null = null; + try { + const batch = await models.AssetBatch.create({userId, idempotencyKey, requestHash, status: 'pending'}, {transaction}); + createdBatchId = batch.id; + const hashes = Array.from(new Set(items.map(item => item.sha256))); + const existingAssets = await models.Asset.findAll({where: {userId, sha256: {[Op.in]: hashes}}, transaction}); + const assetsByHash = new Map(existingAssets.map(asset => [asset.sha256, asset])); + for (const item of items) { + const candidate: any = assetsByHash.get(item.sha256); + const asset = candidate && Number(candidate.bytes) === Number(item.bytes) ? candidate : null; + await models.AssetBatchItem.create({ + ...item, + userId, + assetBatchId: batch.id, + assetId: asset?.id || null, + status: asset ? 'available' : 'missing' + }, {transaction}); + } + await transaction.commit(); + } catch (error) { + if (!transaction.finished) { + await transaction.rollback(); + } + const replay = await models.AssetBatch.findOne({where: {userId, idempotencyKey}}); + if (replay) { + if (replay.requestHash !== requestHash) { + throw new ApiProblemError(409, 'idempotency_key_conflict', 'Idempotency key conflict'); + } + return this.getBatch(userId, replay.id); + } + throw error; + } + return this.getBatch(userId, createdBatchId); + } + + async getBatch(userId: number, batchId: number) { + const batch = await models.AssetBatch.findOne({ + where: {id: batchId, userId}, + include: [{model: models.AssetBatchItem, as: 'items', include: [{model: models.Asset, as: 'asset'}]}], + order: [[{model: models.AssetBatchItem, as: 'items'}, 'logicalId', 'ASC']] + }); + if (!batch) { + throw new ApiProblemError(404, 'asset_batch_not_found', 'Asset batch not found'); + } + return this.serializeBatch(batch); + } + + async completeBatch(userId: number, batchId: number, userApiKeyId: number) { + const batch = await models.AssetBatch.findOne({ + where: {id: batchId, userId}, + include: [{model: models.AssetBatchItem, as: 'items', include: [{model: models.Asset, as: 'asset'}]}] + }); + if (!batch) { + throw new ApiProblemError(404, 'asset_batch_not_found', 'Asset batch not found'); + } + if (batch.status === 'completed') { + return this.serializeBatch(batch); + } + const missing = batch.items.filter(item => !item.assetId || item.status !== 'available'); + if (missing.length) { + throw new ApiProblemError(409, 'asset_batch_incomplete', 'Asset batch incomplete', `${missing.length} batch items still require upload.`, {missingLogicalIds: missing.map(item => item.logicalId)}); + } + const [claimed] = await models.AssetBatch.update({status: 'processing'}, {where: {id: batch.id, userId, status: 'pending'}}); + if (!claimed) { + throw new ApiProblemError(409, 'asset_batch_completion_in_progress', 'Batch completion in progress'); + } + const manifest = buildBatchManifest(batch); + const manifestJson = JSON.stringify(manifest); + const manifestSha256 = sha256(manifestJson); + try { + const content = await app.ms.content.saveData(userId, manifestJson, `asset-batch-${batch.id}.json`, { + userApiKeyId, + driver: {raw: true}, + skipFileCatalog: true, + mimeType: 'application/json', + properties: {assetBatchId: batch.id, manifestSha256} + }); + await setStorageObjectSha256(app, content.storageId, manifestSha256); + await batch.update({status: 'completed', manifestStorageId: content.storageId, manifestSha256}); + return this.getBatch(userId, batch.id); + } catch (error) { + await models.AssetBatch.update({status: 'pending'}, {where: {id: batch.id, status: 'processing'}}); + throw error; + } + } + + async attachAssetToBatch(userId: number, asset: any, batchId?: number, logicalId?: string) { + if (!batchId && !logicalId) { + return; + } + if (!batchId || !logicalId) { + throw new ApiProblemError(400, 'asset_batch_binding_invalid', 'Invalid batch binding', 'Both batchId and logicalId are required.'); + } + const item = await models.AssetBatchItem.findOne({where: {assetBatchId: batchId, userId, logicalId}}); + if (!item) { + throw new ApiProblemError(404, 'asset_batch_item_not_found', 'Asset batch item not found'); + } + if (item.sha256 !== asset.sha256 || Number(item.bytes) !== Number(asset.bytes)) { + throw new ApiProblemError(422, 'asset_batch_item_mismatch', 'Asset does not match batch item'); + } + await item.update({assetId: asset.id, status: 'available'}); + } + + serializeAsset(asset: any, created: boolean) { + const data = typeof asset.toJSON === 'function' ? asset.toJSON() : asset; + const context = getPublicApiContext(app); + return { + schemaVersion: 1, + assetId: `asset_${data.id}`, + storageId: data.storageId, + sha256: data.sha256, + bytes: Number(data.bytes), + mimeType: data.mimeType, + logicalPath: data.logicalPath || null, + created, + pinStatus: data.pinStatus, + urls: { + content: `${context.publicUrl}/ipfs/${data.storageId}`, + metadata: `${context.apiBaseUrl}/assets/${data.storageId}` + } + }; + } + + serializeBatch(batch: any) { + const data = typeof batch.toJSON === 'function' ? batch.toJSON() : batch; + const context = getPublicApiContext(app); + return { + schemaVersion: 1, + batchId: data.id, + status: data.status, + manifest: data.manifestStorageId ? { + storageId: data.manifestStorageId, + sha256: data.manifestSha256, + url: `${context.publicUrl}/ipfs/${data.manifestStorageId}` + } : null, + items: (data.items || []).sort((a, b) => a.logicalId.localeCompare(b.logicalId)).map(item => ({ + logicalId: item.logicalId, + logicalPath: item.logicalPath || null, + sha256: item.sha256, + bytes: Number(item.bytes), + mimeType: item.mimeType, + status: item.status, + requiredUpload: item.status !== 'available', + asset: item.asset ? this.serializeAsset(item.asset, false) : null + })) + }; + } + } + + return new AssetModule(); +} + +function validateIdempotencyKey(key: string) { + if (!key || key.length > 200 || !/^[A-Za-z0-9._:/-]+$/.test(key)) { + throw new ApiProblemError(400, 'idempotency_key_invalid', 'Invalid idempotency key', 'Use 1-200 URL-safe characters.'); + } +} + +function validateBatchItems(value: any): any[] { + if (!Array.isArray(value) || !value.length || value.length > 1000) { + throw new ApiProblemError(400, 'asset_batch_items_invalid', 'Invalid batch items', 'Provide between 1 and 1000 items.'); + } + const logicalIds = new Set(); + return value.map(item => { + const logicalId = String(item?.logicalId || '').trim(); + if (!logicalId || logicalId.length > 200 || logicalIds.has(logicalId)) { + throw new ApiProblemError(400, 'asset_batch_logical_id_invalid', 'Invalid batch logical ID'); + } + logicalIds.add(logicalId); + return { + logicalId, + logicalPath: validateLogicalPath(item.logicalPath), + sha256: validateSha256(item.sha256), + bytes: validateBytes(item.bytes), + mimeType: validateMimeType(item.mimeType) + }; + }); +} + +export function validateSha256(value): string { + const digest = String(value || '').trim().toLowerCase(); + if (!/^[a-f0-9]{64}$/.test(digest)) { + throw new ApiProblemError(400, 'sha256_invalid', 'Invalid SHA-256', 'Expected 64 lowercase hexadecimal characters.'); + } + return digest; +} + +export function validateBytes(value): number { + const bytes = Number(value); + if (!Number.isSafeInteger(bytes) || bytes < 0) { + throw new ApiProblemError(400, 'asset_bytes_invalid', 'Invalid asset byte count'); + } + return bytes; +} + +export function validateMimeType(value): string { + const mimeType = String(value || 'application/octet-stream').trim().toLowerCase(); + if (mimeType.length > 200 || !/^[a-z0-9.+-]+\/[a-z0-9.+-]+$/.test(mimeType)) { + throw new ApiProblemError(400, 'asset_mime_type_invalid', 'Invalid asset MIME type'); + } + return mimeType; +} + +export function validateLogicalPath(value): string | null { + const logicalPath = String(value || '').trim().replace(/^\/+/, ''); + if (!logicalPath) { + return null; + } + if (logicalPath.length > 500 || logicalPath.split('/').includes('..')) { + throw new ApiProblemError(400, 'asset_logical_path_invalid', 'Invalid logical path'); + } + return logicalPath; +} + +export function getAssetRequestHash(input: any): string { + return sha256(JSON.stringify({ + sha256: input.sha256, + bytes: Number(input.bytes), + mimeType: input.mimeType, + logicalPath: input.logicalPath || null, + previewPolicy: input.previewPolicy || 'none', + batchId: input.batchId || null, + logicalId: input.logicalId || null + })); +} + +function buildBatchManifest(batch: any) { + const items = batch.items + .map(item => ({ + logicalId: item.logicalId, + logicalPath: item.logicalPath || null, + storageId: item.asset.storageId, + sha256: item.sha256, + bytes: Number(item.bytes), + mimeType: item.mimeType + })) + .sort((a, b) => a.logicalId.localeCompare(b.logicalId)); + return {schemaVersion: 1, batchId: batch.id, items}; +} + +function sha256(value: string): string { + return createHash('sha256').update(value).digest('hex'); +} + +async function setStorageObjectSha256(app: IGeesomeApp, storageId: string, digest: string) { + const database: any = app.ms.database; + if (typeof database.setStorageObjectSha256 === 'function') { + await database.setStorageObjectSha256(storageId, digest); + return; + } + await database.models.StorageObject.update({sha256: digest}, {where: {storageId}}); +} + +async function getAssetPinStatus(app: IGeesomeApp, content: any): Promise { + if (content.isPinned === true) { + return 'pinned'; + } + const database: any = app.ms.database; + if (typeof database.getStorageObjectPinProvenance !== 'function') { + return 'stored'; + } + const provenance = await database.getStorageObjectPinProvenance(content.storageId); + return provenance?.isConfirmedPinned ? 'confirmed' : 'stored'; +} + +function getErrorCode(error): string { + if (error?.code && typeof error.code === 'string') { + return error.code; + } + if (error?.message && /^[a-z0-9_]+$/.test(error.message)) { + return error.message; + } + return 'asset_upload_failed'; +} diff --git a/app/modules/asset/interface.ts b/app/modules/asset/interface.ts new file mode 100644 index 00000000..f1ef2f0f --- /dev/null +++ b/app/modules/asset/interface.ts @@ -0,0 +1,19 @@ +export default interface IGeesomeAssetModule { + supportsBatches: boolean; + + flushDatabase(): Promise; + + prepareAssetRequest(userId: number, key: string, requestHash: string): Promise; + + createAssetFromUpload(userId: number, data: any, fileName: string, options: any): Promise; + + getAsset(userId: number, storageId: string): Promise; + + linkAssetOperation(requestId: number, operationId: number): Promise; + + createBatch(userId: number, input: any, idempotencyKey: string): Promise; + + getBatch(userId: number, batchId: number): Promise; + + completeBatch(userId: number, batchId: number, userApiKeyId: number): Promise; +} diff --git a/app/modules/asset/models.ts b/app/modules/asset/models.ts new file mode 100644 index 00000000..e85673b7 --- /dev/null +++ b/app/modules/asset/models.ts @@ -0,0 +1,78 @@ +import {DataTypes, Sequelize} from 'sequelize'; + +export default async function (sequelize: Sequelize, databaseModels: any) { + const Asset = sequelize.define('asset', { + storageId: {type: DataTypes.STRING(200), allowNull: false}, + sha256: {type: DataTypes.STRING(64), allowNull: false}, + bytes: {type: DataTypes.BIGINT, allowNull: false}, + mimeType: {type: DataTypes.STRING(200), allowNull: false}, + logicalPath: {type: DataTypes.STRING(500)}, + pinStatus: {type: DataTypes.STRING(40), allowNull: false, defaultValue: 'stored'} + } as any, { + indexes: [ + {name: 'assets_user_storage_unique', fields: ['userId', 'storageId'], unique: true}, + {name: 'assets_user_sha_idx', fields: ['userId', 'sha256', 'id']}, + {name: 'assets_content_idx', fields: ['contentId']} + ] + } as any); + + const AssetIdempotencyKey = sequelize.define('assetIdempotencyKey', { + namespace: {type: DataTypes.STRING(80), allowNull: false, defaultValue: 'assets'}, + key: {type: DataTypes.STRING(200), allowNull: false}, + requestHash: {type: DataTypes.STRING(64), allowNull: false}, + status: {type: DataTypes.STRING(40), allowNull: false, defaultValue: 'pending'}, + asyncOperationId: {type: DataTypes.INTEGER}, + errorCode: {type: DataTypes.STRING(80)} + } as any, { + indexes: [ + {name: 'asset_idempotency_user_namespace_key_unique', fields: ['userId', 'namespace', 'key'], unique: true}, + {name: 'asset_idempotency_asset_idx', fields: ['assetId']}, + {name: 'asset_idempotency_operation_idx', fields: ['asyncOperationId']} + ] + } as any); + + const AssetBatch = sequelize.define('assetBatch', { + idempotencyKey: {type: DataTypes.STRING(200), allowNull: false}, + requestHash: {type: DataTypes.STRING(64), allowNull: false}, + status: {type: DataTypes.STRING(40), allowNull: false, defaultValue: 'pending'}, + manifestStorageId: {type: DataTypes.STRING(200)}, + manifestSha256: {type: DataTypes.STRING(64)} + } as any, { + indexes: [ + {name: 'asset_batches_user_key_unique', fields: ['userId', 'idempotencyKey'], unique: true}, + {name: 'asset_batches_user_created_idx', fields: ['userId', 'createdAt', 'id']} + ] + } as any); + + const AssetBatchItem = sequelize.define('assetBatchItem', { + logicalId: {type: DataTypes.STRING(200), allowNull: false}, + logicalPath: {type: DataTypes.STRING(500)}, + sha256: {type: DataTypes.STRING(64), allowNull: false}, + bytes: {type: DataTypes.BIGINT, allowNull: false}, + mimeType: {type: DataTypes.STRING(200), allowNull: false}, + status: {type: DataTypes.STRING(40), allowNull: false, defaultValue: 'missing'} + } as any, { + indexes: [ + {name: 'asset_batch_items_batch_logical_unique', fields: ['assetBatchId', 'logicalId'], unique: true}, + {name: 'asset_batch_items_batch_status_idx', fields: ['assetBatchId', 'status', 'id']}, + {name: 'asset_batch_items_asset_idx', fields: ['assetId']} + ] + } as any); + + Asset.belongsTo(databaseModels.User, {as: 'user', foreignKey: 'userId'}); + Asset.belongsTo(databaseModels.Content, {as: 'content', foreignKey: 'contentId'}); + AssetIdempotencyKey.belongsTo(databaseModels.User, {as: 'user', foreignKey: 'userId'}); + AssetIdempotencyKey.belongsTo(Asset, {as: 'asset', foreignKey: 'assetId'}); + AssetBatch.belongsTo(databaseModels.User, {as: 'user', foreignKey: 'userId'}); + AssetBatchItem.belongsTo(databaseModels.User, {as: 'user', foreignKey: 'userId'}); + AssetBatchItem.belongsTo(Asset, {as: 'asset', foreignKey: 'assetId'}); + AssetBatchItem.belongsTo(AssetBatch, {as: 'batch', foreignKey: 'assetBatchId'}); + AssetBatch.hasMany(AssetBatchItem, {as: 'items', foreignKey: 'assetBatchId'}); + + await Asset.sync({}); + await AssetIdempotencyKey.sync({}); + await AssetBatch.sync({}); + await AssetBatchItem.sync({}); + + return {Asset, AssetIdempotencyKey, AssetBatch, AssetBatchItem}; +} diff --git a/app/modules/asyncOperation/api.ts b/app/modules/asyncOperation/api.ts index 20f4bcd9..152f3bb6 100644 --- a/app/modules/asyncOperation/api.ts +++ b/app/modules/asyncOperation/api.ts @@ -1,9 +1,45 @@ import {IGeesomeApp} from "../../interface.js"; import IGeesomeAsyncOperationModule from "./interface.js"; import helpers from "../../helpers"; +import {ApiProblemError} from '../api/problem.js'; +import {getPublicApiContext} from '../api/publicUrls.js'; +import {requireIntegrationScopes} from '../api/integrationScopes.js'; export default (app: IGeesomeApp, asyncOperationModule: IGeesomeAsyncOperationModule) => { + /** + * @api {get} /v1/operations/:id Get operation resource + * @apiName OperationGet + * @apiGroup Operations + * @apiUse ApiKey + * @apiParam {Number} id Operation identifier. + * @apiSuccess {String="pending","running","succeeded","failed","cancelled"} status Stable operation state. + * @apiSuccess {Object} [result] Completed operation result. + * @apiSuccess {Object} [problem] Standard problem document for failed work. + */ + app.ms.api.onAuthorizedGet('operations/:id', async (req, res) => { + requireIntegrationScopes(req.apiKey, ['operations:read']); + const operation = await asyncOperationModule.getAsyncOperation(req.user.id, req.params.id); + if (!operation) { + throw new ApiProblemError(404, 'operation_not_found', 'Operation not found'); + } + res.send(serializeOperation(app, operation)); + }); + + /** + * @api {post} /v1/operations/:id/cancel Cancel operation + * @apiName OperationCancel + * @apiGroup Operations + * @apiUse ApiKey + * @apiParam {Number} id Operation identifier. + */ + app.ms.api.onAuthorizedPost('operations/:id/cancel', async (req, res) => { + requireIntegrationScopes(req.apiKey, ['operations:read']); + await asyncOperationModule.cancelAsyncOperation(req.user.id, req.params.id); + const operation = await asyncOperationModule.getAsyncOperation(req.user.id, req.params.id); + res.send(serializeOperation(app, operation)); + }); + /** * @api {post} /v1/user/get-operation-queue/:operationId Get operation queue item * @apiName UserOperationQueue @@ -88,3 +124,53 @@ export default (app: IGeesomeApp, asyncOperationModule: IGeesomeAsyncOperationMo res.send(await asyncOperationModule.cancelAsyncOperation(req.user.id, req.params.id)); }); } + +function serializeOperation(app: IGeesomeApp, operation: any) { + const context = getPublicApiContext(app); + const output = parseOperationOutput(operation.output); + const status = getOperationStatus(operation); + return { + schemaVersion: 1, + operationId: `op_${operation.id}`, + status, + percent: Number(operation.percent || 0), + statusUrl: `${context.apiBaseUrl}/operations/${operation.id}`, + requestId: operation.requestId || null, + result: status === 'succeeded' ? output : null, + problem: status === 'failed' ? output?.problem || { + type: 'about:blank', + title: 'Operation failed', + status: 500, + code: operation.errorType || 'operation_failed', + detail: operation.errorMessage || 'The operation failed.', + requestId: operation.requestId || null + } : null + }; +} + +function getOperationStatus(operation: any): string { + if (operation.cancel) { + return 'cancelled'; + } + if (operation.inProcess) { + return Number(operation.percent || 0) > 0 ? 'running' : 'pending'; + } + if (operation.errorType || operation.errorMessage) { + return 'failed'; + } + return 'succeeded'; +} + +function parseOperationOutput(output: any) { + if (!output) { + return null; + } + if (typeof output !== 'string') { + return output; + } + try { + return JSON.parse(output); + } catch (e) { + return null; + } +} diff --git a/app/modules/asyncOperation/index.ts b/app/modules/asyncOperation/index.ts index 5acb486f..42ef8519 100644 --- a/app/modules/asyncOperation/index.ts +++ b/app/modules/asyncOperation/index.ts @@ -6,6 +6,7 @@ import IGeesomeAsyncOperationModule, {IModuleOperationQueueProcessorOptions, IUs import {CorePermissionName, IListParams, IListParamsOptions} from "../database/interface.js"; import {IGeesomeApp} from "../../interface.js"; import helpers from "../../helpers.js"; +import {getApiProblem} from '../api/problem.js'; const {isObject, last} = _; const log = debug('geesome:app:asyncOperation'); const operationQueueListParams: IListParamsOptions = { @@ -50,7 +51,8 @@ export function getModule(app: IGeesomeApp, models) { userApiKeyId: options.userApiKeyId, name: 'save-data', inProcess: true, - channel: await commonHelper.random() + channel: await commonHelper.random(), + requestId: options.requestId || null }); // TODO: fix hotfix @@ -89,7 +91,9 @@ export function getModule(app: IGeesomeApp, models) { }]); this.updateUserAsyncOperation(asyncOperation.id, { inProcess: false, - contentId: res.id + finishedAt: new Date(), + contentId: res?.contentId || res?.id || null, + output: JSON.stringify(res || null) }); return app.ms.communicator ? app.ms.communicator.publishEvent(asyncOperation.channel, res) : null; }) @@ -98,10 +102,13 @@ export function getModule(app: IGeesomeApp, models) { asyncOperationId: asyncOperation.id, error: getErrorMessage(e) }]); + const problem = getApiProblem(e, options.requestId); return this.updateUserAsyncOperation(asyncOperation.id, { inProcess: false, - errorType: 'unknown', - errorMessage: e && e.message ? e.message : e + finishedAt: new Date(), + errorType: problem.code, + errorMessage: problem.detail, + output: JSON.stringify({problem}) }); }); diff --git a/app/modules/asyncOperation/interface.ts b/app/modules/asyncOperation/interface.ts index c400afd1..6a93cf05 100644 --- a/app/modules/asyncOperation/interface.ts +++ b/app/modules/asyncOperation/interface.ts @@ -73,6 +73,7 @@ export interface IUserAsyncOperation { inProcess: boolean; cancel: boolean; output?: string; + requestId?: string; userId: number; contentId?: number; diff --git a/app/modules/asyncOperation/models.ts b/app/modules/asyncOperation/models.ts index d19fed23..bbd4edcb 100644 --- a/app/modules/asyncOperation/models.ts +++ b/app/modules/asyncOperation/models.ts @@ -51,6 +51,9 @@ export default async function (sequelize: Sequelize) { userApiKeyId: { type: DataTypes.INTEGER }, + requestId: { + type: DataTypes.STRING(128) + }, contentId: { type: DataTypes.INTEGER }, diff --git a/app/modules/content/asyncBusboy.ts b/app/modules/content/asyncBusboy.ts index c3987503..7b15e544 100644 --- a/app/modules/content/asyncBusboy.ts +++ b/app/modules/content/asyncBusboy.ts @@ -2,6 +2,7 @@ import fs from 'fs'; import os from 'os'; import path from 'path'; import busboy from 'busboy'; +import {createHash} from 'node:crypto'; const getDescriptor = Object.getOwnPropertyDescriptor @@ -46,7 +47,20 @@ function onFile (filePromises, file, stream, info) { const tmpName = Math.random().toString(16).substring(2) + '-' + filename const saveTo = path.join(os.tmpdir(), path.basename(tmpName)) const writeStream = fs.createWriteStream(saveTo) - const filePromise = new Promise((resolve, reject) => + const hash = createHash('sha256') + let bytes = 0 + stream.on('data', (chunk) => { + hash.update(chunk) + bytes += chunk.length + }) + const filePromise = new Promise((resolve, reject) => { + stream.on('limit', () => { + const err: any = new Error('Reach file size limit') + err.code = 'Request_file_size_limit' + err.status = 413 + writeStream.destroy(err) + stream.resume() + }) writeStream .on('open', () => stream.pipe(writeStream) @@ -59,6 +73,8 @@ function onFile (filePromises, file, stream, info) { readStream.transferEncoding = readStream.encoding = encoding readStream.mimeType = readStream.mime = mimeType readStream.tempPath = saveTo + readStream.sha256 = hash.digest('hex') + readStream.bytes = bytes readStream.emitFinish = (callback) => { fs.rm(saveTo, { force: true }, function () { callback && callback() @@ -71,7 +87,8 @@ function onFile (filePromises, file, stream, info) { stream.resume() .on('error', reject) reject(err) - })) + }) + }) filePromises.push(filePromise) } diff --git a/app/modules/content/index.ts b/app/modules/content/index.ts index b28db189..dfb4a399 100644 --- a/app/modules/content/index.ts +++ b/app/modules/content/index.ts @@ -29,6 +29,7 @@ import {DriverInput, OutputSize} from "../drivers/interface.js"; import helpers from "../../helpers"; import {recordMemorySnapshot} from "../../memoryProfiler.js"; import {rtrim} from "telegram/Utils"; +import {ApiProblemError, sendApiProblem} from '../api/problem.js'; const {pick, isArray, isNumber, isTypedArray, isString, isBuffer, isObject, isUndefined, merge, last, startsWith, trimStart} = _; const log = debug('geesome:app'); const {getDirSize} = driverHelpers; @@ -1318,7 +1319,7 @@ function getModule(app: IGeesomeApp) { if (!content) { const storageIdAllowed = await isStorageIdAllowedForApi(storageId); if (!storageIdAllowed) { - return res.send(423); + return sendApiProblem(res, new ApiProblemError(423, 'content_locked', 'Content locked', 'The supplied storage ID is not publicly readable.'), req.requestId); } storageResponseHeaders = await getStorageResponseHeadersForApi(storageId); } @@ -1327,7 +1328,7 @@ function getModule(app: IGeesomeApp) { const fileStat = await getGatewayFileStat(dataPath); log('getFileStat', fileStat); if (!fileStat) { - return res.send(404); + return sendApiProblem(res, new ApiProblemError(404, 'content_not_found', 'Content not found', 'No readable content exists for the supplied storage ID.'), req.requestId); } if (fileStat.cid) { content = await app.ms.database.getSharedStorageMetadataByStorageId(ipfsHelper.cidToIpfsHash(fileStat.cid), {includePreviews: true}); @@ -1347,7 +1348,7 @@ function getModule(app: IGeesomeApp) { if (!content) { const storageIdAllowed = await isStorageIdAllowedForApi(storageId); if (!storageIdAllowed) { - return res.send(423); + return sendApiProblem(res, new ApiProblemError(423, 'content_locked', 'Content locked', 'The supplied storage ID is not publicly readable.'), req.requestId); } storageResponseHeaders = await getStorageResponseHeadersForApi(storageId); } @@ -1356,7 +1357,7 @@ function getModule(app: IGeesomeApp) { dataPath = this.prepareContentStorageDataPath(dataPath, content); const dataSize = await this.getGatewayFileSize(dataPath, content); if (dataSize === null) { - return res.send(404); + return sendApiProblem(res, new ApiProblemError(404, 'content_not_found', 'Content not found', 'No readable content exists for the supplied storage ID.'), req.requestId); } log('dataSize', dataSize); @@ -1453,13 +1454,19 @@ function getModule(app: IGeesomeApp) { contentData['x-ipfs-datasize'] = dataSize; } const mimeType = contentData['Content-Type']; + const storageId = content?.storageId || dataPath.split('/')[0]; + if (content?.sha256 && /^[a-f0-9]{64}$/i.test(content.sha256)) { + contentData['Content-Digest'] = `sha-256=:${Buffer.from(content.sha256, 'hex').toString('base64')}:`; + } return { ...contentData, ...extraHeaders, ...getContentServingSecurityHeaders(mimeType), 'Accept-Ranges': 'bytes', 'Cross-Origin-Resource-Policy': 'cross-origin', - 'cache-control': 'public, max-age=29030400, immutable', + 'cache-control': 'public, max-age=31536000, immutable', + 'ETag': `"${storageId}"`, + 'X-Geesome-Storage-Id': storageId, 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'x-ipfs-path': dataPath, 'x-ipfs-roots': last(dataPath.split('/')), diff --git a/app/modules/database/index.ts b/app/modules/database/index.ts index 80a788d7..5f8926da 100644 --- a/app/modules/database/index.ts +++ b/app/modules/database/index.ts @@ -340,6 +340,10 @@ class PostgresDatabase implements IGeesomeDatabaseModule { return this.models.StorageObject.findOne({where: {storageId}}); } + async setStorageObjectSha256(storageId: string, sha256: string) { + await this.models.StorageObject.update({sha256}, {where: {storageId}}); + } + async getStorageObjectByIdentity(identityType: string, identityId: string) { const where = getStorageObjectIdentityWhere(identityType, identityId); if (!where) { diff --git a/app/modules/database/interface.ts b/app/modules/database/interface.ts index 4ed91185..ac17e3ef 100644 --- a/app/modules/database/interface.ts +++ b/app/modules/database/interface.ts @@ -51,6 +51,8 @@ export interface IGeesomeDatabaseModule { getStorageObjectByStorageId(storageId): Promise; + setStorageObjectSha256(storageId: string, sha256: string): Promise; + getStorageObjectByIdentity(identityType: string, identityId: string): Promise; syncStorageObject(storageObjectData: Partial, options?): Promise; @@ -208,7 +210,9 @@ export interface IUserApiKey { valueHash: string; type?: string; permissions?: string; + scopes?: string; expiredOn?: Date; + lastUsedAt?: Date; isDisabled: boolean; } @@ -257,6 +261,7 @@ export interface IStorageObjectRecord { mimeType?: ContentMimeType; extension?: string; size?: number; + sha256?: string; largePreviewSize?: number; largePreviewStorageId?: string; mediumPreviewSize?: number; diff --git a/app/modules/database/migrations/20260731000000-add-agent-api-metadata.cjs b/app/modules/database/migrations/20260731000000-add-agent-api-metadata.cjs new file mode 100644 index 00000000..bbf814aa --- /dev/null +++ b/app/modules/database/migrations/20260731000000-add-agent-api-metadata.cjs @@ -0,0 +1,68 @@ +'use strict'; + +async function hasTable(queryInterface, tableName) { + const [rows] = await queryInterface.sequelize.query(` + SELECT to_regclass(:tableName) AS table_name + `, {replacements: {tableName: `"${tableName}"`}}); + return !!rows[0]?.table_name; +} + +module.exports = { + useTransaction: false, + + up: async (queryInterface) => { + if (await hasTable(queryInterface, 'storageObjects')) { + await queryInterface.sequelize.query(` + ALTER TABLE "storageObjects" + ADD COLUMN IF NOT EXISTS "sha256" VARCHAR(64) + `); + await queryInterface.sequelize.query(` + CREATE INDEX CONCURRENTLY IF NOT EXISTS storage_objects_sha256_idx + ON "storageObjects" ("sha256", "id") + `); + } + + if (await hasTable(queryInterface, 'userApiKeys')) { + await queryInterface.sequelize.query(` + ALTER TABLE "userApiKeys" + ADD COLUMN IF NOT EXISTS "lastUsedAt" TIMESTAMP WITH TIME ZONE, + ADD COLUMN IF NOT EXISTS "scopes" TEXT + `); + } + + if (await hasTable(queryInterface, 'userAsyncOperations')) { + await queryInterface.sequelize.query(` + ALTER TABLE "userAsyncOperations" + ADD COLUMN IF NOT EXISTS "requestId" VARCHAR(128) + `); + } + }, + + down: async (queryInterface) => { + if (await hasTable(queryInterface, 'storageObjects')) { + await queryInterface.sequelize.query(` + DROP INDEX CONCURRENTLY IF EXISTS storage_objects_sha256_idx + `); + await queryInterface.sequelize.query(` + ALTER TABLE "storageObjects" + DROP COLUMN IF EXISTS "sha256" + `); + } + + if (await hasTable(queryInterface, 'userApiKeys')) { + await queryInterface.sequelize.query(` + ALTER TABLE "userApiKeys" + DROP COLUMN IF EXISTS "lastUsedAt", + DROP COLUMN IF EXISTS "scopes" + `); + } + + + if (await hasTable(queryInterface, 'userAsyncOperations')) { + await queryInterface.sequelize.query(` + ALTER TABLE "userAsyncOperations" + DROP COLUMN IF EXISTS "requestId" + `); + } + } +}; diff --git a/app/modules/database/models/storageObject.ts b/app/modules/database/models/storageObject.ts index 940788ef..30c5921a 100644 --- a/app/modules/database/models/storageObject.ts +++ b/app/modules/database/models/storageObject.ts @@ -11,15 +11,16 @@ import {DataTypes, Op, Sequelize} from "sequelize"; export default async function (sequelize: Sequelize, models) { const schemaState = await getStorageObjectIdentitySchemaState(sequelize); const includeIdentityColumns = !schemaState.tableExists || schemaState.hasIdentityColumns; + const includeSha256Column = !schemaState.tableExists || schemaState.hasSha256Column; - const StorageObject = sequelize.define('storageObject', getStorageObjectAttributes(includeIdentityColumns), { - indexes: getStorageObjectIndexes(includeIdentityColumns) + const StorageObject = sequelize.define('storageObject', getStorageObjectAttributes(includeIdentityColumns, includeSha256Column), { + indexes: getStorageObjectIndexes(includeIdentityColumns, includeSha256Column) } as any); return StorageObject.sync({}); }; -function getStorageObjectAttributes(includeIdentityColumns: boolean) { +function getStorageObjectAttributes(includeIdentityColumns: boolean, includeSha256Column: boolean) { const attributes = { storageId: { type: DataTypes.STRING(200), @@ -68,6 +69,10 @@ function getStorageObjectAttributes(includeIdentityColumns: boolean) { } } as any; + if (includeSha256Column) { + attributes.sha256 = {type: DataTypes.STRING(64)}; + } + if (includeIdentityColumns) { attributes.identityType = { type: DataTypes.STRING(80) @@ -86,14 +91,17 @@ function getStorageObjectAttributes(includeIdentityColumns: boolean) { return attributes; } -function getStorageObjectIndexes(includeIdentityIndex: boolean) { +function getStorageObjectIndexes(includeIdentityIndex: boolean, includeSha256Index: boolean) { const indexes: any[] = [ { name: 'storage_objects_storage_id_unique', fields: ['storageId'], unique: true }, { name: 'storage_objects_large_preview_storage_idx', fields: ['largePreviewStorageId'] }, { name: 'storage_objects_medium_preview_storage_idx', fields: ['mediumPreviewStorageId'] }, { name: 'storage_objects_small_preview_storage_idx', fields: ['smallPreviewStorageId'] }, - { name: 'storage_objects_updated_idx', fields: ['updatedAt', 'id'] } + { name: 'storage_objects_updated_idx', fields: ['updatedAt', 'id'] } ]; + if (includeSha256Index) { + indexes.push({name: 'storage_objects_sha256_idx', fields: ['sha256', 'id']}); + } if (includeIdentityIndex) { indexes.push({ @@ -127,10 +135,18 @@ async function getStorageObjectIdentitySchemaState(sequelize: Sequelize) { AND table_name = 'storageObjects' AND column_name = 'identityId' ) AS "hasIdentityId" + , EXISTS ( + SELECT 1 + FROM information_schema.columns + WHERE table_schema = 'public' + AND table_name = 'storageObjects' + AND column_name = 'sha256' + ) AS "hasSha256" `); const row = (rows as any[])[0] || {}; return { tableExists: row.tableExists === true, hasIdentityColumns: row.hasIdentityType === true && row.hasIdentityId === true, + hasSha256Column: row.hasSha256 === true, }; } diff --git a/app/modules/database/models/userApiKey.ts b/app/modules/database/models/userApiKey.ts index 439a6acc..0a8b9d0e 100644 --- a/app/modules/database/models/userApiKey.ts +++ b/app/modules/database/models/userApiKey.ts @@ -25,9 +25,15 @@ export default async function (sequelize, models) { permissions: { type: DataTypes.STRING }, + scopes: { + type: DataTypes.TEXT + }, expiredOn: { type: DataTypes.DATE }, + lastUsedAt: { + type: DataTypes.DATE + }, isDisabled: { type: DataTypes.BOOLEAN, defaultValue: false diff --git a/app/modules/gateway/index.ts b/app/modules/gateway/index.ts index f683bf1f..3b706093 100644 --- a/app/modules/gateway/index.ts +++ b/app/modules/gateway/index.ts @@ -9,6 +9,7 @@ import {trackRuntimeHttpRequest} from '../../memoryProfiler.js'; import {closeHttpServer} from '../../httpServer.js'; import gatewayHelpers from "./helpers.js"; import {cleanupAndRethrow} from '../../resourceCleanup.js'; +import {getRequestId} from '../api/problem.js'; export default async (app: IGeesomeApp, options: {registerApi?: boolean, port?: number | string} = {}) => { app.checkModules(['api']); @@ -35,6 +36,10 @@ async function getModule(app: IGeesomeApp, port) { service.use(morgan('combined')); } service.use((req, res, next) => { + const requestId = getRequestId(req.headers['x-request-id'] as string); + req.requestId = requestId; + res.locals.requestId = requestId; + res.setHeader('X-Request-Id', requestId); trackRuntimeHttpRequest('gateway', req, res); next(); }); diff --git a/bash/cf-nginx.conf b/bash/cf-nginx.conf index 2b420c32..ffd84f51 100755 --- a/bash/cf-nginx.conf +++ b/bash/cf-nginx.conf @@ -13,6 +13,14 @@ server { location / { try_files $uri $uri/ =404; } + location = /.well-known/geesome { proxy_pass http://127.0.0.1:2052/.well-known/geesome; } + location = /.well-known/openapi.json { proxy_pass http://127.0.0.1:2052/.well-known/openapi.json; } + location /api/ { + client_max_body_size 2000m; + proxy_pass http://127.0.0.1:2052/; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + } location /.well-known/ { alias /var/www/%app_domain%/.well-known/; } @@ -34,6 +42,8 @@ server { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } + location = /.well-known/geesome { proxy_pass http://127.0.0.1:2052/.well-known/geesome; } + location /.well-known/ { alias /var/www/%app_domain%/.well-known/; } @@ -53,7 +63,8 @@ server { proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } + location = /.well-known/geesome { proxy_pass http://127.0.0.1:2052/.well-known/geesome; } location /.well-known/ { alias /var/www/gateway.%app_domain%/.well-known/; } -} \ No newline at end of file +} diff --git a/bash/docker-test b/bash/docker-test index bdeacf91..ab45e95e 100755 --- a/bash/docker-test +++ b/bash/docker-test @@ -57,7 +57,7 @@ if [[ "$cold" -eq 1 ]]; then test/.postgres-data "${compose[@]}" down --remove-orphans --volumes else - "${compose[@]}" rm -f -s test >/dev/null 2>&1 || true + "${compose[@]}" rm -f -s test agent_proxy >/dev/null 2>&1 || true fi up_args=(up --remove-orphans --abort-on-container-exit --exit-code-from test) diff --git a/bash/nginx.conf b/bash/nginx.conf index cf01869a..c4fc35f1 100755 --- a/bash/nginx.conf +++ b/bash/nginx.conf @@ -20,6 +20,7 @@ server { location = /swagger.json { proxy_pass http://127.0.0.1:2052/swagger.json; } location = /api-docs.json { proxy_pass http://127.0.0.1:2052/api-docs.json; } location = /.well-known/openapi.json { proxy_pass http://127.0.0.1:2052/.well-known/openapi.json; } + location = /.well-known/geesome { proxy_pass http://127.0.0.1:2052/.well-known/geesome; } location /ipfs/ { proxy_pass http://127.0.0.1:2052/ipfs/; @@ -55,6 +56,8 @@ server { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } + location = /.well-known/geesome { proxy_pass http://127.0.0.1:2052/.well-known/geesome; } + location /.well-known/ { alias /var/www/%app_domain%/.well-known/; } @@ -73,6 +76,7 @@ server { proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } + location = /.well-known/geesome { proxy_pass http://127.0.0.1:2052/.well-known/geesome; } location /.well-known/ { alias /var/www/$host/.well-known/; } @@ -84,4 +88,4 @@ server { listen [::]:80 default_server; server_name _; return 301 https://$host$request_uri; -} \ No newline at end of file +} diff --git a/bash/nodomain-nginx.conf b/bash/nodomain-nginx.conf index d0524a51..df05fbb7 100644 --- a/bash/nodomain-nginx.conf +++ b/bash/nodomain-nginx.conf @@ -18,6 +18,7 @@ server { location = /swagger.json { proxy_pass http://127.0.0.1:2052/swagger.json; } location = /api-docs.json { proxy_pass http://127.0.0.1:2052/api-docs.json; } location = /.well-known/openapi.json { proxy_pass http://127.0.0.1:2052/.well-known/openapi.json; } + location = /.well-known/geesome { proxy_pass http://127.0.0.1:2052/.well-known/geesome; } location /ipfs/ { proxy_pass http://127.0.0.1:2052/ipfs/; diff --git a/bash/uncert-nginx.conf b/bash/uncert-nginx.conf index 0bc01629..88b82360 100755 --- a/bash/uncert-nginx.conf +++ b/bash/uncert-nginx.conf @@ -8,6 +8,11 @@ server { location / { try_files $uri $uri/ =404; } + location = /.well-known/geesome { proxy_pass http://127.0.0.1:2052/.well-known/geesome; } + location /api/ { + client_max_body_size 2000m; + proxy_pass http://127.0.0.1:2052/; + } location /.well-known/ { alias /var/www/%app_domain%/.well-known/; @@ -24,8 +29,9 @@ server { location / { try_files $uri $uri/ =404; } + location = /.well-known/geesome { proxy_pass http://127.0.0.1:2052/.well-known/geesome; } location /.well-known/ { alias /var/www/gateway.%app_domain%/.well-known/; } -} \ No newline at end of file +} diff --git a/check/databaseMigrationIntegrity.ts b/check/databaseMigrationIntegrity.ts index a1e9e666..188595b6 100644 --- a/check/databaseMigrationIntegrity.ts +++ b/check/databaseMigrationIntegrity.ts @@ -178,6 +178,11 @@ const coveredMigrations: CoveredMigration[] = [ file: '20260623000001-drop-content-user-storage-unique.cjs', verifies: ['drops reverted same-user content storage unique index'], }, + { + module: 'database', + file: '20260731000000-add-agent-api-metadata.cjs', + verifies: ['storage-object SHA-256 metadata', 'API-key integration metadata', 'async-operation request correlation'], + }, { module: 'group', file: '20260506000000-add-post-timeline-indexes.cjs', @@ -254,6 +259,10 @@ const expectedColumns: ExpectedColumn[] = [ {table: 'storageObjects', columns: ['identityId'], type: 'character varying'}, {table: 'storageObjects', columns: ['identityUrl'], type: 'text'}, {table: 'storageObjects', columns: ['identityUpdatedAt'], type: 'timestamp with time zone'}, + {table: 'storageObjects', columns: ['sha256'], type: 'character varying'}, + {table: 'userApiKeys', columns: ['lastUsedAt'], type: 'timestamp with time zone'}, + {table: 'userApiKeys', columns: ['scopes'], type: 'text'}, + {table: 'userAsyncOperations', columns: ['requestId'], type: 'character varying'}, {table: 'storageObjectReferences', columns: ['sourceStorageId'], type: 'character varying'}, {table: 'storageObjectReferences', columns: ['targetStorageId'], type: 'character varying'}, {table: 'storageObjectReferences', columns: ['referenceType'], type: 'character varying'}, @@ -297,6 +306,12 @@ const expectedIndexes: ExpectedIndex[] = [ {name: 'storage_objects_medium_preview_storage_idx', table: 'storageObjects', columns: ['mediumPreviewStorageId']}, {name: 'storage_objects_small_preview_storage_idx', table: 'storageObjects', columns: ['smallPreviewStorageId']}, {name: 'storage_objects_updated_idx', table: 'storageObjects', columns: ['updatedAt', 'id']}, + {name: 'storage_objects_sha256_idx', table: 'storageObjects', columns: ['sha256', 'id']}, + {name: 'assets_user_storage_unique', table: 'assets', columns: ['userId', 'storageId'], unique: true}, + {name: 'assets_user_sha_idx', table: 'assets', columns: ['userId', 'sha256', 'id']}, + {name: 'asset_idempotency_user_namespace_key_unique', table: 'assetIdempotencyKeys', columns: ['userId', 'namespace', 'key'], unique: true}, + {name: 'asset_batches_user_key_unique', table: 'assetBatches', columns: ['userId', 'idempotencyKey'], unique: true}, + {name: 'asset_batch_items_batch_logical_unique', table: 'assetBatchItems', columns: ['assetBatchId', 'logicalId'], unique: true}, {name: 'storage_object_refs_source_target_type_unique', table: 'storageObjectReferences', columns: ['sourceStorageId', 'targetStorageId', 'referenceType'], unique: true}, {name: 'storage_object_refs_target_type_idx', table: 'storageObjectReferences', columns: ['targetStorageId', 'referenceType']}, {name: 'storage_object_refs_source_idx', table: 'storageObjectReferences', columns: ['sourceStorageId']}, @@ -749,6 +764,10 @@ const countChecks: CountCheck[] = [ }, duplicateCheck('storage object storageId duplicates', 'storageObjects', ['storageId']), duplicateCheck('storage object reference duplicates', 'storageObjectReferences', ['sourceStorageId', 'targetStorageId', 'referenceType']), + duplicateCheck('asset owner storage duplicates', 'assets', ['userId', 'storageId']), + duplicateCheck('asset idempotency duplicates', 'assetIdempotencyKeys', ['userId', 'namespace', 'key']), + duplicateCheck('asset batch idempotency duplicates', 'assetBatches', ['userId', 'idempotencyKey']), + duplicateCheck('asset batch logical item duplicates', 'assetBatchItems', ['assetBatchId', 'logicalId']), { name: 'storage object identity pairs are complete', requirements: [{table: 'storageObjects', columns: ['identityType', 'identityId']}], diff --git a/check/databaseScalabilityInventory.ts b/check/databaseScalabilityInventory.ts index 17674a99..b5af3bd9 100644 --- a/check/databaseScalabilityInventory.ts +++ b/check/databaseScalabilityInventory.ts @@ -64,6 +64,7 @@ function modelRows(): ModelRow[] { const contentSource = read('app/modules/database/models/content.ts'); const contentModuleSource = read('app/modules/content/index.ts'); const storageObjectSource = read('app/modules/database/models/storageObject.ts'); + const assetSource = read('app/modules/asset/models.ts'); const storageObjectReferenceSource = read('app/modules/database/models/storageObjectReference.ts'); const storageSpaceSnapshotSource = read('app/modules/database/models/storageSpaceSnapshot.ts'); const storageObjectIntegritySource = read('check/databaseStorageObjectsIntegrity.ts'); @@ -467,6 +468,28 @@ function modelRows(): ModelRow[] { : 'canonical physical storage metadata remains represented only by user-owned Content rows', ], }, + { + area: 'Integration assets', + source: 'app/modules/asset/models.ts', + model: 'Asset / AssetIdempotencyKey', + indexes: [ + has(assetSource, 'assets_user_storage_unique') ? 'userId,storageId unique owner identity' : 'missing owner/storage uniqueness', + has(assetSource, 'assets_user_sha_idx') ? 'userId,sha256,id owner-visible hash preflight' : 'missing owner/hash preflight index', + has(assetSource, 'asset_idempotency_user_namespace_key_unique') ? 'userId,namespace,key unique idempotency identity' : 'missing durable idempotency uniqueness' + ], + notes: ['asset rows reference existing content/storage bytes; owner-scoped hash lookup avoids disclosing global hash existence'] + }, + { + area: 'Integration asset batches', + source: 'app/modules/asset/models.ts', + model: 'AssetBatch / AssetBatchItem', + indexes: [ + has(assetSource, 'asset_batches_user_key_unique') ? 'userId,idempotencyKey unique batch identity' : 'missing batch idempotency uniqueness', + has(assetSource, 'asset_batch_items_batch_logical_unique') ? 'assetBatchId,logicalId unique item identity' : 'missing batch logical-item uniqueness', + has(assetSource, 'asset_batch_items_batch_status_idx') ? 'assetBatchId,status,id completion scan' : 'missing batch status scan index' + ], + notes: ['batch preflight is bounded by the submitted item list and completion loads only one owner-scoped batch with its indexed items'] + }, { area: 'Storage object references', source: 'app/modules/database/models/storageObjectReference.ts', diff --git a/docker-compose.yml b/docker-compose.yml index c4b35c6d..158f1c07 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -19,6 +19,11 @@ services: environment: - IPFS_PROFILE - SSG_RUNTIME + - DOMAIN + - GEESOME_PUBLIC_URL + - GEESOME_API_BASE_PATH + - GEESOME_VERSION + - GEESOME_MAX_UPLOAD_BYTES # Debug/logging knobs. Interpolated so they can be set via a project-dir # .env file (or the shell/systemd env). Default empty -> off. See DEBUG.md. - DEBUG=${DEBUG:-} diff --git a/docs/README.md b/docs/README.md index a046419b..9a00ebb1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -14,6 +14,9 @@ API reference output with handwritten architecture, operations, and module notes public discovery, immutable asset, auth, error, async, and batch contracts. - [Agent-friendly API implementation plan](./agent-friendly-api-implementation-plan.md): current-state gaps, delivery phases, workstreams, rollout, and verification. +- [Agent-friendly asset API](./agent-friendly-api.md): discovery-first curl and + executable Node upload/verification workflow. +- [API problem codes](./api-problems.md): stable error codes and client behavior. - [Active TODO](./todo.md): unfinished deterministic implementation sections. - [Implemented work](./implemented.md): delivered foundations and verification history previously mixed into the TODO. @@ -27,7 +30,8 @@ For implementation slices, list deterministic TODO section ids with A running node exposes machine-readable docs pointers so users and agents can start with only the node URL: -- `GET /v1` returns a discovery JSON document with docs links and route metadata. +- `GET /.well-known/geesome` returns the canonical absolute public discovery + contract; `GET /v1` remains the backward-compatible route index. - `GET /v1/openapi.json` returns the OpenAPI 3 document. - `GET /v1/apidoc.json` returns raw apiDoc data. - `GET /openapi.json`, `/swagger.json`, `/api-docs.json`, and diff --git a/docs/agent-friendly-api-implementation-plan.md b/docs/agent-friendly-api-implementation-plan.md index d2cfcf01..f5a50345 100644 --- a/docs/agent-friendly-api-implementation-plan.md +++ b/docs/agent-friendly-api-implementation-plan.md @@ -21,6 +21,13 @@ credentials, upload and verify immutable assets idempotently, follow async work, and resume a multi-file release without source-code knowledge or deployment-specific path guesses. +Implementation status: completed on `codex/agent-friendly-api` for issue #1312. +The delivered surface includes origin-only discovery, problem responses, +immutable asset upload/read verification, stable operation resources, scoped +credential introspection, resumable batches, deterministic manifests, +production-shaped nginx coverage, generated inventories, and executable docs. +The items under Deferred Follow-Ups remain intentionally out of scope. + The implementation should extend the existing `api`, `content`, `storage`, `gateway`, `asyncOperation`, `database`, and `pin` foundations. It should not create a second HTTP server, a second content store, or an unrelated job system. diff --git a/docs/agent-friendly-api.md b/docs/agent-friendly-api.md new file mode 100644 index 00000000..d0137930 --- /dev/null +++ b/docs/agent-friendly-api.md @@ -0,0 +1,45 @@ +# Agent-Friendly Asset API + +An integration needs only the site origin and a scoped bearer token. Start at +`/.well-known/geesome`; do not guess whether the deployment uses `/v1`, +`/api/v1`, or another public prefix. + +Recommended scopes are `assets:write`, `assets:read-private`, +`operations:read`, and `asset-batches:write`. Inspect the active credential at +`GET {apiBaseUrl}/integrations/credentials/current`. + +## Curl example + +```bash +ORIGIN=https://node.example +TOKEN=replace-with-scoped-token +FILE=./release/character.webp +DISCOVERY=$(curl --fail --silent --show-error "$ORIGIN/.well-known/geesome") +API_BASE=$(printf '%s' "$DISCOVERY" | jq -r .apiBaseUrl) +DIGEST=$(shasum -a 256 "$FILE" | awk '{print $1}') + +curl --fail-with-body "$API_BASE/assets" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Idempotency-Key: release-2026-07-character" \ + -F "file=@$FILE" \ + -F "expectedSha256=$DIGEST" \ + -F "logicalPath=release/character.webp" +``` + +Retry the same bytes and metadata with the same key. A completed replay returns +the original asset with `200`; different request data returns +`409 idempotency_key_conflict`. + +For a release, create `POST {apiBaseUrl}/asset-batches` with an idempotency key +and an `items` array of `logicalId`, `logicalPath`, `sha256`, `bytes`, and +`mimeType`. Upload only items whose `requiredUpload` is true, including +`batchId` and `logicalId` in each asset multipart request. Complete with +`POST {apiBaseUrl}/asset-batches/{batchId}/complete` and verify the returned +manifest SHA-256. + +The executable Node example is +`examples/agent-friendly-assets.mjs`. It discovers the API, uploads one file, +and verifies the immutable read headers without a hardcoded API prefix. + +See [API problem codes](./api-problems.md) and the live `openapiUrl` returned by +discovery for the machine-readable contract. diff --git a/docs/api-problems.md b/docs/api-problems.md new file mode 100644 index 00000000..0f1e5370 --- /dev/null +++ b/docs/api-problems.md @@ -0,0 +1,25 @@ +# API Problem Codes + +New integration routes return RFC 9457-style `application/problem+json`. +Every problem contains `type`, `title`, `status`, a stable GeeSome `code`, and +the same `requestId` exposed in `X-Request-Id`. Internal exception messages and +credential material are never returned. + +## Common codes + +| Code | Status | Meaning | +| --- | ---: | --- | +| `credentials_required` | 401 | The bearer token is missing. | +| `invalid_credentials` | 401 | The bearer token is invalid, expired, disabled, or revoked. | +| `insufficient_scope` | 403 | The credential lacks one or more `requiredScopes`. | +| `idempotency_key_conflict` | 409 | The key was already used with different request data. | +| `idempotency_request_in_progress` | 409 | The original request is still running. | +| `upload_limit_exceeded` | 413 | Multipart or account upload limits were exceeded. | +| `asset_digest_mismatch` | 422 | Uploaded bytes do not match `expectedSha256`. | +| `asset_not_found` | 404 | No asset with that storage ID is visible to this owner. | +| `asset_batch_incomplete` | 409 | One or more declared batch items still require upload. | +| `operation_not_found` | 404 | The operation does not exist or is not visible to this owner. | +| `internal_error` | 500 | The request failed without exposing implementation details. | + +Clients should branch on `status` and `code`, log `requestId`, and treat all +other fields as explanatory or forward-compatible extensions. diff --git a/docs/database-scalability-inventory.md b/docs/database-scalability-inventory.md index 83a23c17..6a67d757 100644 --- a/docs/database-scalability-inventory.md +++ b/docs/database-scalability-inventory.md @@ -8,7 +8,7 @@ This file is generated by `npm run database:scalability:update`. Do not hand-edi ## Summary -- Model/query areas reviewed: 28 +- Model/query areas reviewed: 30 - Query hotspots reviewed: 31 - Primary risk area: group post timelines and generated manifests at 100k+ posts per group. - Current review doc: [database-scalability-review.md](database-scalability-review.md). @@ -26,6 +26,8 @@ This file is generated by `npm run database:scalability:update`. Do not hand-edi | Groups | Group | name,isRemote unique local non-collateral; manifestStorageId; manifestStaticStorageId; creatorId,type,isDeleted,createdAt,id creator list index; isDeleted,staticStorageUpdatedAt static rebind index | has creator listing index; membership/admin through indexes present | `app/modules/group/models/group.ts` | | Contents | Content | storageId,userId non-unique ownership lookup; manifestStaticStorageId unique; userId,manifestStorageId actor-scoped lookup | has user listing index; has manifest lookup index; has preview lookup indexes; same storageId across different users and legacy same-user duplicate rows remain valid; soft-deleted rows no longer block re-upload; actor-scoped helpers and deterministic shared reads handle duplicate candidates; shared storage reads prefer storageObject with Content fallback, shared manifest reads use id ordering, and new library rows require an actor | `app/modules/database/models/content.ts` | | Storage objects | StorageObject | storageId unique physical identity; identityType,identityId canonical lookup with GeeSome manifest producer; preview storage lookup indexes; updatedAt,id registry scan | model-sync-created A2 on-ramp records one physical storage metadata row per storageId from content writes plus nullable ownerless/federated identity metadata; GeeSome content manifest imports seed that identity on the canonical storage object, database callers can resolve canonical rows by identity pair, public shared metadata reads prefer storageId metadata, local and restored legacy pin state remains canonical, per-account remote pin attempts are recorded separately, delete safety checks both the canonical local pin bit and protected remote-pin states, and restored-backup rehearsal repairs missing/mismatched canonical rows | `app/modules/database/models/storageObject.ts` | +| Integration assets | Asset / AssetIdempotencyKey | userId,storageId unique owner identity; userId,sha256,id owner-visible hash preflight; userId,namespace,key unique idempotency identity | asset rows reference existing content/storage bytes; owner-scoped hash lookup avoids disclosing global hash existence | `app/modules/asset/models.ts` | +| Integration asset batches | AssetBatch / AssetBatchItem | userId,idempotencyKey unique batch identity; assetBatchId,logicalId unique item identity; assetBatchId,status,id completion scan | batch preflight is bounded by the submitted item list and completion loads only one owner-scoped batch with its indexed items | `app/modules/asset/models.ts` | | Storage object references | StorageObjectReference | sourceStorageId,targetStorageId,referenceType unique edge; targetStorageId,referenceType reverse lookup; sourceStorageId cleanup lookup | model-sync-created child-reference edge table can retain generated-output parent-to-child storage links separately from user-library content rows | `app/modules/database/models/storageObjectReference.ts` | | Storage space snapshots | StorageSpaceSnapshot | createdAt,id latest-snapshot scan; userId,createdAt,id operator history scan | model-sync cached storage analyzer snapshots store the bounded analyzer payload with list limit, duration, and optional refresher user | `app/modules/database/models/storageSpaceSnapshot.ts` | | Object cache | Object | storageId,resolveProp unique cache key | active data-structure cache keys resolved and unresolved storage paths separately | `app/modules/database/models/object.ts` | diff --git a/docs/decisions/agent-friendly-api-contract.md b/docs/decisions/agent-friendly-api-contract.md new file mode 100644 index 00000000..36aca960 --- /dev/null +++ b/docs/decisions/agent-friendly-api-contract.md @@ -0,0 +1,23 @@ +# Agent-Friendly API Contract Decision + +Status: accepted and implemented for API schema version 1. + +- `GET /.well-known/geesome` is the origin-only bootstrap route. +- `GEESOME_PUBLIC_URL` is the trusted absolute public origin and + `GEESOME_API_BASE_PATH` is the externally routed API prefix. Request `Host` + is never used to construct advertised URLs. +- New integration failures use `application/problem+json` with stable codes and + request IDs. +- `Content` remains the owner/library record; `StorageObject` remains physical + metadata; `Asset` records the integration-facing owner, CID, SHA-256, size, + MIME type, logical path, and pin state. +- Idempotency is unique by owner, endpoint namespace, and key. Reuse with a + different normalized request fingerprint is a conflict. +- Multipart SHA-256 and byte count are calculated while writing the temporary + file and verified before content publication. +- Asset creation returns `201`; deferred work returns `202` with `Location` and + `Retry-After`; operation states are `pending`, `running`, `succeeded`, + `failed`, and `cancelled`. +- Batch completion emits deterministically ordered JSON and binds the immutable + manifest with SHA-256. Signing remains deferred until key lifecycle policy is + defined. diff --git a/docs/implemented.md b/docs/implemented.md index e660c348..183f6a45 100644 --- a/docs/implemented.md +++ b/docs/implemented.md @@ -10,7 +10,7 @@ Current unfinished work lives in [todo.md](./todo.md). Detailed design and operational notes remain in their dedicated documents and module-local `docs` directories. -Last consolidated: 2026-07-30. +Last consolidated: 2026-07-31. ## Runtime And Test Foundation @@ -92,6 +92,40 @@ follow-ups in the active production-tuning section. - `/v1`, conventional OpenAPI paths, documentation headers, and IPFS-published links expose API and module documentation from a running node. +## Agent-Friendly Public Asset API + +- `GET /.well-known/geesome` advertises absolute, trusted public URLs, + capabilities, limits, storage behavior, compatibility links, OpenAPI, and + health without trusting the inbound Host header. +- Request IDs and RFC 9457-style problem documents standardize new route, + authentication, parser, content, and storage errors without returning secret + or internal exception details. +- Scoped integration credentials expose safe current-key metadata and throttled + `lastUsedAt`; asset, operation, and batch routes enforce documented scopes. +- Immutable asset uploads require SHA-256, calculate digest and bytes while + streaming to temporary storage, reject mismatches before publication, and + persist owner-scoped idempotency with deterministic replay/conflict behavior. +- Asset reads expose stable metadata and immutable content `HEAD`/`GET` responses + include SHA-256 `Content-Digest`, CID ETag, storage ID, request ID, and + year-long immutable caching. +- Stable operation resources provide `pending`, `running`, `succeeded`, + `failed`, and `cancelled` states with `201`/`202`, `Location`, and + `Retry-After` semantics for asset creation. +- Resumable owner-scoped batches preflight existing assets, bind item uploads, + avoid duplicate asset rows, and store deterministically ordered SHA-bound + manifests through the existing immutable content path. +- All nginx templates expose the canonical bootstrap route; the Docker gate + includes a production-shaped proxy and verifies discovery, OpenAPI, upload, + replay, immutable reads, async polling, and 50-item batch completion. +- OpenAPI now includes public servers, required multipart/header fields, + success/problem statuses, problem schema, and required scope metadata. Curl + and executable Node examples begin with discovery rather than guessing a + deployment prefix. +- Backpack Game Core's shared Geesome provider and the Meat Master character + publication consumer now start from `/.well-known/geesome`, upload through + the advertised `/assets` contract with SHA-256 and idempotency, and preserve + immutable CID read-back verification without a hardcoded `/api` prefix. + ## Static-Site Foundation - Generated group sites prefer the group avatar as favicon and retain the diff --git a/docs/security-route-inventory.md b/docs/security-route-inventory.md index 141189d5..82c6c63d 100644 --- a/docs/security-route-inventory.md +++ b/docs/security-route-inventory.md @@ -8,10 +8,10 @@ This file is generated by `npm run security:route-inventory:update`. Do not hand ## Summary -- API route registrations: 255 +- API route registrations: 263 - Public registrations: 42 -- Authorized registrations: 213 -- Authorized registrations without nearby core-permission checks: 135 +- Authorized registrations: 221 +- Authorized registrations without nearby core-permission checks: 143 Interpretation notes: @@ -71,6 +71,7 @@ Interpretation notes: | api | POST | `/v1/admin/set-user-limit` | authorized | - | token only; module/user ownership checks expected | `app/modules/api/api.ts` | | api | GET | `/v1/api/v0/refs*` | public | - | public entrypoint | `app/modules/api/api.ts` | | api | POST | `/v1/get-user-by-api-token` | authorized | - | token only; module/user ownership checks expected | `app/modules/api/api.ts` | +| api | GET | `/v1/integrations/credentials/current` | authorized | - | token only; module/user ownership checks expected | `app/modules/api/api.ts` | | api | GET | `/v1/ipld/*` | public | - | public entrypoint | `app/modules/api/api.ts` | | api | GET | `/v1/is-empty` | public | - | public entrypoint | `app/modules/api/api.ts` | | api | POST | `/v1/login/password` | public | - | authentication flow; public entrypoint | `app/modules/api/api.ts` | @@ -85,6 +86,13 @@ Interpretation notes: | api | GET | `/v1/user/permissions/core/is-have/:permissionName` | authorized | - | token only; module/user ownership checks expected | `app/modules/api/api.ts` | | api | POST | `/v1/user/regenerate-previews` | authorized | - | token only; module/user ownership checks expected | `app/modules/api/api.ts` | | api | POST | `/v1/user/update` | authorized | - | token only; module/user ownership checks expected | `app/modules/api/api.ts` | +| asset | POST | `/v1/asset-batches` | authorized | - | token only; module/user ownership checks expected | `app/modules/asset/api.ts` | +| asset | GET | `/v1/asset-batches/:batchId` | authorized | - | token only; module/user ownership checks expected | `app/modules/asset/api.ts` | +| asset | POST | `/v1/asset-batches/:batchId/complete` | authorized | - | token only; module/user ownership checks expected | `app/modules/asset/api.ts` | +| asset | POST | `/v1/assets` | authorized | - | token only; module/user ownership checks expected | `app/modules/asset/api.ts` | +| asset | GET | `/v1/assets/:storageId` | authorized | - | token only; module/user ownership checks expected | `app/modules/asset/api.ts` | +| asyncOperation | GET | `/v1/operations/:id` | authorized | - | token only; module/user ownership checks expected | `app/modules/asyncOperation/api.ts` | +| asyncOperation | POST | `/v1/operations/:id/cancel` | authorized | - | token only; module/user ownership checks expected | `app/modules/asyncOperation/api.ts` | | asyncOperation | POST | `/v1/user/cancel-async-operation/:id` | authorized | - | token only; module/user ownership checks expected | `app/modules/asyncOperation/api.ts` | | asyncOperation | POST | `/v1/user/find-async-operations` | authorized | - | token only; module/user ownership checks expected | `app/modules/asyncOperation/api.ts` | | asyncOperation | POST | `/v1/user/get-async-operation/:id` | authorized | - | token only; module/user ownership checks expected | `app/modules/asyncOperation/api.ts` | diff --git a/docs/todo.md b/docs/todo.md index 4c16fec0..ac89cb97 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -254,36 +254,6 @@ Verification: - Focused authorization and abuse tests for every changed route. - -## Agent-Friendly Public API - -Goal: let an integration discover, authenticate, upload, verify, and resume -immutable asset publication when it starts with only the public site origin. - -Source of truth: - -- [Recommendations](./agent-friendly-api-recommendations.md) -- [Implementation plan](./agent-friendly-api-implementation-plan.md) - -Delivery order: - -1. Freeze versioned response fixtures and public-origin rules. -2. Standardize request IDs/problems and expose absolute discovery through the - production-shaped proxy. -3. Add the idempotent SHA-256-verified asset façade and verifiable reads. -4. Normalize operation resources and least-privilege integration credentials. -5. Add resumable batches, deterministic hash-bound manifests, executable - examples, and the Meat Master consumer contract smoke. - -Verification: - -- Use the per-workstream gates and verification matrix in the implementation - plan. -- Run API docs generation and route/security inventory checks for route changes. -- Run migration integrity/scalability checks for asset and batch persistence. -- Run the final cross-module suite with `npm run test:docker`. - - ## Static Sites: Settings And Delivery diff --git a/examples/agent-friendly-assets.mjs b/examples/agent-friendly-assets.mjs new file mode 100644 index 00000000..0db97177 --- /dev/null +++ b/examples/agent-friendly-assets.mjs @@ -0,0 +1,41 @@ +import {createHash, randomUUID} from 'node:crypto'; +import {readFile} from 'node:fs/promises'; +import path from 'node:path'; + +const [origin, token, filePath] = process.argv.slice(2); +if (!origin || !token || !filePath) { + console.error('Usage: node examples/agent-friendly-assets.mjs '); + process.exit(2); +} + +const discovery = await getJson(`${origin.replace(/\/+$/, '')}/.well-known/geesome`); +const bytes = await readFile(filePath); +const expectedSha256 = createHash('sha256').update(bytes).digest('hex'); +const form = new FormData(); +form.append('file', new Blob([bytes]), path.basename(filePath)); +form.append('expectedSha256', expectedSha256); +form.append('logicalPath', path.basename(filePath)); + +const response = await fetch(`${discovery.apiBaseUrl}/assets`, { + method: 'POST', + headers: {Authorization: `Bearer ${token}`, 'Idempotency-Key': `example:${randomUUID()}`}, + body: form +}); +if (!response.ok) { + throw new Error(`upload failed (${response.status}): ${await response.text()}`); +} +const asset = await response.json(); +const head = await fetch(asset.urls.content, {method: 'HEAD'}); +const expectedDigest = `sha-256=:${Buffer.from(expectedSha256, 'hex').toString('base64')}:`; +if (!head.ok || head.headers.get('content-digest') !== expectedDigest) { + throw new Error('immutable read digest verification failed'); +} +console.log(JSON.stringify({discovery: discovery.apiBaseUrl, asset, verified: true}, null, 2)); + +async function getJson(url) { + const response = await fetch(url); + if (!response.ok) { + throw new Error(`discovery failed (${response.status}): ${await response.text()}`); + } + return response.json(); +} diff --git a/package.json b/package.json index f63fb4d5..1ab1f775 100644 --- a/package.json +++ b/package.json @@ -33,6 +33,7 @@ "activitypub:bluesky-bridge-smoke": "node --import tsx --experimental-global-customevent check/activityPubBlueskyBridgeSmoke.ts", "activitypub:remote-server-smoke": "node --import tsx --experimental-global-customevent check/activityPubRemoteServerSmoke.ts", "activitypub:ownership-challenge-smoke": "node --import tsx --experimental-global-customevent check/activityPubOwnershipChallengeSmoke.ts", + "agent-api:example": "node examples/agent-friendly-assets.mjs", "bluesky:atproto-smoke": "node --import tsx --experimental-global-customevent check/blueskyAtProtoPublicSmoke.ts", "bluesky:credentialed-smoke": "node --import tsx --experimental-global-customevent check/blueskyCredentialedSmoke.ts", "todo:context": "node --import tsx check/todoSectionContext.ts", diff --git a/test/agent-api-nginx.conf b/test/agent-api-nginx.conf new file mode 100644 index 00000000..e44f7713 --- /dev/null +++ b/test/agent-api-nginx.conf @@ -0,0 +1,17 @@ +events {} +http { + server { + listen 80; + client_max_body_size 2000m; + resolver 127.0.0.11 valid=1s; + set $api_upstream test:7772; + location = /.well-known/geesome { proxy_pass http://$api_upstream; } + location = /openapi.json { proxy_pass http://$api_upstream; } + location = /.well-known/openapi.json { proxy_pass http://$api_upstream; } + location /api/ { + rewrite ^/api/(.*)$ /$1 break; + proxy_pass http://$api_upstream; + } + location /ipfs/ { proxy_pass http://$api_upstream; } + } +} diff --git a/test/agentFriendlyApiIntegration.test.ts b/test/agentFriendlyApiIntegration.test.ts new file mode 100644 index 00000000..a885b975 --- /dev/null +++ b/test/agentFriendlyApiIntegration.test.ts @@ -0,0 +1,247 @@ +import assert from 'node:assert'; +import {createHash} from 'node:crypto'; +import {IGeesomeApp} from '../app/interface.js'; +import {getApiKeyScopes} from '../app/modules/api/integrationScopes.js'; +import {getAssetRequestHash} from '../app/modules/asset/index.js'; + +describe('agent-friendly asset integration', function () { + this.timeout(60_000); + + let app: IGeesomeApp; + let user; + let apiKey; + let apiToken: string; + const proxyOrigin = process.env.AGENT_API_PROXY_ORIGIN; + const advertisedOrigin = proxyOrigin || 'https://geesome.example'; + const apiPort = Number(process.env.PORT || 7796); + + beforeEach(async () => { + const appConfig = (await import('../app/config.js')).default; + appConfig.storageConfig.jsNode.pass = 'test test test test test test test test test test'; + app = await (await import('../app/index.js')).default({ + storageConfig: appConfig.storageConfig, + port: apiPort, + apiConfig: {publicUrl: advertisedOrigin, publicBasePath: '/api/v1'} + }); + await app.flushDatabase(); + const setup = await app.setup({email: 'asset-admin@example.com', name: 'asset-admin', password: 'admin'}); + user = setup.user; + apiToken = await app.generateUserApiKey(user.id, { + title: 'asset publisher', + scopes: ['assets:write', 'assets:read-private', 'operations:read', 'asset-batches:write'] + }); + apiKey = (await app.getUserByApiToken(apiToken)).apiKey; + }); + + it('serves the upload and immutable-read contract over HTTP', async () => { + const bytes = Buffer.from('agent-friendly-http-asset'); + const expectedSha256 = sha256(bytes); + const contentCountBefore = await (app.ms.database as any).models.Content.count(); + const mismatchForm = new FormData(); + mismatchForm.append('file', new Blob([bytes], {type: 'text/plain'}), 'http-asset.txt'); + mismatchForm.append('expectedSha256', '0'.repeat(64)); + const apiBaseUrl = proxyOrigin ? `${proxyOrigin}/api/v1` : `http://127.0.0.1:${apiPort}/v1`; + const mismatchResponse = await fetch(`${apiBaseUrl}/assets`, { + method: 'POST', + headers: {Authorization: `Bearer ${apiToken}`, 'Idempotency-Key': 'release:http-mismatch'}, + body: mismatchForm + }); + assert.equal(mismatchResponse.status, 422); + assert.match(mismatchResponse.headers.get('content-type') || '', /application\/problem\+json/); + assert.equal((await mismatchResponse.json() as any).code, 'asset_digest_mismatch'); + assert.equal(await (app.ms.database as any).models.Content.count(), contentCountBefore); + + const form = new FormData(); + form.append('file', new Blob([bytes], {type: 'text/plain'}), 'http-asset.txt'); + form.append('expectedSha256', expectedSha256); + form.append('logicalPath', 'release/http-asset.txt'); + + const createResponse = await fetch(`${apiBaseUrl}/assets`, { + method: 'POST', + headers: {Authorization: `Bearer ${apiToken}`, 'Idempotency-Key': 'release:http-asset'}, + body: form + }); + assert.equal(createResponse.status, 201); + assert(createResponse.headers.get('x-request-id')); + const asset: any = await createResponse.json(); + assert.equal(asset.sha256, expectedSha256); + assert.equal(asset.bytes, bytes.length); + + const replayForm = new FormData(); + replayForm.append('file', new Blob([bytes], {type: 'text/plain'}), 'http-asset.txt'); + replayForm.append('expectedSha256', expectedSha256); + replayForm.append('logicalPath', 'release/http-asset.txt'); + const replayResponse = await fetch(`${apiBaseUrl}/assets`, { + method: 'POST', + headers: {Authorization: `Bearer ${apiToken}`, 'Idempotency-Key': 'release:http-asset'}, + body: replayForm + }); + assert.equal(replayResponse.status, 200); + assert.equal((await replayResponse.json() as any).storageId, asset.storageId); + + const contentUrl = proxyOrigin ? asset.urls.content : `http://127.0.0.1:${apiPort}/ipfs/${asset.storageId}`; + const contentResponse = await fetch(contentUrl, {method: 'HEAD'}); + assert.equal(contentResponse.status, 200); + assert.equal(contentResponse.headers.get('content-digest'), `sha-256=:${Buffer.from(expectedSha256, 'hex').toString('base64')}:`); + assert.equal(contentResponse.headers.get('etag'), `\"${asset.storageId}\"`); + assert.match(contentResponse.headers.get('cache-control') || '', /immutable/); + }); + + it('resolves every bootstrap URL through the production-shaped proxy', async function () { + if (!proxyOrigin) { + this.skip(); + } + const discoveryResponse = await fetch(`${proxyOrigin}/.well-known/geesome`); + assert.equal(discoveryResponse.status, 200); + const discovery: any = await discoveryResponse.json(); + assert.equal(discovery.apiBaseUrl, `${proxyOrigin}/api/v1`); + for (const url of [discovery.openapiUrl, discovery.healthUrl]) { + const response = await fetch(url); + assert.equal(response.status, 200, url); + } + const openapi = await getJson(discovery.openapiUrl); + assert.equal(openapi.servers[0].url, discovery.apiBaseUrl); + }); + + it('returns and resolves a stable async operation resource', async () => { + const bytes = Buffer.from('agent-friendly-async-asset'); + const expectedSha256 = sha256(bytes); + const form = new FormData(); + form.append('file', new Blob([bytes], {type: 'text/plain'}), 'async-asset.txt'); + form.append('expectedSha256', expectedSha256); + form.append('async', 'true'); + const apiBaseUrl = proxyOrigin ? `${proxyOrigin}/api/v1` : `http://127.0.0.1:${apiPort}/v1`; + const response = await fetch(`${apiBaseUrl}/assets`, { + method: 'POST', + headers: {Authorization: `Bearer ${apiToken}`, 'Idempotency-Key': 'release:http-async'}, + body: form + }); + assert.equal(response.status, 202); + assert.equal(response.headers.get('retry-after'), '2'); + const accepted: any = await response.json(); + assert.match(accepted.operationId, /^op_\d+$/); + const operationNumber = accepted.operationId.slice(3); + let operation: any; + for (let attempt = 0; attempt < 50; attempt += 1) { + const poll = await fetch(`${apiBaseUrl}/operations/${operationNumber}`, {headers: {Authorization: `Bearer ${apiToken}`}}); + assert.equal(poll.status, 200); + operation = await poll.json(); + if (['succeeded', 'failed', 'cancelled'].includes(operation.status)) { + break; + } + await new Promise(resolve => setTimeout(resolve, 100)); + } + assert.equal(operation.status, 'succeeded'); + assert.equal(operation.result.sha256, expectedSha256); + }); + + afterEach(async () => { + await app.stop(); + }); + + it('stores, replays, verifies, batches, and completes immutable assets', async () => { + const firstBytes = Buffer.from('agent-friendly-asset-one'); + const firstSha = sha256(firstBytes); + const firstHash = getAssetRequestHash({ + sha256: firstSha, + bytes: firstBytes.length, + mimeType: 'text/plain', + logicalPath: 'release/one.txt', + previewPolicy: 'none', + batchId: null, + logicalId: null + }); + const firstRequest = await app.ms.asset.prepareAssetRequest(user.id, 'release:first', firstHash); + const first = await app.ms.asset.createAssetFromUpload(user.id, firstBytes, 'one.txt', { + userApiKeyId: apiKey.id, + idempotencyRequestId: firstRequest.request.id, + sha256: firstSha, + bytes: firstBytes.length, + mimeType: 'text/plain', + logicalPath: 'release/one.txt', + previewPolicy: 'none' + }); + + assert.equal(first.sha256, firstSha); + assert.equal(first.bytes, firstBytes.length); + assert.equal(first.created, true); + assert.equal(first.urls.content, `${advertisedOrigin}/ipfs/${first.storageId}`); + const storageObject = await app.ms.database.getStorageObjectByStorageId(first.storageId); + assert.equal(storageObject.sha256, firstSha); + + const replay = await app.ms.asset.prepareAssetRequest(user.id, 'release:first', firstHash); + assert.equal(replay.created, false); + assert.equal(replay.asset.storageId, first.storageId); + await assert.rejects( + app.ms.asset.prepareAssetRequest(user.id, 'release:first', `${firstHash.slice(0, -1)}${firstHash.endsWith('0') ? '1' : '0'}`), + (error: any) => error.code === 'idempotency_key_conflict' + ); + await assert.rejects( + app.ms.asset.getAsset(user.id + 999, first.storageId), + (error: any) => error.code === 'asset_not_found' + ); + + const secondBytes = Buffer.from('agent-friendly-asset-two'); + const secondSha = sha256(secondBytes); + const secondHash = getAssetRequestHash({ + sha256: secondSha, + bytes: secondBytes.length, + mimeType: 'text/plain', + logicalPath: 'release/two.txt', + previewPolicy: 'none', + batchId: null, + logicalId: null + }); + const secondRequest = await app.ms.asset.prepareAssetRequest(user.id, 'release:second', secondHash); + const second = await app.ms.asset.createAssetFromUpload(user.id, secondBytes, 'two.txt', { + userApiKeyId: apiKey.id, + idempotencyRequestId: secondRequest.request.id, + sha256: secondSha, + bytes: secondBytes.length, + mimeType: 'text/plain', + logicalPath: 'release/two.txt', + previewPolicy: 'none' + }); + + const batch = await app.ms.asset.createBatch(user.id, {items: [ + {logicalId: 'one', logicalPath: 'release/one.txt', sha256: firstSha, bytes: firstBytes.length, mimeType: 'text/plain'}, + {logicalId: 'two', logicalPath: 'release/two.txt', sha256: secondSha, bytes: secondBytes.length, mimeType: 'text/plain'} + ]}, 'release:batch'); + assert.equal(batch.items.filter(item => item.requiredUpload).length, 0); + assert.deepEqual(batch.items.map(item => item.asset.storageId).sort(), [first.storageId, second.storageId].sort()); + + const completed = await app.ms.asset.completeBatch(user.id, batch.batchId, apiKey.id); + assert.equal(completed.status, 'completed'); + assert.match(completed.manifest.sha256, /^[a-f0-9]{64}$/); + assert.equal(completed.manifest.url, `${advertisedOrigin}/ipfs/${completed.manifest.storageId}`); + const completionReplay = await app.ms.asset.completeBatch(user.id, batch.batchId, apiKey.id); + assert.equal(completionReplay.manifest.sha256, completed.manifest.sha256); + + const fiftyItems = Array.from({length: 50}, (_, index) => ({ + logicalId: `item-${String(index).padStart(2, '0')}`, + logicalPath: `release/item-${index}.txt`, + sha256: index % 2 ? firstSha : secondSha, + bytes: index % 2 ? firstBytes.length : secondBytes.length, + mimeType: 'text/plain' + })); + const resumedBatch = await app.ms.asset.createBatch(user.id, {items: fiftyItems}, 'release:fifty'); + assert.equal(resumedBatch.items.length, 50); + assert.equal(resumedBatch.items.filter(item => item.requiredUpload).length, 0); + const resumedComplete = await app.ms.asset.completeBatch(user.id, resumedBatch.batchId, apiKey.id); + assert.equal(resumedComplete.status, 'completed'); + const [assetCountRows]: any = await (app.ms.database as any).sequelize.query('SELECT COUNT(*)::int AS count FROM assets'); + assert.equal(assetCountRows[0].count, 2); + assert(getApiKeyScopes(apiKey).includes('assets:write')); + assert(apiKey.lastUsedAt); + }); +}); + +function sha256(value: Buffer): string { + return createHash('sha256').update(value).digest('hex'); +} + +async function getJson(url: string) { + const response = await fetch(url); + assert.equal(response.status, 200); + return response.json(); +} diff --git a/test/agentFriendlyApiUnit.test.ts b/test/agentFriendlyApiUnit.test.ts new file mode 100644 index 00000000..65bbb9ac --- /dev/null +++ b/test/agentFriendlyApiUnit.test.ts @@ -0,0 +1,92 @@ +import assert from 'node:assert'; +import {ApiProblemError, getApiProblem, getRequestId} from '../app/modules/api/problem.js'; +import { + getApiKeyScopes, + normalizeIntegrationScopes, + requireIntegrationScopes, + serializeApiKey +} from '../app/modules/api/integrationScopes.js'; +import { + getAssetRequestHash, + validateBytes, + validateLogicalPath, + validateMimeType, + validateSha256 +} from '../app/modules/asset/index.js'; +import {CorePermissionName} from '../app/modules/database/interface.js'; +import {buildOpenApiFromApiDoc} from '../app/apiDocSpec.js'; +import {getPublicApiContext} from '../app/modules/api/publicUrls.js'; + +describe('agent-friendly API contracts', function () { + this.timeout(30_000); + it('preserves valid request IDs and replaces invalid values', () => { + assert.equal(getRequestId('consumer:release-1'), 'consumer:release-1'); + assert.match(getRequestId('invalid request id'), /^req_[0-9a-f-]+$/); + }); + + it('serializes stable problem details without internal error text', () => { + const explicit = getApiProblem(new ApiProblemError(422, 'asset_digest_mismatch', 'Asset digest mismatch', 'Expected bytes did not match.'), 'req_test'); + assert.equal(explicit.status, 422); + assert.equal(explicit.code, 'asset_digest_mismatch'); + assert.equal(explicit.requestId, 'req_test'); + + const internal = getApiProblem(new Error('database_password_secret'), 'req_internal'); + assert.equal(internal.status, 500); + assert.equal(internal.code, 'internal_error'); + assert.equal(internal.detail, 'The server could not complete the request.'); + }); + + it('maps integration scopes to legacy permissions and redacts key secrets', () => { + assert.deepEqual(normalizeIntegrationScopes(['operations:read', 'assets:write']), ['assets:write', 'operations:read']); + const legacyKey: any = {permissions: JSON.stringify([CorePermissionName.UserSaveData])}; + assert(getApiKeyScopes(legacyKey).includes('assets:write')); + requireIntegrationScopes(legacyKey, ['assets:write', 'operations:read']); + assert.throws( + () => requireIntegrationScopes({permissions: '[]'} as any, ['assets:write']), + (error: ApiProblemError) => error.code === 'insufficient_scope' && error.status === 403 + ); + const serialized = serializeApiKey({ + id: 7, + title: 'publisher', + valueHash: 'secret-hash', + scopes: JSON.stringify(['assets:write']), + isDisabled: false + }); + assert.equal(serialized.id, 7); + assert.deepEqual(serialized.scopes, ['assets:write']); + assert.equal((serialized as any).valueHash, undefined); + }); + + it('validates asset metadata and hashes a stable request contract', () => { + const digest = 'a'.repeat(64); + assert.equal(validateSha256(digest.toUpperCase()), digest); + assert.equal(validateBytes('42'), 42); + assert.equal(validateMimeType('IMAGE/WEBP'), 'image/webp'); + assert.equal(validateLogicalPath('/games/character.webp'), 'games/character.webp'); + assert.throws(() => validateLogicalPath('../secret'), (error: ApiProblemError) => error.code === 'asset_logical_path_invalid'); + + const first = getAssetRequestHash({sha256: digest, bytes: 42, mimeType: 'image/webp', logicalPath: 'character.webp'}); + const replay = getAssetRequestHash({sha256: digest, bytes: 42, mimeType: 'image/webp', logicalPath: 'character.webp'}); + const conflict = getAssetRequestHash({sha256: digest, bytes: 43, mimeType: 'image/webp', logicalPath: 'character.webp'}); + assert.equal(first, replay); + assert.notEqual(first, conflict); + assert.match(first, /^[a-f0-9]{64}$/); + }); + + it('publishes actionable OpenAPI upload requirements and responses', () => { + const spec = buildOpenApiFromApiDoc('v1', undefined, 'https://geesome.example/api/v1'); + const create = spec.paths['/assets'].post; + assert.equal(spec.servers[0].url, 'https://geesome.example/api/v1'); + assert.deepEqual(create['x-required-scopes'], ['assets:write']); + assert.deepEqual(create.requestBody.content['multipart/form-data'].schema.required, ['file', 'expectedSha256']); + assert(create.parameters.some(parameter => parameter.name === 'Idempotency-Key' && parameter.required)); + assert(create.responses['201']); + assert(create.responses['401']); + assert(create.responses['403']); + }); + + it('rejects unsafe public-origin configuration', () => { + assert.throws(() => getPublicApiContext({config: {apiConfig: {publicUrl: 'javascript:alert(1)'}}} as any), /GEESOME_PUBLIC_URL/); + assert.throws(() => getPublicApiContext({config: {apiConfig: {publicUrl: 'https://user:pass@example.com'}}} as any), /GEESOME_PUBLIC_URL/); + }); +}); diff --git a/test/apiCallbackHandling.test.ts b/test/apiCallbackHandling.test.ts index 6d5bb4d8..897145f6 100644 --- a/test/apiCallbackHandling.test.ts +++ b/test/apiCallbackHandling.test.ts @@ -30,7 +30,8 @@ describe("api callback handling", function () { res.send({ok: true}, 200); }); - assert.deepEqual(response.sent, [[{ok: true}, 200]]); + assert.equal(response.statusCode, 200); + assert.deepEqual(response.sent, [[{ok: true}]]); } finally { api?.stop(); } diff --git a/test/apiHeaders.test.ts b/test/apiHeaders.test.ts index b5692a65..0df6f857 100644 --- a/test/apiHeaders.test.ts +++ b/test/apiHeaders.test.ts @@ -56,7 +56,7 @@ describe("api headers", function () { try { api = await apiModule({ - config: {port}, + config: {port, apiConfig: {publicUrl: 'https://geesome.example', publicBasePath: '/api/v1'}}, docsStorageId: "docs-root", ms: { database: {getUsersCount: async () => 0}, @@ -68,19 +68,26 @@ describe("api headers", function () { const {res, body} = await requestJson(port, "/v1"); assert.equal(res.statusCode, 200); - assert.equal(res.headers["x-api-docs"], "/ipfs/docs-root"); - assert.equal(res.headers["x-api-docs-openapi"], "/v1/openapi.json"); - assert.equal(res.headers["x-api-docs-discovery"], "/v1"); - assert.equal(res.headers["x-api-docs-ipfs"], "/ipfs/docs-root"); + assert.equal(res.headers["x-api-docs"], "https://geesome.example/ipfs/docs-root"); + assert.equal(res.headers["x-api-docs-openapi"], "https://geesome.example/api/v1/openapi.json"); + assert.equal(res.headers["x-api-docs-discovery"], "https://geesome.example/api/v1"); + assert.equal(res.headers["x-api-docs-ipfs"], "https://geesome.example/ipfs/docs-root"); + assert.match(String(res.headers["x-request-id"]), /^req_/); assert.match(String(res.headers.link), /rel="service-desc"/); assert.match(String(res.headers.link), /modules\.md/); - assert.equal(body.docs.openapi, "/v1/openapi.json"); - assert.equal(body.docs.apidoc, "/v1/apidoc.json"); - assert.equal(body.docs.apiHtml, "/ipfs/docs-root"); - assert.equal(body.docs.repoDocs, "/ipfs/docs-root/README.md"); - assert.equal(body.docs.moduleDocs, "/ipfs/docs-root/modules.md"); - assert.equal(body.docs.agentMap, "/ipfs/docs-root/agent-map.md"); - assert.equal(body.docs.conventionalOpenapi.wellKnown, "/.well-known/openapi.json"); + assert.equal(body.docs.openapi, "https://geesome.example/api/v1/openapi.json"); + assert.equal(body.docs.apidoc, "https://geesome.example/api/v1/apidoc.json"); + assert.equal(body.docs.apiHtml, "https://geesome.example/ipfs/docs-root"); + assert.equal(body.docs.repoDocs, "https://geesome.example/ipfs/docs-root/README.md"); + assert.equal(body.docs.moduleDocs, "https://geesome.example/ipfs/docs-root/modules.md"); + assert.equal(body.docs.agentMap, "https://geesome.example/ipfs/docs-root/agent-map.md"); + assert.equal(body.docs.conventionalOpenapi.wellKnown, "https://geesome.example/.well-known/openapi.json"); + + const discovery = await requestJson(port, "/.well-known/geesome"); + assert.equal(discovery.body.apiBaseUrl, "https://geesome.example/api/v1"); + assert.equal(discovery.body.openapiUrl, "https://geesome.example/api/v1/openapi.json"); + assert.equal(discovery.body.gatewayBaseUrl, "https://geesome.example"); + assert.equal(discovery.body.storage.contentDigest, "sha-256"); } finally { if (previousPort === undefined) { delete process.env.PORT; diff --git a/test/asyncOperationSecurityUnit.test.ts b/test/asyncOperationSecurityUnit.test.ts index 2a3f2734..c6f8a8d7 100644 --- a/test/asyncOperationSecurityUnit.test.ts +++ b/test/asyncOperationSecurityUnit.test.ts @@ -346,7 +346,9 @@ describe("async operation ownership controls", function () { assert.equal(operations[0].inProcess, false); assert.equal(operations[0].contentId, 42); assert.equal(operations[1].inProcess, false); - assert.equal(operations[1].errorMessage, "expected_async_failure"); + assert.equal(operations[1].errorType, "internal_error"); + assert.equal(operations[1].errorMessage, "The server could not complete the request."); + assert.equal(JSON.parse(operations[1].output).problem.code, "internal_error"); }); it("closes completed queue heads before processing the next module queue item", async () => { diff --git a/test/authRouteErrors.test.ts b/test/authRouteErrors.test.ts index 00a4036e..774b8cfc 100644 --- a/test/authRouteErrors.test.ts +++ b/test/authRouteErrors.test.ts @@ -25,7 +25,8 @@ describe("auth route errors", function () { } as any, response); await flushAsyncHandlers(); - assert.deepEqual(response.sent, [[403]]); + assert.equal(response.sent[0][1], 403); + assert.equal(response.sent[0][0].code, 'forbidden'); }); assert.deepEqual(consoleErrors, []); @@ -58,7 +59,8 @@ describe("auth route errors", function () { } as any, response); await flushAsyncHandlers(); - assert.deepEqual(response.sent, [[403]]); + assert.equal(response.sent[0][1], 403); + assert.equal(response.sent[0][0].code, 'forbidden'); }); assert.deepEqual(consoleErrors, []); diff --git a/test/contentHeaders.test.ts b/test/contentHeaders.test.ts index a814801e..85c609c5 100644 --- a/test/contentHeaders.test.ts +++ b/test/contentHeaders.test.ts @@ -33,6 +33,7 @@ describe("content headers", function () { let statusCode = 0; let ended = false; const contentSize = 5; + const sha256 = 'a'.repeat(64); const content = await contentModule({ checkModules: () => null, ms: { @@ -48,6 +49,7 @@ describe("content headers", function () { database: { getSharedStorageMetadataByStorageId: async () => ({ storageId: "file.txt", + sha256, mimeType: ContentMimeType.Text, size: contentSize }) @@ -78,6 +80,10 @@ describe("content headers", function () { assert.equal(headers["x-ipfs-datasize"], contentSize); assert.equal(headers["Content-Type"], ContentMimeType.Text); assert.equal(headers["X-Content-Type-Options"], "nosniff"); + assert.equal(headers["ETag"], '"file.txt"'); + assert.equal(headers["X-Geesome-Storage-Id"], "file.txt"); + assert.equal(headers["Content-Digest"], `sha-256=:${Buffer.from(sha256, 'hex').toString('base64')}:`); + assert.equal(headers["cache-control"], "public, max-age=31536000, immutable"); assert.equal(ended, true); }); @@ -536,7 +542,8 @@ describe("content headers", function () { setHeader: () => null } as any, "missing.txt"); - assert.deepEqual(sends, [[404]]); + assert.equal(sends[0][0].code, 'content_not_found'); + assert.equal(sends[0][1], 404); assert.equal(streamRequested, false); assert.equal(statOptions.attempts, 0); }); @@ -763,7 +770,8 @@ describe("content headers", function () { setHeader: () => null } as any, "missing.txt"); - assert.deepEqual(sends, [[404]]); + assert.equal(sends[0][0].code, 'content_not_found'); + assert.equal(sends[0][1], 404); assert.equal(streamRequested, false); assert.equal(statOptions.attempts, 0); }); @@ -803,7 +811,8 @@ describe("content headers", function () { setHeader: () => null } as any, "forbidden.txt"); - assert.deepEqual(sends, [[423]]); + assert.equal(sends[0][0].code, 'content_locked'); + assert.equal(sends[0][1], 423); assert.equal(statRequested, false); }); diff --git a/test/contentRouteErrors.test.ts b/test/contentRouteErrors.test.ts index 817c2c69..34037e25 100644 --- a/test/contentRouteErrors.test.ts +++ b/test/contentRouteErrors.test.ts @@ -27,12 +27,12 @@ describe("content route errors", function () { const getResponse = getResponseStub(); await routes["GET content-data/*"]({route: "content-data/missing.txt"} as any, getResponse); await flushAsyncHandlers(); - assert.deepEqual(getResponse.sent, [[400]]); + assertProblem(getResponse, 400, 'content_request_invalid'); const headResponse = getResponseStub(); await routes["HEAD content-data/*"]({route: "content-data/missing.txt"} as any, headResponse); await flushAsyncHandlers(); - assert.deepEqual(headResponse.sent, [[400]]); + assertProblem(headResponse, 400, 'content_request_invalid'); }); assert.deepEqual(consoleErrors, []); @@ -69,12 +69,12 @@ describe("content route errors", function () { const getResponse = getResponseStub(); await getHandler({headers: {}, route: "/file.txt"} as any, getResponse); await flushAsyncHandlers(); - assert.deepEqual(getResponse.sent, [[400]]); + assertProblem(getResponse, 400, 'content_request_invalid'); const headResponse = getResponseStub(); await headHandler({headers: {}, route: "/file.txt"} as any, headResponse); await flushAsyncHandlers(); - assert.deepEqual(headResponse.sent, [[400]]); + assertProblem(headResponse, 400, 'content_request_invalid'); }); assert.deepEqual(consoleErrors, []); @@ -103,12 +103,12 @@ describe("content route errors", function () { const getResponse = getResponseStub(); await routes["UNVERSION_GET /ipns/*"]({route: "/ipns/static-id/path.txt?x=1"} as any, getResponse); await flushAsyncHandlers(); - assert.deepEqual(getResponse.sent, [[400]]); + assertProblem(getResponse, 400, 'content_request_invalid'); const headResponse = getResponseStub(); await routes["UNVERSION_HEAD /ipns/*"]({route: "/ipns/static-id/path.txt?x=1"} as any, headResponse); await flushAsyncHandlers(); - assert.deepEqual(headResponse.sent, [[400]]); + assertProblem(headResponse, 400, 'content_request_invalid'); }); assert.deepEqual(consoleErrors, []); @@ -147,6 +147,11 @@ function getResponseStub() { }; } +function assertProblem(response, status: number, code: string) { + assert.equal(response.sent[0][1], status); + assert.equal(response.sent[0][0].code, code); +} + async function captureConsoleErrors(callback) { const originalConsoleError = console.error; const consoleErrors: any[] = []; diff --git a/test/docker-compose.yml b/test/docker-compose.yml index 1ed20883..44871aea 100644 --- a/test/docker-compose.yml +++ b/test/docker-compose.yml @@ -15,6 +15,8 @@ services: condition: service_started chat_ipfs_b: condition: service_started + agent_proxy: + condition: service_started volumes: - "${GEESOME_DATA:-./.geesome-data}:/geesome-node/data" - "${GEESOME_FRONTEND_DIST:-./.geesome-frontend}:/geesome-node/frontend/docker-dist" @@ -32,6 +34,13 @@ services: - STORAGE_URL=http://go_ipfs:5001 - CHAT_TEST_STORAGE_URL_B=http://geesome_chat_ipfs_b:5001 - NPM_CONFIG_UPDATE_NOTIFIER=false + - AGENT_API_PROXY_ORIGIN=http://agent_proxy + + agent_proxy: + container_name: geesome_agent_api_proxy + image: nginx:1.27-alpine + volumes: + - "./agent-api-nginx.conf:/etc/nginx/nginx.conf:ro" postgres: container_name: geesome_db diff --git a/test/storageRouteErrors.test.ts b/test/storageRouteErrors.test.ts index 1072f3c4..f8400d72 100644 --- a/test/storageRouteErrors.test.ts +++ b/test/storageRouteErrors.test.ts @@ -25,7 +25,8 @@ describe("storage route errors", function () { await flushAsyncHandlers(); assert.equal(storageHeadersSet, true); - assert.deepEqual(response.sent, [[null, 502]]); + assert.equal(response.sent[0][0].code, 'storage_backend_unavailable'); + assert.equal(response.sent[0][1], 502); }); }); From fc80cceac191a2fe42bc9e4c8b056afc2973394b Mon Sep 17 00:00:00 2001 From: MicrowaveDev Date: Fri, 31 Jul 2026 20:19:09 +0100 Subject: [PATCH 3/3] feat: compact completed asset batches --- app/modules/asset/api.ts | 5 ++ app/modules/asset/index.ts | 84 +++++++++++++++++++++++- app/modules/asset/interface.ts | 6 ++ app/modules/asset/models.ts | 3 +- check/databaseScalabilityInventory.ts | 3 +- docs/agent-friendly-api.md | 7 ++ docs/database-scalability-inventory.md | 2 +- test/agentFriendlyApiIntegration.test.ts | 21 ++++++ 8 files changed, 126 insertions(+), 5 deletions(-) diff --git a/app/modules/asset/api.ts b/app/modules/asset/api.ts index 4e928ab4..d3b5ea27 100644 --- a/app/modules/asset/api.ts +++ b/app/modules/asset/api.ts @@ -58,6 +58,7 @@ export default (app: IGeesomeApp, assetModule: IGeesomeAssetModule) => { * @apiBody {Object[]} items Expected manifest items. * @apiSuccess {Number} batchId Batch identifier. * @apiSuccess {Object[]} items Batch item upload requirements. + * @apiSuccess {Boolean} itemsRetained Whether resumable item metadata is still inside its retention window. */ app.ms.api.onAuthorizedPost('asset-batches', async (req, res) => { requireIntegrationScopes(req.apiKey, ['asset-batches:write']); @@ -71,6 +72,9 @@ export default (app: IGeesomeApp, assetModule: IGeesomeAssetModule) => { * @apiGroup AssetBatches * @apiUse ApiKey * @apiParam {Number} batchId Batch identifier. + * @apiSuccess {String="pending","processing","completed"} status Stable batch status. + * @apiSuccess {Object[]} items Retained batch item metadata, or an empty array after compaction. + * @apiSuccess {Boolean} itemsRetained Whether resumable item metadata is still inside its retention window. */ app.ms.api.onAuthorizedGet('asset-batches/:batchId', async (req, res) => { requireIntegrationScopes(req.apiKey, ['asset-batches:write']); @@ -85,6 +89,7 @@ export default (app: IGeesomeApp, assetModule: IGeesomeAssetModule) => { * @apiParam {Number} batchId Batch identifier. * @apiSuccess {String="completed"} status Completed status. * @apiSuccess {Object} manifest Immutable hash-bound manifest. + * @apiSuccess {Boolean} itemsRetained Whether resumable item metadata is still inside its retention window. */ app.ms.api.onAuthorizedPost('asset-batches/:batchId/complete', async (req, res) => { requireIntegrationScopes(req.apiKey, ['asset-batches:write']); diff --git a/app/modules/asset/index.ts b/app/modules/asset/index.ts index 09c8546a..19cf777f 100644 --- a/app/modules/asset/index.ts +++ b/app/modules/asset/index.ts @@ -1,23 +1,55 @@ import {createHash} from 'node:crypto'; +import debug from 'debug'; import {Op} from 'sequelize'; +import {startIntervalWorker} from '../../backgroundWorker.js'; +import type {IBackgroundWorker} from '../../backgroundWorker.js'; +import helpers from '../../helpers.js'; import {IGeesomeApp} from '../../interface.js'; import {ApiProblemError} from '../api/problem.js'; import {getPublicApiContext} from '../api/publicUrls.js'; import IGeesomeAssetModule from './interface.js'; +const log = debug('geesome:app:asset'); +const completedBatchItemRetentionDays = parseNonNegativeInteger(process.env.ASSET_BATCH_ITEM_RETENTION_DAYS, 30); +const completedBatchCleanupIntervalMs = helpers.parsePositiveInteger(process.env.ASSET_BATCH_CLEANUP_INTERVAL_MS, 24 * 60 * 60 * 1000); +const completedBatchCleanupLimit = helpers.parsePositiveInteger(process.env.ASSET_BATCH_CLEANUP_LIMIT, 100); + export default async (app: IGeesomeApp) => { app.checkModules(['api', 'database', 'content', 'asyncOperation']); const database: any = app.ms.database; const models = await (await import('./models.js')).default(database.sequelize, database.models); const module = getModule(app, models); + await module.cleanupCompletedBatches(); + module.startCompletedBatchCleanupWorker(); (await import('./api.js')).default(app, module); return module; }; export function getModule(app: IGeesomeApp, models: any): IGeesomeAssetModule { + let cleanupWorker: IBackgroundWorker | null = null; + class AssetModule implements IGeesomeAssetModule { supportsBatches = true; + async stop() { + const worker = cleanupWorker; + cleanupWorker = null; + await worker?.stop(); + } + + startCompletedBatchCleanupWorker() { + if (cleanupWorker) { + return; + } + cleanupWorker = startIntervalWorker( + () => this.cleanupCompletedBatches(), + { + intervalMs: completedBatchCleanupIntervalMs, + onError: error => log('cleanupCompletedBatches error', error) + } + ); + } + async flushDatabase() { await models.AssetBatchItem.destroy({where: {}}); await models.AssetBatch.destroy({where: {}}); @@ -175,7 +207,7 @@ export function getModule(app: IGeesomeApp, models: any): IGeesomeAssetModule { if (!batch) { throw new ApiProblemError(404, 'asset_batch_not_found', 'Asset batch not found'); } - if (batch.status === 'completed') { + if (batch.status === 'completed' || batch.status === 'compacted') { return this.serializeBatch(batch); } const missing = batch.items.filter(item => !item.assetId || item.status !== 'available'); @@ -206,6 +238,44 @@ export function getModule(app: IGeesomeApp, models: any): IGeesomeAssetModule { } } + async cleanupCompletedBatches(options: any = {}) { + const retentionDays = parseNonNegativeInteger(options.retentionDays, completedBatchItemRetentionDays); + const limit = helpers.parsePositiveInteger(options.limit, completedBatchCleanupLimit); + const now = options.now instanceof Date ? options.now : new Date(); + const cutoff = new Date(now.getTime() - retentionDays * 24 * 60 * 60 * 1000); + const batches = await models.AssetBatch.findAll({ + attributes: ['id'], + where: {status: 'completed', updatedAt: {[Op.lte]: cutoff}}, + order: [['updatedAt', 'ASC'], ['id', 'ASC']], + limit + }); + let compacted = 0; + let itemsDeleted = 0; + for (const batch of batches) { + const transaction = await models.AssetBatch.sequelize.transaction(); + try { + const [claimed] = await models.AssetBatch.update( + {status: 'compacting'}, + {where: {id: batch.id, status: 'completed', updatedAt: {[Op.lte]: cutoff}}, transaction} + ); + if (!claimed) { + await transaction.commit(); + continue; + } + itemsDeleted += await models.AssetBatchItem.destroy({where: {assetBatchId: batch.id}, transaction}); + await models.AssetBatch.update({status: 'compacted'}, {where: {id: batch.id, status: 'compacting'}, transaction}); + await transaction.commit(); + compacted++; + } catch (error) { + if (!transaction.finished) { + await transaction.rollback(); + } + throw error; + } + } + return {examined: batches.length, compacted, itemsDeleted}; + } + async attachAssetToBatch(userId: number, asset: any, batchId?: number, logicalId?: string) { if (!batchId && !logicalId) { return; @@ -246,10 +316,12 @@ export function getModule(app: IGeesomeApp, models: any): IGeesomeAssetModule { serializeBatch(batch: any) { const data = typeof batch.toJSON === 'function' ? batch.toJSON() : batch; const context = getPublicApiContext(app); + const isCompacted = data.status === 'compacted'; return { schemaVersion: 1, batchId: data.id, - status: data.status, + status: isCompacted ? 'completed' : data.status, + itemsRetained: !isCompacted, manifest: data.manifestStorageId ? { storageId: data.manifestStorageId, sha256: data.manifestSha256, @@ -364,6 +436,14 @@ function sha256(value: string): string { return createHash('sha256').update(value).digest('hex'); } +function parseNonNegativeInteger(value: any, fallback: number): number { + const parsed = Number.parseInt(value, 10); + if (!Number.isFinite(parsed) || parsed < 0) { + return fallback; + } + return parsed; +} + async function setStorageObjectSha256(app: IGeesomeApp, storageId: string, digest: string) { const database: any = app.ms.database; if (typeof database.setStorageObjectSha256 === 'function') { diff --git a/app/modules/asset/interface.ts b/app/modules/asset/interface.ts index f1ef2f0f..162af82a 100644 --- a/app/modules/asset/interface.ts +++ b/app/modules/asset/interface.ts @@ -1,6 +1,8 @@ export default interface IGeesomeAssetModule { supportsBatches: boolean; + stop(): Promise; + flushDatabase(): Promise; prepareAssetRequest(userId: number, key: string, requestHash: string): Promise; @@ -16,4 +18,8 @@ export default interface IGeesomeAssetModule { getBatch(userId: number, batchId: number): Promise; completeBatch(userId: number, batchId: number, userApiKeyId: number): Promise; + + cleanupCompletedBatches(options?: any): Promise<{examined: number; compacted: number; itemsDeleted: number}>; + + startCompletedBatchCleanupWorker(): void; } diff --git a/app/modules/asset/models.ts b/app/modules/asset/models.ts index e85673b7..95358105 100644 --- a/app/modules/asset/models.ts +++ b/app/modules/asset/models.ts @@ -40,7 +40,8 @@ export default async function (sequelize: Sequelize, databaseModels: any) { } as any, { indexes: [ {name: 'asset_batches_user_key_unique', fields: ['userId', 'idempotencyKey'], unique: true}, - {name: 'asset_batches_user_created_idx', fields: ['userId', 'createdAt', 'id']} + {name: 'asset_batches_user_created_idx', fields: ['userId', 'createdAt', 'id']}, + {name: 'asset_batches_status_updated_idx', fields: ['status', 'updatedAt', 'id']} ] } as any); diff --git a/check/databaseScalabilityInventory.ts b/check/databaseScalabilityInventory.ts index b5af3bd9..12a88a88 100644 --- a/check/databaseScalabilityInventory.ts +++ b/check/databaseScalabilityInventory.ts @@ -485,10 +485,11 @@ function modelRows(): ModelRow[] { model: 'AssetBatch / AssetBatchItem', indexes: [ has(assetSource, 'asset_batches_user_key_unique') ? 'userId,idempotencyKey unique batch identity' : 'missing batch idempotency uniqueness', + has(assetSource, 'asset_batches_status_updated_idx') ? 'status,updatedAt,id retention scan' : 'missing completed-batch retention index', has(assetSource, 'asset_batch_items_batch_logical_unique') ? 'assetBatchId,logicalId unique item identity' : 'missing batch logical-item uniqueness', has(assetSource, 'asset_batch_items_batch_status_idx') ? 'assetBatchId,status,id completion scan' : 'missing batch status scan index' ], - notes: ['batch preflight is bounded by the submitted item list and completion loads only one owner-scoped batch with its indexed items'] + notes: ['batch preflight is bounded by the submitted item list and completion loads only one owner-scoped batch with its indexed items; completed item rows are compacted after a configurable 30-day retention window in bounded cleanup runs while the idempotency/manifest batch record and assets remain'] }, { area: 'Storage object references', diff --git a/docs/agent-friendly-api.md b/docs/agent-friendly-api.md index d0137930..93334149 100644 --- a/docs/agent-friendly-api.md +++ b/docs/agent-friendly-api.md @@ -37,6 +37,13 @@ and an `items` array of `logicalId`, `logicalPath`, `sha256`, `bytes`, and `POST {apiBaseUrl}/asset-batches/{batchId}/complete` and verify the returned manifest SHA-256. +Completed batch item rows are retained for 30 days by default so interrupted +clients can inspect recent uploads. A bounded daily cleanup then removes only +the item rows and returns `itemsRetained: false`; the compact completed batch, +its idempotency key, manifest, and all uploaded assets remain available. +Operators can configure `ASSET_BATCH_ITEM_RETENTION_DAYS`, +`ASSET_BATCH_CLEANUP_INTERVAL_MS`, and `ASSET_BATCH_CLEANUP_LIMIT`. + The executable Node example is `examples/agent-friendly-assets.mjs`. It discovers the API, uploads one file, and verifies the immutable read headers without a hardcoded API prefix. diff --git a/docs/database-scalability-inventory.md b/docs/database-scalability-inventory.md index 6a67d757..76b80de7 100644 --- a/docs/database-scalability-inventory.md +++ b/docs/database-scalability-inventory.md @@ -27,7 +27,7 @@ This file is generated by `npm run database:scalability:update`. Do not hand-edi | Contents | Content | storageId,userId non-unique ownership lookup; manifestStaticStorageId unique; userId,manifestStorageId actor-scoped lookup | has user listing index; has manifest lookup index; has preview lookup indexes; same storageId across different users and legacy same-user duplicate rows remain valid; soft-deleted rows no longer block re-upload; actor-scoped helpers and deterministic shared reads handle duplicate candidates; shared storage reads prefer storageObject with Content fallback, shared manifest reads use id ordering, and new library rows require an actor | `app/modules/database/models/content.ts` | | Storage objects | StorageObject | storageId unique physical identity; identityType,identityId canonical lookup with GeeSome manifest producer; preview storage lookup indexes; updatedAt,id registry scan | model-sync-created A2 on-ramp records one physical storage metadata row per storageId from content writes plus nullable ownerless/federated identity metadata; GeeSome content manifest imports seed that identity on the canonical storage object, database callers can resolve canonical rows by identity pair, public shared metadata reads prefer storageId metadata, local and restored legacy pin state remains canonical, per-account remote pin attempts are recorded separately, delete safety checks both the canonical local pin bit and protected remote-pin states, and restored-backup rehearsal repairs missing/mismatched canonical rows | `app/modules/database/models/storageObject.ts` | | Integration assets | Asset / AssetIdempotencyKey | userId,storageId unique owner identity; userId,sha256,id owner-visible hash preflight; userId,namespace,key unique idempotency identity | asset rows reference existing content/storage bytes; owner-scoped hash lookup avoids disclosing global hash existence | `app/modules/asset/models.ts` | -| Integration asset batches | AssetBatch / AssetBatchItem | userId,idempotencyKey unique batch identity; assetBatchId,logicalId unique item identity; assetBatchId,status,id completion scan | batch preflight is bounded by the submitted item list and completion loads only one owner-scoped batch with its indexed items | `app/modules/asset/models.ts` | +| Integration asset batches | AssetBatch / AssetBatchItem | userId,idempotencyKey unique batch identity; status,updatedAt,id retention scan; assetBatchId,logicalId unique item identity; assetBatchId,status,id completion scan | batch preflight is bounded by the submitted item list and completion loads only one owner-scoped batch with its indexed items; completed item rows are compacted after a configurable 30-day retention window in bounded cleanup runs while the idempotency/manifest batch record and assets remain | `app/modules/asset/models.ts` | | Storage object references | StorageObjectReference | sourceStorageId,targetStorageId,referenceType unique edge; targetStorageId,referenceType reverse lookup; sourceStorageId cleanup lookup | model-sync-created child-reference edge table can retain generated-output parent-to-child storage links separately from user-library content rows | `app/modules/database/models/storageObjectReference.ts` | | Storage space snapshots | StorageSpaceSnapshot | createdAt,id latest-snapshot scan; userId,createdAt,id operator history scan | model-sync cached storage analyzer snapshots store the bounded analyzer payload with list limit, duration, and optional refresher user | `app/modules/database/models/storageSpaceSnapshot.ts` | | Object cache | Object | storageId,resolveProp unique cache key | active data-structure cache keys resolved and unresolved storage paths separately | `app/modules/database/models/object.ts` | diff --git a/test/agentFriendlyApiIntegration.test.ts b/test/agentFriendlyApiIntegration.test.ts index a885b975..43507424 100644 --- a/test/agentFriendlyApiIntegration.test.ts +++ b/test/agentFriendlyApiIntegration.test.ts @@ -216,6 +216,27 @@ describe('agent-friendly asset integration', function () { assert.equal(completed.manifest.url, `${advertisedOrigin}/ipfs/${completed.manifest.storageId}`); const completionReplay = await app.ms.asset.completeBatch(user.id, batch.batchId, apiKey.id); assert.equal(completionReplay.manifest.sha256, completed.manifest.sha256); + assert.equal(completionReplay.itemsRetained, true); + + const cleanup = await app.ms.asset.cleanupCompletedBatches({ + retentionDays: 0, + now: new Date(Date.now() + 1000), + limit: 10 + }); + assert.deepEqual(cleanup, {examined: 1, compacted: 1, itemsDeleted: 2}); + const compactedReplay = await app.ms.asset.createBatch(user.id, {items: [ + {logicalId: 'one', logicalPath: 'release/one.txt', sha256: firstSha, bytes: firstBytes.length, mimeType: 'text/plain'}, + {logicalId: 'two', logicalPath: 'release/two.txt', sha256: secondSha, bytes: secondBytes.length, mimeType: 'text/plain'} + ]}, 'release:batch'); + assert.equal(compactedReplay.status, 'completed'); + assert.equal(compactedReplay.itemsRetained, false); + assert.equal(compactedReplay.items.length, 0); + assert.equal(compactedReplay.manifest.sha256, completed.manifest.sha256); + const compactedCompletionReplay = await app.ms.asset.completeBatch(user.id, batch.batchId, apiKey.id); + assert.equal(compactedCompletionReplay.manifest.sha256, completed.manifest.sha256); + const [retainedAssetRows]: any = await (app.ms.database as any).sequelize.query('SELECT COUNT(*)::int AS count FROM assets'); + assert.equal(retainedAssetRows[0].count, 2); + assert.equal(await (app.ms.database as any).models.Content.count({where: {storageId: completed.manifest.storageId}}), 1); const fiftyItems = Array.from({length: 50}, (_, index) => ({ logicalId: `item-${String(index).padStart(2, '0')}`,