Loom

Loom turns FHIR-shaped data in ArangoDB into authorized, queryable flat dataframes in ClickHouse.

ArangoDB is the canonical graph store: Loom loads NDJSON, extracts FHIR resources and relationships, profiles the populated graph, and compiles typed dataframe recipes into scoped AQL. ClickHouse is the publication store: Loom streams resolved recipe outputs into versioned physical tables, records their schema and visibility in an Arango-backed catalog, and exposes them through a stable reader API.

Loom is deliberately not an arbitrary AQL gateway, ClickHouse SQL proxy, or a replacement FHIR model. It is a recipe-driven graph-to-flat data service with explicit authorization and publication boundaries.

flowchart LR
    NDJSON["FHIR NDJSON"] --> Ingest["Ingest and catalog"]
    Ingest --> Arango["ArangoDB\nresources, fhir_edge, field catalog"]
    Recipe["Versioned dataframe recipe"] --> Compile["Resolve, validate, compile"]
    Arango --> Compile
    Compile --> AQL["Scoped AQL stream"]
    AQL --> Publish["Atomic publication catalog commit"]
    Publish --> ClickHouse["Versioned flat tables"]
    Publish --> Catalog["Arango publication catalog"]
    Catalog["Arango publication catalog"] --> Flat["POST /graphql/flat"]
    ClickHouse --> Flat
    Arango --> Graph["POST /graphql/graph"]
    Compile --> Graph

Loading

What Loom owns

• FHIR NDJSON ingestion into ArangoDB resource collections and fhir_edge.
• Immutable dataset generations and active-generation selection.
• Populated-field, reference, traversal, and pivot discovery.
• Strict, versioned dataframe recipes and compiler-backed AQL execution.
• Scoped publication of recipe outputs into ClickHouse.
• A durable Arango catalog that maps logical dataframe names to current READY ClickHouse outputs.
• Multi-project, auth_resource_path-scoped flat-data reads.

Loom does not expose arbitrary ClickHouse tables, arbitrary SQL, or an Elasticsearch/Guppy fallback. A logical dataType is a catalog alias, never a browser-supplied physical table name.

Runtime surfaces

Surface	Purpose
arango-fhir-proto	Operator CLI for loading data, loading immutable generations, catalog discovery, and local dataframe materialization.
arango-fhir-server	HTTP server for graph compilation/control and flat dataframe reads.
POST /graphql/graph	Arango graph and recipe control plane: builder introspection, dataframe/recipe validation, preview, execution, and publication.
POST /graphql/flat	ClickHouse reader: discover published datasets, fetch rows with filters/keyset cursors, and aggregate registered outputs.
POST /api/v1/imports	Legacy one-resource import compatibility path; disabled in dataset-generation mode.

GET /graphql/graph serves GraphQL Playground for the graph API. GET /apollo opens Apollo Sandbox pointed at /graphql/graph. There is intentionally no /graphql compatibility route.

Data lifecycle

1. Load FHIR NDJSON into ArangoDB. Loom creates resource collections, fhir_edge, and fhir_field_catalog.
2. For production snapshots, load an immutable dataset generation and let the active manifest select it for a project.
3. Resolve a registered recipe against the populated graph and caller scope.
4. Compile the resolved recipe to parameterized AQL and stream rows from ArangoDB.
5. Publish output streams to new ClickHouse physical tables. Loom injects the reserved auth_resource_path column, records schema/provenance in Arango, and atomically advances the logical publication pointer only once every output is READY.
6. Query the logical dataframe through /graphql/flat. The reader resolves authorized current outputs from the catalog; it never accepts a physical ClickHouse table name.

The configured ClickHouse database is created by the server during startup if necessary. Publication creates its tables itself. No DDL or alias-management HTTP endpoint is required. The ClickHouse account needs database creation plus CREATE, INSERT, SELECT, and DROP privileges.

Local development

The local Compose stack starts both ArangoDB and ClickHouse:

rtk docker compose -f experimental/docker-compose.yml up -d
make generate-fhir
make generate-graphql
make build

Load the bundled example data into a mutable local namespace:

./bin/arango-fhir-proto load \
  --url http://127.0.0.1:8529 \
  --database fhir_proto \
  --meta-dir META \
  --project ARANGODB_PROTO \
  --auth-resource-path EllrottLab-GDC_Data

For a production-shaped immutable snapshot, use load-generation instead. It requires an opaque generation identifier and deliberately forbids truncation:

./bin/arango-fhir-proto load-generation \
  --url http://127.0.0.1:8529 \
  --database fhir_proto \
  --meta-dir META \
  --project ARANGODB_PROTO \
  --generation local-meta-1 \
  --auth-resource-path EllrottLab-GDC_Data

Start Loom locally with an unrestricted development principal:

./bin/arango-fhir-server \
  --listen :8080 \
  --no-auth \
  --url http://127.0.0.1:8529 \
  --database fhir_proto \
  --clickhouse-url clickhouse://127.0.0.1:9000 \
  --clickhouse-database loom

Add --dataset-generations when reading the active immutable generation. That mode disables the legacy one-file import endpoint.

Useful local URLs:

• Graph Playground
• Apollo Sandbox
• Flat reader: http://127.0.0.1:8080/graphql/flat
• Health check

Run the checked-in graph dataframe example after the server is up:

make dataframe-demo

See the Quickstart for the complete example request and response flow.

Graph and flat GraphQL contracts

Graph: compile, inspect, and publish

/graphql/graph is where a caller asks Loom about the Arango graph or drives recipe work. Its schema includes builder introspection, direct dataframe execution, and recipe operations such as validation, explanation, preview, execution, and materializeDataframeRecipeBundle.

Publication is intentionally a recipe-level operation: Loom must read the authorized Arango graph, resolve the recipe against the selected generation, and then write its output to ClickHouse. It is not a raw “insert rows” API.

Flat: discover and read published dataframes

/graphql/flat is the stable ClickHouse reader. The schema is intentionally small and data-driven:

query ExplorerRows($input: DataframeRowsInput!) {
  dataframeRows(input: $input) {
    materialization { name revision rowCount columns { name logicalType } }
    columns
    rows
    totalCount
    pageInfo { hasNextPage endCursor }
  }
}

{
  "input": {
    "dataType": "cases",
    "first": 100,
    "sort": { "column": "case_id" }
  }
}

The public reader input contains a logical dataType, not a project, generation, materialization ID, or physical table. Loom derives the authorized project set from the principal, finds each project’s current READY publication, and federates compatible outputs. Rows remain permissive JSON; columns and capabilities are discovered at runtime so a new publication does not require a GraphQL regeneration or server restart.

Authorization and tenancy

In production, Loom uses either basic authentication or Calypr/Fence-backed authorization. Calypr mode resolves the caller’s allowed auth_resource_path values and applies them to graph reads and published flat reads. Publication injects auth_resource_path into every ClickHouse output; the flat reader treats it as an authorization predicate, never as a client-controlled filter.

Set --no-auth only for local development. It creates an explicit unrestricted principal and must not be used in a deployment.

The server can be configured with YAML:

server:
  listen: ":8080"
  backend: arango
  url: http://arangodb:8529
  database: fhir_proto
  dataset_generations: true
  clickhouse:
    url: clickhouse://clickhouse:9000
    database: loom

auth:
  mode: calypr

Basic mode reads LOOM_AUTH_BASIC_USERNAME and LOOM_AUTH_BASIC_PASSWORD if they are not supplied in the config file.

Repository guide

Path	Responsibility
cmd/	Operator CLI, server executable, and developer tools.
internal/ingest	NDJSON loading, validation, graph extraction, and ingest lifecycle.
internal/dataset	Immutable generation and active-manifest contracts.
internal/catalog	Evidence of populated fields, references, and authorization paths.
internal/dataframe/compiler	Typed plan IR, lowering, optimization, and AQL rendering.
internal/dataframe/recipe	Recipe contract, validation, schema resolution, execution, and control services.
internal/dataframe/publication	Backend-neutral bounded streaming publication contract.
internal/dataframe/materialization	ClickHouse table lifecycle, durable publication pointers, and federated reads.
internal/store/arango	ArangoDB boundary.
internal/store/clickhouse	Typed ClickHouse driver boundary and DDL/DML.
graphqlapi	Graph control-plane schema and resolvers.
graphqlapi/clickhouse	Dedicated flat-reader GraphQL schema and resolvers.

Build, generation, and tests

make build                 # server and CLI binaries
make generate-fhir         # generated FHIR structs and schema metadata
make generate-graphql      # gqlgen bindings
make graphql-check         # GraphQL and dataframe checks
make test                  # full Go test suite
make conformance           # compiler conformance corpus

The generated FHIR metadata and GraphQL bindings are checked in. Regenerate them when their inputs change; do not hand-edit generated files.

For a direct build and full verification:

go build .
go test ./...

Further reading

• Quickstart
• Developer architecture
• ClickHouse reader contract and execution plan
• Explorer/Loom parity plan
• Experimental local stack

The Helm chart lives in the separate gen3-helm repository under helm/loom.