Skip to content

Add builder-driven full extraction flow with embeddings #20

Description

@laynepenney

Context

The current builder work focuses on coupled prompt + Stage 1 response schema generation. That is useful, but we also need a first-class end-to-end path that runs the full extraction packet, including embeddings, through the builder.

This should cover both model calls:

  1. extraction call: build prompt + response format, call the extraction model, parse Stage 1 JSON
  2. embedding call: build embedding inputs, call the embedding model, attach embedding metadata

Then the builder should finalize the output with the full producer/source context rather than requiring callers to manually stitch these pieces together.

Scope

Add an API that can execute a complete extraction using builder configuration:

  • extraction model/provider/options
  • embedding model/provider/options
  • source metadata such as source_id, source_type, user_id, and kind
  • full produced_by metadata for the extraction model
  • embedding records with model, input, dimensions, space, and computed_at
  • capability/profile selection shared by prompt, schema, finalization, and verification

The builder should still support the current dry-build mode that returns prompt/schema/response format without making network calls.

Design Questions

  • Should this live directly on ExtractionBuilder as execute() / run(), or as a separate runner that consumes builder.build()?
  • Should the library ship provider-agnostic interfaces plus OpenAI helpers, or OpenAI-only helpers first?
  • Should embeddings be configured as a single embedding of the source text by default, with optional summary/entities embeddings later?
  • How should we represent auxiliary producer metadata for the embedding model versus the final extraction-producing model?

Acceptance Criteria

  • TypeScript and Python both expose a builder-driven E2E API.
  • One fixture demonstrates a real extraction API call plus a real embedding API call, not fabricated payloads.
  • Finalized extraction validates against SynaptExtraction.
  • The output includes structured produced_by for the extraction model.
  • The output includes embeddings[] records with model URI, dimensions, input, vector, and computed timestamp.
  • Tests cover prompt/schema parity, finalization, validation, and the embedding metadata path without requiring live API calls by default.
  • Docs show both dry-build and execute-with-embeddings flows.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions