Docengine is a format-neutral Go engine for reading, editing, recovering, and organizing large local UTF-8 documents. It provides immutable snapshots, transactional edits, crash recovery, atomic persistence, coordinate mapping, virtualization, search, revision-bound ranges, and multi-source composition without requiring the entire document to be loaded into memory.
The engine understands bytes, UTF-8 boundaries, revisions, ranges, and abstract measures. It deliberately does not understand Markdown, JSON, programming languages, or any other document format. Parsing, rendering, layout, file-type policy, and product workflows belong to the host application.
Docengine is pre-1.0. It is suitable for evaluation and integration where API changes can be accommodated, but its public API and persistent formats do not yet carry a 1.0 compatibility guarantee.
Docengine currently targets Go 1.26:
go get github.com/moresleep512/docengine/docThe module path is:
github.com/moresleep512/docengine
The doc package is
the recommended application-facing API. It separates read-only and read-write
permissions through return types.
package main
import (
"fmt"
"log"
"github.com/moresleep512/docengine/doc"
)
func main() {
document, err := doc.New("notes.txt")
if err != nil {
log.Fatal(err)
}
defer document.Close()
if _, err := document.Write("Hello, 世界\n"); err != nil {
log.Fatal(err)
}
if _, err := document.Insert(7, "local "); err != nil {
log.Fatal(err)
}
if _, err := document.Save(); err != nil {
log.Fatal(err)
}
result, err := document.Search("世界")
if err != nil {
log.Fatal(err)
}
fmt.Printf("revision=%d matches=%d\n",
result.Revision, len(result.Matches))
}Use doc.Read for an existing file without mutation methods, doc.Open for an
existing writable file, and doc.New for exclusive creation. Context-aware
variants are available for open, reads, mutations, search, persistence,
maintenance, and cleanup.
All offsets are byte offsets and all ranges are half-open [start, end).
Mutations and structural ranges that must preserve valid UTF-8 require rune
boundaries; byte-oriented reads may address any in-bounds byte range. Use
Coordinates when a host needs line, rune, and column mapping.
-
A disk-backed Piece Tree with immutable, revision-bound snapshots.
-
Ordered edit batches that commit atomically and form one undo/redo unit.
-
An append-only recovery v2 journal with atomic batch replay and damaged-tail repair.
-
Full-file UTF-8 validation, content identity, external-change detection, and symbolic-link target pinning.
-
Streaming, conflict-checked atomic save with POSIX directory synchronization and Windows write-through replacement.
-
Explicit dirty, durability-uncertain, recovery-uncertain, and persistence fault states.
-
Bounded change history, affinity-aware anchors, ranges, and incremental byte/line/rune coordinate indexes.
-
Immutable snapshots that remain readable across edits and parent closure until their leases are released.
-
Bounded resumable event streams and lifecycle, recovery, history, and persistence diagnostics.
-
UTF-8-safe logical pages, opaque fragments, host-defined fixed-point measures, overscan, caching, and revision/generation checks.
-
Streaming literal and RE2 search with optional pure-Go SQLite/FTS5 persistent candidate indexes. Every indexed candidate is verified against its source snapshot.
-
Immutable revision-bound interval sets for host annotations and decorations.
-
Immutable multi-source compositions with bidirectional source mapping.
Every subsystem has explicit resource limits. Zero-valued options select bounded defaults rather than unbounded operation.
flowchart TB
Host["Host application<br/>editor · CLI · service · format adapter"]
Native["Native host · language binding"]
CAPI["C ABI"]
Doc["doc<br/>permission-separated public API"]
Session["document.Session<br/>revision · transaction · lifecycle"]
Store["store<br/>Piece Tree · Snapshot"]
Recovery["recovery<br/>v2 journal"]
Save["save<br/>atomic persistence"]
Coordinate["coordinate<br/>index · ChangeMap"]
Virtual["virtual<br/>Page · Fragment"]
Search["search<br/>scan · persistent index"]
Interval["interval<br/>immutable ranges"]
Composition["composition<br/>multi-source snapshot"]
OS["files · io.ReaderAt · operating system"]
Host --> Doc
Native --> CAPI
CAPI --> Doc
Doc --> Session
Doc --> Interval
Doc --> Composition
Session --> Store
Session --> Recovery
Session --> Save
Session --> Coordinate
Session --> Virtual
Session --> Search
Store --> OS
Recovery --> OS
Save --> OS
Search --> OS
Most applications should depend only on doc. The lower-level
document.Session API is available for hosts that need direct control over
storage directories, journal policy, snapshot leases, checkpoints, or
subsystem lifetimes. Lower packages remain independently usable over immutable
io.ReaderAt-style sources.
Native applications and foreign-language bindings can use the versioned, coarse-grained C API. It exposes the main document lifecycle without passing Go pointers or Go-owned objects across the ABI boundary.
See MODULES.md for package responsibilities and dependency constraints.
Opening a document validates the complete file as UTF-8, computes its content identity, records its requested and resolved paths, and replays a matching recovery journal. A symbolic link is resolved once when opened; later saves continue to target that resolved file even if the link is redirected.
An edit becomes visible only after its complete recovery batch is appended. An incomplete or corrupt journal tail is discarded without exposing a partial batch. A mismatched or ambiguous journal is isolated and causes open to fail instead of applying edits to an uncertain base.
Save streams a selected immutable revision to a temporary file, checks for
external changes, atomically replaces the destination, and establishes a new
recovery generation. A replacement may have committed even when a later
durability or rebind step reports an error; State preserves that distinction.
If the session cannot safely resume mutation after commit, it becomes
permanently read-only until reopened.
Reads and immutable snapshots may run concurrently. Mutations are serialized and revision checked. Save may run concurrently with later edits: it commits the selected revision while newer edits remain dirty and recoverable.
Snapshots, coordinate indexes, pagers, search indexes, event subscriptions, and
compositions own bounded leases or resources. Close each value that exposes a
Close method. Session closure stops accepting new work, cancels background
work, waits for accepted operations, and then releases owned resources.
Read-only doc.Reader is an API permission boundary, not a promise that opening
the document performs no recovery or runtime-directory writes.
-
API.md — high-level API guide and lower-level Session guidance.
-
C API — native ABI, ownership rules, build instructions, and C examples.
-
MODULES.md — package architecture and invariants.
-
TODO.md — known limitations and work required before 1.0.
-
Go Reference — generated package and symbol documentation.
English documentation is canonical. README_zh.md provides the Chinese project overview.
CI runs on Windows and Ubuntu and requires:
-
module verification, formatting, vet, and the complete test suite;
-
100% statement coverage for all eleven tested Go packages, including the C bridge;
-
race-enabled tests on Linux with randomized ordering and repeated runs;
-
state-machine and property fuzz smoke tests for the facade, store, recovery, Session, coordinates, virtualization, search, intervals, and composition;
-
shared-library builds plus strict C and C++ consumer compilation;
-
additional Windows stress coverage for journal durability event ordering.
The tests include fault injection around open, recovery, journaling, save, directory synchronization, atomic replacement, snapshot generations, compaction, persistent search, close barriers, and resource budgets.
The canonical repository is GitHub. A mirror is maintained on Codeberg.
Docengine is available under the MIT License.