perf: MD outline ~15x + full Inline fast path (~2.3x) - #145
Draft
tiensonqin wants to merge 7 commits into
Draft
Conversation
Outline mode now extracts only node refs, tags, and properties: - Fast direct scanner for #tag / [[page]] / ((block)) - Markdown-specific outline parser with peek dispatch - Skip front-matter probing and position tracking on outline path - Lighter heading title lookahead and outline inline pre-checks Also adds bench/time_parse.exe for Logseq-sized Markdown fixtures. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
|
|
Replace Angstrom block choice for MD outline with a dedicated line scanner (headings, properties, lists, quotes, fences, footnotes) while keeping Outline_inline for refs/tags. Fixes parse_md_outline empty-title edge cases via the new path. ~0.026s vs ~0.30s master on 1.2MB fixture. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
cursor
Bot
changed the title
Tried Parseff (OCaml 5.3 effects) for MD Inline: ~4x slower than Angstrom on short Logseq titles (effects/look_ahead overhead), so not adopted. Instead add a pure-OCaml MD Inline fast path (plain/#tag/[[page]]/ ((block))) with Angstrom fallback. Full parse ~0.19s vs ~0.43s master (~2.3x) on the 1.2MB fixture; outline stays ~15x. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
cursor
Bot
changed the title
Route all Markdown (outline and full) through the line scanner. Full mode uses Inline.parse, Src fences with options/pos_meta, latex env, quote continuation, and heading anchors — ~7–9× vs master on logseq_large while keeping outline_only ~15×. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
The generative MD fixture builder omitted the size field after it was added to Type.Heading, which broke dune builds of that test helper. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Optimize logseq/mldoc Markdown parsing for the Logseq hot path (outline-first, then full).
Performance (1.2MB
examples/logseq_large.md, 3-iter avg, same machine)parse_outline_onlyApproach
parse_outline_only: Skip heavy nested inline once block type is known (existing flag). Outline keeps tags /[[page]]/((block))only (drops emphasis, etc.).Md_outline: Line-oriented MD document parser (no Angstrom blockchoicebacktracking). Used for outline and full Markdown.Inline.parse, fencedSrc(language/options/pos_meta), latex env, quote continuation, heading anchors.try_fast_md_inlinefor plain/tag/page/block refs.Tests
dune runtestgreen (markdown, outline markdown, org, zip, export-md, export-opml).Cloud Agent env
OCaml 5.3 + Node 22 install script proposed (draft build
bld-20260829-0c025de9-…verified). Save the environment in the Cursor dashboard to persist it.Not in this PR
_opam/local switch (untracked)examples/logseq_large.md(bench fixture)Test plan
dune runtestdune exec ./bench/time_parse.exevs master