feat: file activity as timestamped events and add worklog refresh - #55
Draft
Bhekani Khumalo (bhekanik) wants to merge 32 commits into
Draft
feat: file activity as timestamped events and add worklog refresh#55Bhekani Khumalo (bhekanik) wants to merge 32 commits into
Bhekani Khumalo (bhekanik) wants to merge 32 commits into
Conversation
Bhekani Khumalo (bhekanik)
force-pushed
the
feat/m2-event-ledger
branch
7 times, most recently
from
August 27, 2026 16:36
1f0683d to
3311824
Compare
A source answers two questions about one system: what was there when we first looked at a week, and what has happened since. It never decides which week anything belongs to. It reports each thing with the timestamp that thing carries, and the ledger files it by that. That division is the history rule in code. State is kept as it was when first fetched; a change discovered later is recorded on the date it happened, so a past week is amended with events that always belonged to it rather than rewritten because of something that came after. A source reporting current state rather than what happened and when would make that impossible. Frozen here, in its own commit, because the Slack source is being written against it in parallel.
The ledger stores two kinds of thing and treats them differently. A snapshot is an item as first seen, written once and never again: it is what the week the item arrived in is entitled to show. An event carries its own timestamp, and a week is exactly the events whose timestamps fall inside it. So a comment written in September belongs to September even though it hangs off an August ticket, and August's log is not rewritten to mention it. Recording is idempotent. Re-fetching a week matches every event it already holds and writes nothing, which is what lets a refresh tell which weeks actually changed and leave the rest of the file system alone. It lives under the cache directory, honouring XDG_CACHE_HOME: refetchable data, large, and the vault is synced to iCloud and indexed by Obsidian. generateEventMarkdown writes a week from that. It is the successor to generateMarkdown, which described items as they stand today; this describes what happened during the week and dates each thing by when it happened, so a coach reading a past week cannot narrate it from a later state.
A week a source has never been read for gets the expensive first look. After that the source is only asked what has happened since its watermark, and whatever comes back is filed by its own timestamp, which is how a change lands in the week it happened rather than the week we noticed it. The watermark moves to the time the run started rather than the time it finished, so an event created while we were fetching is picked up next time instead of falling in the gap.
The three existing fetchers become sources behind the frozen interface. Each one now answers the delta question as well as the window question, and dates what it finds by the timestamp the thing itself carries: a comment by its own created time, a transition by its changelog entry, a review by when it was submitted. Where a system offers no timestamp for a change, the source dates it at the moment it was spotted and says so, so the week can be honest about the difference. refresh goes back over the weeks since the current team started, asks every source what has changed, files each change in the week it happened, and writes again only the weeks whose event set is not what it was. A week that has not changed is not regenerated, not re-prompted and not rewritten, so running it twice costs one round of delta queries and nothing else. Going back further than the current team takes an explicit --since, because it is roughly one AI call per week regenerated. A week that already has a brag book is amended rather than replaced: the entry goes back to the model with only the material that is new and the instruction to add to it. The per-week generation moves out of the weekly command's loop so both commands run the same code.
…to rewrite history The weekly command now fetches into the ledger and writes the week from it, the same path refresh takes, so a week written today and the same week amended next month are built the same way from the same events. Its report counts items and reviews from the week's own events rather than from a fetch response. The prompt told the coach to always fetch the latest ticket status, to prefer current status over what the work log showed, and that the brag book should reflect reality at generation time. For a past week that is an instruction to rewrite history: a ticket that closed in September would be written into August's entry as though it had been known then. It now says a week is a closed record, that a past week uses state as of its end, that a transition after the week belongs to the week it happened in, and that live status is fair only for the current week, which is stated per generation. refreshWeeks is the decision on its own, with the writing passed in, so the property the command rests on can be tested: a run that finds nothing calls the writer zero times and leaves the vault and the cache byte-identical.
…t the end of Sunday Two shapes had drifted apart. A payload is unknown rather than a JSON object, because it goes through JSON on the way to disk and comes back a stranger: only the source that wrote it can read all of it, and everyone else narrows. The context now promises only the config, with auth, identity, state and the warning channel offered as capabilities a particular source may want. Slack needs none of them; these three do, and say so from isAvailable when they are missing, which is what that call is for. getWeekEnd returns Sunday at midnight, which is the start of the last day. The existing queries hide it by sending date-only strings, but a source comparing timestamps would drop everything that happened on the Sunday. weekWindow ends the week at the last instant of that day, in one place, so no source has to know. The refresh table also reports what each source cost, since one of them asks a model and answers in minutes rather than seconds.
Rendering reads the keys every source agreed to speak through one zod view instead of reaching into payloads a field at a time. Adapters name what they put in a payload. ledgerRoot takes the one env var it depends on.
The sources list lived in two commands and could drift; it lives with the adapters now. refresh hands the model the events a run added rather than the whole week, which its own comment already claimed. generateEventMarkdown took a config it never read, and RecordResult carried the keys of its own map.
Generation goes through aiQueryStructured and toBragBookResult, so a week that does not validate never reaches the vault. Both documents are written atomically with the brag book first, in the weekly command and in refresh alike: the entry is the record that cannot be rebuilt without another generation. The context every source runs with is built in one place now, so neither command can hand a source something the other does not.
Watermarks are per week, not per source, so a refresh scoped to one week can no longer claim to have read on behalf of weeks it never asked about. They move to the newest thing observed rather than to the clock, so a run that finds nothing writes nothing. A week whose events changed stays owed a write until it gets one. A file with a row this version cannot read is reported with its row and left exactly as it is. A source name that could be a path is refused.
Deltas go looking for work the ledger has never heard of instead of only asking about ids it already holds, so a ticket, page or pull request created after a week was first fetched can still be found. Jira reads the changelog in both directions and pages it; GitHub searches on activity so a merge lands in the week it happened; Confluence walks the version history and asks for comments. A comment is the user's only when the user wrote it.
The prompt asked the model not to drop anything; now the run checks. Every achievement line and coaching heading the entry already holds must come back word for word, or nothing is written and the failure names the first line that went missing. Also: --since must be a date that happened and --week a week that exists, and week ids come from the ISO helper so the end of December belongs to the next year's first week.
…ce must go looking for
…ead whole Whichever command generates a week, an existing brag book is added to rather than replaced: --force means generate it again, not throw away what it says. The preservation gate compares each section against the same section, so moving an achievement into prose no longer passes. A week whose cached events cannot all be read is not written into the vault at all. What is new for the prompt is measured against what the last successful write covered, not against the state at the start of the run, so an event discovered days early still reaches the model.
…oday Window queries select what was alive during the week and file each event by its own date, so an issue moved on the Monday and touched again three weeks later is no longer missed by the week it moved in. Deltas look for pull requests the user reviewed or commented on, not only ones they opened. ETags are scoped to the watermark they were earned under, so replaying one against an earlier scan cannot answer 304 for events it never saw. Jira comments, GitHub reviews and Confluence versions are read to the end rather than one page deep.
The end-to-end fixture answered with a document that dropped the seeded achievement, which the pipeline now refuses to write. The fixture keeps it and adds to it, which is what a regeneration is.
…puts who made it Issue search describes a merged pull request as closed unless you read pull_request.merged_at, so every merge was recorded as a closure; the fixtures now mirror the documented search shape so that cannot pass again. Comments on pull requests are read and filed, and reviewed-by discovery uses the same overlap bounds as the authored search. The first fetch of a Confluence week walks the version history instead of collapsing it to the current version, and versions somebody else wrote are no longer counted as the user's. The containment guard asks the path module rather than looking for a slash. A week's write marker reaches disk before the next week is attempted.
One test per trigger: a merge described the way search describes it, a review left in a week the pull request outlived, a conversation with no formal review, a page edited three times in one week, a colleague's edit on a page the user once touched, the containment guard under Windows path rules, and a run that writes one week and fails on the next.
A source can now say its read was incomplete, and the ledger declines to mark the window or advance the watermark when it does — a page of history that would not load, or a walk that hit its own limit, is asked for again rather than written off. Confluence keeps a page when its history could not be read in full, instead of reading a half-answer as proof that nobody edited it. GitHub's comment lists are paged with its own since filter and to exhaustion, and the caps say plainly that what they hide is the newest. Replies on the user's own pull requests are read, which is most of what gets said on one.
A reply on the user's own pull request, a delta that asks GitHub to skip what it already has, a version history whose second page will not load, and a source that says it did not finish — the last of which must leave its window unmarked and its reading position where it was.
Unreadable metadata stops a run rather than being worked around: it is the record of which weeks have already been written up, and without it every week would be offered again with no way to note that it had. A week a source did not finish reading is not written up either, in both commands. A 304 on one page no longer ends a walk, so a page that failed after an earlier one succeeded is asked for again. Jira histories are read from the offset they actually start at. And a refresh scoped to one source asks only for that source's credentials.
A truncated meta.json stopping a run before any AI call, a week held back because a source did not finish, a page asked for again after a 304 on the page before it, a Jira history read from the offset it really starts at, and a single-source refresh that does not demand the other source's token.
A week whose event file cannot be rewritten now refuses newly fetched events and
keeps its reading position, so repairing the file brings them back. Metadata that
exists must look like metadata: empty, whitespace and {} are damage, not a fresh
install. Confluence says nothing at all about a page whose history would not load,
rather than recording somebody else's edit as the user's. Weeks left half-read are
remembered on disk, so a run scoped to another source still declines to write them.
Week arithmetic is UTC throughout, and an event key names what its id is an id of.
Empty, whitespace and {} metadata each refusing a run; a review and a comment
sharing a number both kept; a version-2 cache keeping its record of what has been
written up; a week held back by another source's unfinished read; an event at
23:30 on a Sunday filed where its window was looking; a corrupt week's events
refused and fetched again after repair; and a run stopping before it rewrites a
legacy document.
Bhekani Khumalo (bhekanik)
force-pushed
the
feat/m2-event-ledger
branch
from
August 27, 2026 17:04
3311824 to
a5da637
Compare
…until its events are filed The preservation gate looked for the literal string `## Achievements` while the validator accepted bolding, a trailing colon, closing hashes, another depth and setext underlining — so a book written any of those ways passed validation with its achievements invisible to the gate. Both now go through the same remark parse and the same normalisation, and a coaching heading is compared by that too while being quoted back as it was written. A source's ETags are buffered until the events they arrived with are filed. An ETag is a claim to have already seen something, and a fetch into a week whose file cannot be rewritten has its events thrown away: keeping the claim had the next fetch answered 304, so the repair never brought them back.
Pushing to an open pull request produced no event at all, so a branch opened last week and worked on all of this one had nothing to say about this week. Commits are now read, filtered to the user by login or committer email, dated by when each was authored, and summarised in the work log as a span and a count rather than thirty near-identical lines. An existing but empty event file is damage rather than an empty week. The preservation gate reads achievements wherever they are written, protects coaching headings at every depth, and protects what a document says before its first heading. And a range of weeks is walked a week at a time, so a range that starts mid-week no longer steps over the last one.
Achievements written inside the coaching markers, a coaching heading at another depth, and a line before the first heading; an empty and a whitespace event file; a pull request opened before the week and pushed to during it, by login and by committer email; a range walked from a Tuesday to the Monday after next; and a week of commits rendered as one line.
GitHub carries commit.author and commit.committer as different fields with different addresses in them. A rebase, an amend from another machine, or a commit made through the web UI leaves the user as committer while the author line says something else — and with no linked account either, checking only the author address missed the commit entirely and the window was marked read. Matching now takes the account login, the author address or the committer address. Dating still prefers the author date, which is when the work was written rather than when it was rewritten onto a branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Activity is now recorded as timestamped events in a cache-side ledger, sources are plugins behind one interface, and
worklog refreshpicks up what has changed since the last run and rewrites only the weeks it belongs to.The rule this serves
Before this, a week was fetched as "how these items look right now". Regenerating an August week in September narrated August from September's state — a ticket closed last week read as finished work in a week where it was still in progress.
Ledger layout
Why this shape:
refreshwrites only the weeks whose event set changed, which is the property the whole command rests on.source|id), elsesource|kind|itemId|at. Re-fetching a week matches instead of duplicating, files are only written when their contents differ, and a second run is byte-identical in both the vault and the cache — asserted incommands/__tests__/refresh.test.ts.The frozen
Sourceinterfacelib/sdk/sources.ts, matching what M3 (#54) built against, extended only with optional fields:Two questions and nothing else: what happened in this window (the expensive first look) and what has happened since this moment (the cheap one every later run asks).
worklog refreshAsks each source the delta question, files each answer in the week it happened, and rewrites only the weeks whose event set changed. A run that finds nothing makes no AI call and writes no file. A week that does need rewriting has its existing brag book sent back with only the events this run added, and the instruction is to add to the entry rather than replace it; the vault writers from #47 make re-applying what was already applied a no-op.
The closing table shows what each source contributed per week and how long each source took — Slack's Glean path takes minutes where the HTTP sources take seconds, and a user waiting deserves to know which.
Dating what has no date
A description edit has no timestamp of its own. It is dated at the moment it was found and its payload marked
spotted: true; the work log then carries a## Datingsection saying so plainly, so the model is never quietly told a change happened on a day it did not.Prompt
prompts/weekly-brag-prompt.mdno longer tells the model to fetch the current status of a ticket and describe reality at generation time. It now says the week is a closed record, andgeneration_contextstates whether it is the current week or a past one.Also
getWeekEndreturns Sunday at 00:00 — the start of the last day. Window ends are pushed to the last instant of Sunday in one place (weekWindow) rather than in each source.allSources()is one list, read by both commands.Tests
652 passing, 23 files. New:
lib/sdk/__tests__/ledger.test.ts(28),lib/sdk/__tests__/source-adapters.test.ts(30),commands/__tests__/refresh.test.ts(6), plus 8 inmarkdown.test.ts.anti-slop-check --base origin/mainis down to 3 findings, allno-unknown-parameterson the three functions whose job is to be the parsing boundary (keepParsable,writeJsonAtomic,renderable).