Skip to content

feat: file activity as timestamped events and add worklog refresh - #55

Draft
Bhekani Khumalo (bhekanik) wants to merge 32 commits into
mainfrom
feat/m2-event-ledger
Draft

feat: file activity as timestamped events and add worklog refresh#55
Bhekani Khumalo (bhekanik) wants to merge 32 commits into
mainfrom
feat/m2-event-ledger

Conversation

@bhekanik

Copy link
Copy Markdown
Collaborator

Activity is now recorded as timestamped events in a cache-side ledger, sources are plugins behind one interface, and worklog refresh picks up what has changed since the last run and rewrites only the weeks it belongs to.

The rule this serves

We keep the state of things as they were when we first fetched them. If there is a change later, we record it on the date the change happened: a new comment goes on the week of the comment's timestamp; a description change goes on the week it happened, or if we cannot know that, the week we spotted it. We can add things we missed in history, but we do not rewrite history.

Before this, a week was fetched as "how these items look right now". Regenerating an August week in September narrated August from September's state — a ticket closed last week read as finished work in a week where it was still in progress.

Ledger layout

$XDG_CACHE_HOME/worklog/ledger/     (~/.cache/worklog/ledger by default)
├── meta.json                  # version, and per source: fetchedAt, windows[], state{}
├── events/<weekId>.json       # append-only, sorted by (at, key)
└── snapshots/<source>.json    # id -> { id, firstSeenAt, payload }, written once per item

Why this shape:

  • Sharded by week, not by source. Reading or writing one week touches one file. refresh writes only the weeks whose event set changed, which is the property the whole command rests on.
  • Snapshots separate from events. A snapshot is written once and never again — it is what a week is entitled to show about an item. Everything after that is an event with its own date. That split is the history rule made structural rather than remembered.
  • A week is its events' timestamps. A comment written in September lands in September's file even though the ticket is from August. No source decides week membership.
  • In the cache dir, not the vault. Refetchable, large, and the vault is synced to iCloud and indexed by Obsidian. Deleting it costs a refetch.
  • JSON, parsed on read. A person can read and therefore edit these files, and a machine can lose power mid-write. Every row goes through zod on the way in; a row that no longer parses is dropped and the rest of the ledger still opens.
  • Idempotent recording. Event identity is the system's own id when there is one (source|id), else source|kind|itemId|at. Re-fetching a week matches instead of duplicating, files are only written when their contents differ, and a second run is byte-identical in both the vault and the cache — asserted in commands/__tests__/refresh.test.ts.

The frozen Source interface

lib/sdk/sources.ts, matching what M3 (#54) built against, extended only with optional fields:

export interface SourceSnapshot { id: string; firstSeenAt: string; payload: unknown; }

export interface SourceEvent {
  source: string; kind: string; itemId: string; at: string; payload: unknown; id?: string;
}

export interface SourceBatch {
  snapshots: SourceSnapshot[]; events: SourceEvent[]; warnings: string[];
}

export interface SourceState {
  get(key: string): string | undefined;
  set(key: string, value: string): void;
}

export interface SourceContext {
  config: WorklogConfig;
  log?: Logger;
  headers?: FetchHeaders;
  identity?: { atlassianAccountId: string; githubUsername: string };
  onWarning?: (message: string) => void;
  state?: SourceState;
}

export type SourceAvailability = { ok: true } | { ok: false; reason: string };
export interface SourceWindow { start: Date; end: Date; }

export interface Source {
  name: string;
  isAvailable(ctx: SourceContext): Promise<SourceAvailability>;
  fetchWindow(window: SourceWindow, ctx: SourceContext): Promise<SourceBatch>;
  fetchSince(since: Date, itemIds: string[], ctx: SourceContext): Promise<SourceBatch>;
}

Two questions and nothing else: what happened in this window (the expensive first look) and what has happened since this moment (the cheap one every later run asks).

worklog refresh

worklog refresh                    # every week since the current team started
worklog refresh --since 2026-01-01
worklog refresh --week 2026-W07
worklog refresh --source jira

Asks each source the delta question, files each answer in the week it happened, and rewrites only the weeks whose event set changed. A run that finds nothing makes no AI call and writes no file. A week that does need rewriting has its existing brag book sent back with only the events this run added, and the instruction is to add to the entry rather than replace it; the vault writers from #47 make re-applying what was already applied a no-op.

The closing table shows what each source contributed per week and how long each source took — Slack's Glean path takes minutes where the HTTP sources take seconds, and a user waiting deserves to know which.

Dating what has no date

A description edit has no timestamp of its own. It is dated at the moment it was found and its payload marked spotted: true; the work log then carries a ## Dating section saying so plainly, so the model is never quietly told a change happened on a day it did not.

Prompt

prompts/weekly-brag-prompt.md no longer tells the model to fetch the current status of a ticket and describe reality at generation time. It now says the week is a closed record, and generation_context states whether it is the current week or a past one.

Also

  • The weekly command generates from the ledger too, so a week written today and the same week amended next month are built the same way from the same events.
  • getWeekEnd returns Sunday at 00:00 — the start of the last day. Window ends are pushed to the last instant of Sunday in one place (weekWindow) rather than in each source.
  • allSources() is one list, read by both commands.

Tests

652 passing, 23 files. New: lib/sdk/__tests__/ledger.test.ts (28), lib/sdk/__tests__/source-adapters.test.ts (30), commands/__tests__/refresh.test.ts (6), plus 8 in markdown.test.ts. anti-slop-check --base origin/main is down to 3 findings, all no-unknown-parameters on the three functions whose job is to be the parsing boundary (keepParsable, writeJsonAtomic, renderable).

@bhekanik
Bhekani Khumalo (bhekanik) force-pushed the feat/m2-event-ledger branch 7 times, most recently from 1f0683d to 3311824 Compare August 27, 2026 16:36
A source answers two questions about one system: what was there when we first
looked at a week, and what has happened since. It never decides which week
anything belongs to. It reports each thing with the timestamp that thing
carries, and the ledger files it by that.

That division is the history rule in code. State is kept as it was when first
fetched; a change discovered later is recorded on the date it happened, so a
past week is amended with events that always belonged to it rather than
rewritten because of something that came after. A source reporting current
state rather than what happened and when would make that impossible.

Frozen here, in its own commit, because the Slack source is being written
against it in parallel.
The ledger stores two kinds of thing and treats them differently. A snapshot is
an item as first seen, written once and never again: it is what the week the
item arrived in is entitled to show. An event carries its own timestamp, and a
week is exactly the events whose timestamps fall inside it. So a comment written
in September belongs to September even though it hangs off an August ticket, and
August's log is not rewritten to mention it.

Recording is idempotent. Re-fetching a week matches every event it already holds
and writes nothing, which is what lets a refresh tell which weeks actually
changed and leave the rest of the file system alone.

It lives under the cache directory, honouring XDG_CACHE_HOME: refetchable data,
large, and the vault is synced to iCloud and indexed by Obsidian.

generateEventMarkdown writes a week from that. It is the successor to
generateMarkdown, which described items as they stand today; this describes what
happened during the week and dates each thing by when it happened, so a coach
reading a past week cannot narrate it from a later state.
A week a source has never been read for gets the expensive first look. After
that the source is only asked what has happened since its watermark, and
whatever comes back is filed by its own timestamp, which is how a change lands
in the week it happened rather than the week we noticed it.

The watermark moves to the time the run started rather than the time it
finished, so an event created while we were fetching is picked up next time
instead of falling in the gap.
The three existing fetchers become sources behind the frozen interface. Each
one now answers the delta question as well as the window question, and dates
what it finds by the timestamp the thing itself carries: a comment by its own
created time, a transition by its changelog entry, a review by when it was
submitted. Where a system offers no timestamp for a change, the source dates it
at the moment it was spotted and says so, so the week can be honest about the
difference.

refresh goes back over the weeks since the current team started, asks every
source what has changed, files each change in the week it happened, and writes
again only the weeks whose event set is not what it was. A week that has not
changed is not regenerated, not re-prompted and not rewritten, so running it
twice costs one round of delta queries and nothing else. Going back further than
the current team takes an explicit --since, because it is roughly one AI call
per week regenerated.

A week that already has a brag book is amended rather than replaced: the entry
goes back to the model with only the material that is new and the instruction to
add to it. The per-week generation moves out of the weekly command's loop so
both commands run the same code.
…to rewrite history

The weekly command now fetches into the ledger and writes the week from it, the
same path refresh takes, so a week written today and the same week amended next
month are built the same way from the same events. Its report counts items and
reviews from the week's own events rather than from a fetch response.

The prompt told the coach to always fetch the latest ticket status, to prefer
current status over what the work log showed, and that the brag book should
reflect reality at generation time. For a past week that is an instruction to
rewrite history: a ticket that closed in September would be written into
August's entry as though it had been known then. It now says a week is a closed
record, that a past week uses state as of its end, that a transition after the
week belongs to the week it happened in, and that live status is fair only for
the current week, which is stated per generation.

refreshWeeks is the decision on its own, with the writing passed in, so the
property the command rests on can be tested: a run that finds nothing calls the
writer zero times and leaves the vault and the cache byte-identical.
…t the end of Sunday

Two shapes had drifted apart. A payload is unknown rather than a JSON object,
because it goes through JSON on the way to disk and comes back a stranger: only
the source that wrote it can read all of it, and everyone else narrows. The
context now promises only the config, with auth, identity, state and the warning
channel offered as capabilities a particular source may want. Slack needs none
of them; these three do, and say so from isAvailable when they are missing,
which is what that call is for.

getWeekEnd returns Sunday at midnight, which is the start of the last day. The
existing queries hide it by sending date-only strings, but a source comparing
timestamps would drop everything that happened on the Sunday. weekWindow ends
the week at the last instant of that day, in one place, so no source has to
know.

The refresh table also reports what each source cost, since one of them asks a
model and answers in minutes rather than seconds.
Rendering reads the keys every source agreed to speak through one zod view
instead of reaching into payloads a field at a time. Adapters name what they
put in a payload. ledgerRoot takes the one env var it depends on.
The sources list lived in two commands and could drift; it lives with the
adapters now. refresh hands the model the events a run added rather than the
whole week, which its own comment already claimed. generateEventMarkdown took
a config it never read, and RecordResult carried the keys of its own map.
Generation goes through aiQueryStructured and toBragBookResult, so a week that
does not validate never reaches the vault. Both documents are written atomically
with the brag book first, in the weekly command and in refresh alike: the entry
is the record that cannot be rebuilt without another generation.

The context every source runs with is built in one place now, so neither command
can hand a source something the other does not.
Watermarks are per week, not per source, so a refresh scoped to one week can no
longer claim to have read on behalf of weeks it never asked about. They move to
the newest thing observed rather than to the clock, so a run that finds nothing
writes nothing. A week whose events changed stays owed a write until it gets one.
A file with a row this version cannot read is reported with its row and left
exactly as it is. A source name that could be a path is refused.
Deltas go looking for work the ledger has never heard of instead of only asking
about ids it already holds, so a ticket, page or pull request created after a
week was first fetched can still be found. Jira reads the changelog in both
directions and pages it; GitHub searches on activity so a merge lands in the week
it happened; Confluence walks the version history and asks for comments. A
comment is the user's only when the user wrote it.
The prompt asked the model not to drop anything; now the run checks. Every
achievement line and coaching heading the entry already holds must come back
word for word, or nothing is written and the failure names the first line that
went missing. Also: --since must be a date that happened and --week a week that
exists, and week ids come from the ISO helper so the end of December belongs to
the next year's first week.
…ead whole

Whichever command generates a week, an existing brag book is added to rather than
replaced: --force means generate it again, not throw away what it says. The
preservation gate compares each section against the same section, so moving an
achievement into prose no longer passes. A week whose cached events cannot all be
read is not written into the vault at all. What is new for the prompt is measured
against what the last successful write covered, not against the state at the
start of the run, so an event discovered days early still reaches the model.
…oday

Window queries select what was alive during the week and file each event by its
own date, so an issue moved on the Monday and touched again three weeks later is
no longer missed by the week it moved in. Deltas look for pull requests the user
reviewed or commented on, not only ones they opened. ETags are scoped to the
watermark they were earned under, so replaying one against an earlier scan cannot
answer 304 for events it never saw. Jira comments, GitHub reviews and Confluence
versions are read to the end rather than one page deep.
The end-to-end fixture answered with a document that dropped the seeded
achievement, which the pipeline now refuses to write. The fixture keeps it and
adds to it, which is what a regeneration is.
…puts who made it

Issue search describes a merged pull request as closed unless you read
pull_request.merged_at, so every merge was recorded as a closure; the fixtures
now mirror the documented search shape so that cannot pass again. Comments on
pull requests are read and filed, and reviewed-by discovery uses the same overlap
bounds as the authored search. The first fetch of a Confluence week walks the
version history instead of collapsing it to the current version, and versions
somebody else wrote are no longer counted as the user's. The containment guard
asks the path module rather than looking for a slash. A week's write marker
reaches disk before the next week is attempted.
One test per trigger: a merge described the way search describes it, a review
left in a week the pull request outlived, a conversation with no formal review, a
page edited three times in one week, a colleague's edit on a page the user once
touched, the containment guard under Windows path rules, and a run that writes
one week and fails on the next.
A source can now say its read was incomplete, and the ledger declines to mark the
window or advance the watermark when it does — a page of history that would not
load, or a walk that hit its own limit, is asked for again rather than written
off. Confluence keeps a page when its history could not be read in full, instead
of reading a half-answer as proof that nobody edited it. GitHub's comment lists
are paged with its own since filter and to exhaustion, and the caps say plainly
that what they hide is the newest. Replies on the user's own pull requests are
read, which is most of what gets said on one.
A reply on the user's own pull request, a delta that asks GitHub to skip what it
already has, a version history whose second page will not load, and a source that
says it did not finish — the last of which must leave its window unmarked and its
reading position where it was.
Unreadable metadata stops a run rather than being worked around: it is the record
of which weeks have already been written up, and without it every week would be
offered again with no way to note that it had. A week a source did not finish
reading is not written up either, in both commands. A 304 on one page no longer
ends a walk, so a page that failed after an earlier one succeeded is asked for
again. Jira histories are read from the offset they actually start at. And a
refresh scoped to one source asks only for that source's credentials.
A truncated meta.json stopping a run before any AI call, a week held back because
a source did not finish, a page asked for again after a 304 on the page before it,
a Jira history read from the offset it really starts at, and a single-source
refresh that does not demand the other source's token.
A week whose event file cannot be rewritten now refuses newly fetched events and
keeps its reading position, so repairing the file brings them back. Metadata that
exists must look like metadata: empty, whitespace and {} are damage, not a fresh
install. Confluence says nothing at all about a page whose history would not load,
rather than recording somebody else's edit as the user's. Weeks left half-read are
remembered on disk, so a run scoped to another source still declines to write them.
Week arithmetic is UTC throughout, and an event key names what its id is an id of.
Empty, whitespace and {} metadata each refusing a run; a review and a comment
sharing a number both kept; a version-2 cache keeping its record of what has been
written up; a week held back by another source's unfinished read; an event at
23:30 on a Sunday filed where its window was looking; a corrupt week's events
refused and fetched again after repair; and a run stopping before it rewrites a
legacy document.
…until its events are filed

The preservation gate looked for the literal string `## Achievements` while the
validator accepted bolding, a trailing colon, closing hashes, another depth and
setext underlining — so a book written any of those ways passed validation with
its achievements invisible to the gate. Both now go through the same remark parse
and the same normalisation, and a coaching heading is compared by that too while
being quoted back as it was written.

A source's ETags are buffered until the events they arrived with are filed. An
ETag is a claim to have already seen something, and a fetch into a week whose file
cannot be rewritten has its events thrown away: keeping the claim had the next
fetch answered 304, so the repair never brought them back.
Pushing to an open pull request produced no event at all, so a branch opened last
week and worked on all of this one had nothing to say about this week. Commits are
now read, filtered to the user by login or committer email, dated by when each was
authored, and summarised in the work log as a span and a count rather than thirty
near-identical lines.

An existing but empty event file is damage rather than an empty week. The
preservation gate reads achievements wherever they are written, protects coaching
headings at every depth, and protects what a document says before its first
heading. And a range of weeks is walked a week at a time, so a range that starts
mid-week no longer steps over the last one.
Achievements written inside the coaching markers, a coaching heading at another
depth, and a line before the first heading; an empty and a whitespace event file;
a pull request opened before the week and pushed to during it, by login and by
committer email; a range walked from a Tuesday to the Monday after next; and a
week of commits rendered as one line.
GitHub carries commit.author and commit.committer as different fields with
different addresses in them. A rebase, an amend from another machine, or a commit
made through the web UI leaves the user as committer while the author line says
something else — and with no linked account either, checking only the author
address missed the commit entirely and the window was marked read.

Matching now takes the account login, the author address or the committer
address. Dating still prefers the author date, which is when the work was
written rather than when it was rewritten onto a branch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant