Merge dev to master - #11
Conversation
…gration suites The two *.integration.test.ts suites need real thesis PDFs under .claude/pdf_examples/, which are gitignored (someone's actual skripsi). On a fresh clone `bun test` fails noisily on those. Add a `test:unit` script that excludes `**/*.integration.test.ts` and point both CONTRIBUTING and README at it as the default contributor command. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /results/$jobId loader called getFullResults, which threw a plain
Error('Job not found'). That error bubbled through MatchInnerImpl into
the root errorComponent, so missing reports rendered as a generic
"Something went sideways" page and the dev console filled with React
error-boundary warnings.
Catch that specific error in the loader and rethrow as notFound() so
TanStack Router treats it as a routing signal instead of a render-time
error. Add a root-level notFoundComponent (NotFoundPage) so all routes
that already throw notFound() — settings, history, and now results —
get a branded 404 instead of TanStack's default bare fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…t root The previous fix put errorComponent/notFoundComponent on the root route, which only covers the root segment itself — leaf-route catch boundaries without their own errorComponent still fell back to TanStack's bare "Something went wrong! Show Error" UI, causing the visible flash before notFound() propagated. Move the branded pages to defaultErrorComponent / defaultNotFoundComponent on the router so every leaf inherits them. Drop them from __root.tsx now that the router defaults cover the same surface. Also throw notFound() directly from getFullResults instead of throwing a plain Error and converting it in the route loader. @tanstack/react-start serializes notFound() natively across the server function boundary, so the loader sees the routing signal directly with no transient "error" status — eliminating the flash entirely. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-quota 302s (#2) * test(perf): add evaluation benchmark gated on PERF=1 Reads .claude/pdf_examples/thesis_example.pdf, runs the full extract + analyze pipeline once for warm-up then 20 timed iterations, and asserts p50 ≤ 15s and p99 ≤ 30s via soft expectations so both percentiles surface in one run. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(kbbi): share dictionary load and warm negative cache rows too Both lookup.ts and suggester.ts previously issued their own SELECT against the 221k-row dictionary table on every job. Consolidate into a single warm pass in dict-store.ts that serves both consumers, makes warming idempotent across runs, and warms the full dictionary_cache including rows where source is null so prior negative lookups stay hot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(kbbi): replace linear suggester scan with BK-tree Builds one BK-tree per first-letter bucket on first warm. Query is pruned by the triangle inequality (only children with edit distance in [d - maxDist, d + maxDist] are visited), turning the per-token suggest cost from O(bucket size) Levenshteins into a fraction of that for tight max distances (1 or 2). loadDictBuckets is kept as a warm-up hook so callers that pre-load the suggester continue to work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(kbbi): collapse concurrent isKnownWord calls via single-flight map Two parallel passes in the analyzer (detectPdfSplitFragments and the main candidate loop) can probe the same token at the same time and each would walk the full affix + cache + external pipeline. Keying the in-flight promise by lowercased token serializes those into one resolution and shares the result, with the entry cleared on settle so sequential calls still re-evaluate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(kbbi): batch dictionary_cache writes and memoize affix stripping writeCache now enqueues into a per-process Map keyed by word and the orchestrator flushes the batch with one INSERT ... ON CONFLICT DO UPDATE after KBBI + EYD finish. Removes N single-row writes from the hot path. stripAffixes is memoized so the affix queue isn't rebuilt for tokens that appear across multiple pages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(kbbi): tokenize each page once and share between fragment + main pass detectPdfSplitFragments used to re-walk content.matchAll(TOKEN_RE) and the main candidate loop did the same matchAll a second time. Extract tokenizePage and feed the resulting array into both probeJoinCandidates and the filter loop so the regex runs once per page. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(kbbi): polite scraping — shared headers, per-host throttle, retry-after All four scrape sources now send a Chrome 120 UA, Accept, Accept-Language id-ID, Sec-Fetch hints, and a self-Referer so requests look like a real browser rather than a Node fetch. A per-host queue (throttleHost) keeps outbound calls to one site separated by ~400–600 ms with jitter, so the analyzer's 8-way concurrency no longer fans out as a burst against a single domain. 429/503 responses parse Retry-After and pause that host for the requested duration; subsequent lookups against the paused host fail open without poisoning dictionary_cache. Starting source is rotated by a deterministic hash of the keyword so the four hosts share the load instead of stacking on kbbi.kemendikdasmen.go.id. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(perf): exercise external KBBI integration alongside the analyzer bench Adds two more PERF=1 tests inside evaluation.bench.test.ts: - per-source cari() smoke test against a known KBBI entry ('buku'), one source at a time, asserting at least one source responds and surfacing individual source health via soft assertions - isKnownWord drill that purges dictionary_cache for a hand-picked list of words unlikely to live in the local dump and asserts at least one of them genuinely falls through to the external scrape path, with per-word known/databaseOnly/ms logging Both tests are gated on PERF=1 so normal `bun test` runs remain hermetic and don't hit live KBBI mirrors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(kbbi): stop misclassifying Indonesian luring/daring/agar/lama as English typo-js's en_US dictionary contains 'luring' (= enticing), 'daring' (= bold), 'agar' (= the gel), and 'lama' (= Tibetan monk), so isEnglishWord short-circuited isKnownWord into returning isEnglish=true for words that are firmly Indonesian in academic prose. Add the four to the INDONESIAN_WHITELIST so isEnglishWord rejects them up front. Also reorder the cache check in isKnownWord so a positive dictionary_cache hit (a word previously confirmed by an external KBBI lookup) beats isEnglishWord. Negative cache rows still fall through to the English fallback to preserve current behavior for loanwords like 'online' or 'blog'. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * wip(kbbi): proxy rotation scaffold + mock-server stress test Adds KBBI_PROXY_URLS (and unused KBBI_PROXY_LOCAL_ADDRS) optional env vars, a round-robin URL pool in utils/proxy.ts, and wires nextProxyUrl into cari.ts via fetch's Bun-native `proxy` option. Includes a vitest stress test that spins up a quota-enforcing mock KBBI server plus three local forward proxies that inject X-Proxy-Id, with phases for (A) the quota tripping when going direct, (B) rotation defeating the quota, and (C) a direct-vs-rotated RPS comparison. Phase B / C currently fail because vitest runs the test suite under Node 22 (not Bun) so the `proxy` option is silently ignored — the proxies never receive any traffic. Next step is to make the rotator expose either a Bun-style `proxy` string or an undici ProxyAgent dispatcher depending on runtime, then re-run the stress test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(kbbi): proxy rotation now actually routes through the proxy undici v7's ProxyAgent defaults to HTTP CONNECT tunneling, so a vanilla forward-proxy can't see the target URL or inject request headers. Pass { uri, proxyTunnel: false } so undici sends the absolute-URI form on HTTP targets and the mini-proxy in the stress test can identify clients via X-Proxy-Id. cari.ts now sets both `proxy` (for Bun's native fetch in production) and `dispatcher` (for Node-via-vitest), either of which is silently ignored by the other runtime. Stress test now passes all three phases: direct trips quota at K=11, rotated 30 reqs across 3 proxies clears with zero 302s, and the direct-vs-rotated RPS comparison reports the proxy-hop overhead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(kbbi): make proxy.ts actually export nextProxy + verify under Bun The previous fix commit only updated cari.ts and the stress test; the underlying proxy.ts still exported nextProxyUrl, so cari.ts imported an undefined symbol and would have crashed in production. The stress test masked it by shipping its own ProxyAgent helper. Replace nextProxyUrl with nextProxy() that returns both a proxy URL (for Bun's native fetch `proxy:` option) and a memoised undici ProxyAgent (for Node's `dispatcher:` option), with proxyTunnel: false so the proxy sees the absolute-URI form instead of a CONNECT tunnel. Adds .claude/scripts/verify-proxy-bun.ts that runs under Bun (not Node-via-vitest), spins up the same quota-enforcing mock plus three mini-proxies, imports nextProxy(), and asserts: (A) direct fetch trips the quota at K=10, (B) rotation defeats the quota with even distribution across all three proxies, (C) RPS comparison. Bun verdict: 1647 RPS direct, 1143 RPS rotated, zero quota trips. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(kbbi): auto-rotate via Tor sidecar with in-app HTTP→SOCKS bridge The official kbbi.kemendikdasmen.go.id mirror replies with HTTP 302 → /Beranda/BatasSehari (a "daily limit" landing page) once an IP has hit its per-day quota, not 429. The previous PR detected that signal but required operators to supply their own proxies via KBBI_PROXY_URLS to actually defeat the quota. This commit makes rotation work out of the box for the Docker deploy path while staying opt-in for bare-metal. What's wired up * docker-compose adds a `tor` sidecar pinned by digest (peterdavehello/tor-socks-proxy@sha256:0cc12d…), so the deployer doesn't get surprised by a future `:latest` rebase. The app service declares TOR_SOCKS_HOST=tor / TOR_SOCKS_PORT=9150 in env so the inner network can reach it. * New optional env vars TOR_SOCKS_HOST and TOR_SOCKS_PORT (defaults 127.0.0.1 / 9050 — works for local Tor installs too). * New src/services/evaluation/kbbi/utils/socks-bridge.ts: a CONNECT- only HTTP server that translates incoming HTTPS-tunnel requests into SOCKS5 commands against Tor. All four KBBI sources are HTTPS, so CONNECT-only is sufficient. No new deps (node:net + node:http). * proxy.ts gains an `ensureProxyPoolReady()` step the orchestrator calls before any lookup: it probes Tor on TOR_SOCKS_HOST:PORT with a 1.5 s timeout, and if reachable starts the bridge and registers its URL into the rotation pool. If Tor is unreachable, it logs a one-line notice and falls through to KBBI_PROXY_URLS or direct. Verification * tests/perf/kbbi-proxy.stress.test.ts (mock-proxy stress, no real Tor) bumped to 100 reqs across 10 proxies; zero quota trips, even per-client distribution. ~712 RPS rotated under Node-via-vitest. * tests/perf/kbbi-tor-bridge.test.ts (new): testcontainers spins up the pinned Tor image, waits for SOCKS port + a successful CONNECT through the bridge to api.ipify.org, then runs 100 requests at concurrency 10. Latest run: bootstrap 18.3 s, real IP 103.x vs Tor exit 185.x, 100/100 ok, p50=596 ms p95=1.8 s, 2 distinct Tor exit IPs observed (natural circuit rotation). Gated PERF=1 PERF_TOR=1 because it needs Docker + Internet + ~30-60 s extra. * .claude/scripts/verify-proxy-bun.ts bumped to 100 reqs / 10 proxies to confirm the same rotation works under the actual Bun runtime. Bun vs Node fetch — same belt-and-suspenders idea as before: cari.ts sets both `proxy: url` (Bun honors, Node ignores) and `dispatcher: ProxyAgent` (Node honors, Bun ignores), so the same call works in production (Bun) and in vitest (Node 22). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(kbbi): address gemini-review feedback on cache flush + throttle abort - flushCacheWrites: no longer clears pendingWrites before the DB insert resolves. Per-row deletion runs after the insert, and only removes rows whose entry hasn't been replaced by a newer write that arrived during the await. Failure preserves the queue so retry on the next flush is possible. (#2 reviews 3294102633, 3294114918) - throttleHost: accepts an optional AbortSignal and forwards it to the timer, so callers aborted mid-queue don't suspend forever holding the per-host slot. cari.ts forwards options.signal through to it. (#2 review 3294102634) Verified: kbbi-proxy.stress (100 reqs / 10 proxies / 0 quota) still passes; per-client distribution unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(kbbi): runtime toggle + per-source Tor scoping + BatasSehari detection Operator/user split for the Tor sidecar instead of always-on. Three moving parts: 1. Runtime opt-in toggle (configurations table). New row `kbbi.use_tor_proxy` (0 = off, default; 1 = on) seeded by `deploy/seed/configurations.sql`. The settings page now renders boolean-typed configs as a shadcn Switch via a new BooleanConfigurationCard variant; CONFIG_DISPLAY gains a 'boolean' kind. ensureProxyPoolReady() reads the config before doing anything; if off, it skips the Tor probe entirely (no 1.5s latency, no log). If on but Tor is unreachable, it logs a one-line warning and falls back to direct. 2. Tor scoped to kbbi.kemendikdasmen.go.id only. nextProxy() now takes the source name; the Tor bridge URL is only returned when source matches the quota-limited mirror. The other three KBBI sources never pay the Tor hop. cari.ts forwards the iterating source into nextProxy(source). 3. Compose profile keeps the sidecar idle by default. The `tor` service in docker-compose is now under `profiles: ["tor"]`, so plain `docker compose up` doesn't even start it. Operators enabling the toggle run `docker compose --profile tor up`. The app's depends_on no longer references tor (it's no longer a hard dependency). 4. BatasSehari detection. cari.ts checks res.url after the fetch resolves; if the request was redirected to /Beranda/BatasSehari for kemendikdasmen, treat it like a 429 — mark rateLimited, pause the host for 1 hour (long enough that the quota likely resets), and fall through to the other three mirrors. Cache isn't poisoned because conclusive && !rateLimited gates the write. Verified: mock-proxy stress test still passes (100 reqs / 10 proxies / 0 quota / 723 RPS); lint clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(kbbi): address copilot-review feedback on BK-tree, pause cap, body drain, runtime toggle - BK-tree query uses uncapped Levenshtein for node distance instead of the capped one. Triangle-inequality pruning requires the real distance from query to node; the capped value can shrink to bestCap+1 once bestCap tightens, which mis-computes the [lo, hi] range of valid child edges and skips correct suggestions for far-away nodes. (#2 review 3294574830) - throttle: MAX_PAUSE_MS was clamped to 5 minutes, so the 1-hour BatasSehari pause we set in cari.ts was silently truncated and the same daily quota page could be hit again 5 min later. Raised to 24 hours and bumped the kemendikdasmen quota pause to 12 hours so it spans the daily reset window. (#2 review 3294574774) - cari: every rate-limit-style branch (BatasSehari redirect, 429/503, non-ok response) now cancels res.body before continuing so Node / undici can release the socket back to the pool instead of leaving it held until GC. Matters on the high-volume lookup paths this PR adds. (#2 review 3294574800) - proxy: a `torEnabled` flag is set by ensureProxyPoolReady from the current `kbbi.use_tor_proxy` config. nextProxy() checks the flag before returning the Tor bridge URL, so flipping the toggle off takes effect on the next evaluation without needing to restart the process or tear down the bridge. (#2 review 3294574840) Verified: kbbi-proxy mock stress still passes — 100 reqs / 10 proxies / 0 quota / 717 RPS rotated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tor): drop compose profile, always-start sidecar; toggle is the only switch Previous design split control across two layers (operator runs 'docker compose --profile tor up', user flips kbbi.use_tor_proxy in /settings). That added cognitive load and required the operator to remember the flag. Tor at idle costs ~30 MB RAM, which is trivial for a thesis-tool deploy. - docker-compose.yml: remove profiles: ["tor"] from the tor service so plain 'docker compose up' starts it alongside db and app. - proxy.ts: rewrite the unreachable-Tor warning so it doesn't tell operators to run a now-nonexistent --profile flag. - configurations.sql + lib/configurations.ts: update the kbbi.use_tor_proxy description to match the always-on reality and surface the graceful-fallback behavior to operators reading the settings panel. - deploy seed gains the kbbi.use_tor_proxy row that was missed in the earlier feat commit (the edit landed in the wrong working tree). Verified: mock-proxy stress test still passes (100 reqs / 10 proxies / 0 quota / 651 RPS rotated). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
MAX_PAGES_VISIBLE=12 was clipping the page list to the first 12 and hiding the rest behind '& N lainnya'. For findings that legitimately span 16+ pages (e.g. an unitalicised foreign term used throughout the thesis), the truncation hides exactly the data the user wants to act on. Drop the cap and let the existing flex-wrap row grow. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bring the history list in line with the cleaner "paper-stack" look: filename + relative time on the left, scattered doodles in the middle, inline stats (sitasi/cocok/kalimat/HLM/DTK for Track, temuan/HLM/DTK for Evaluation), and a status pill on the right. Drop the vertical marginalia rule and the cramped right-column metadata that duplicated what is now in the inline stats. Cards get a hard 5px offset shadow that grows to 7px on hover and collapses to 2px on press, matching the 3D feel of the new home/feature cards. Bump the list gap so adjacent shadows don't overlap. Also smooth the relative-time copy from "3 jam lalu" to the friendlier "3 jam yang lalu" wording the design comp uses; this helper is only read by the history row. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ages Grid graph-paper bg now reads across every band on the home page (Citation Tracer / Evaluation / Closing CTA), and across the cream upload bands on /track and /evaluation. Home page hero card and the feature cards pick up the same offset 3D shadow as the new history cards, so the visual language stays consistent. Two scroll perf fixes were needed once the grid lived on four sections of one page: - Drop the radial-gradient mask on .section-band[data-grid]::before. Compositing that mask over a tiled background on every paint frame was the home-page scroll bottleneck. Each section is its own colored band, so the grid lines reading edge-to-edge looks fine without the fade. - Promote the grid pseudo to its own compositor layer (translateZ(0) + will-change: transform) so the tiled bg rasterizes once and just translates during scroll instead of repainting. - Add content-visibility:auto + contain-intrinsic-size to the non-hero home sections so off-screen sections skip layout + paint entirely. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ctable model (#3) * chore(deps): add @huggingface/transformers for local embeddings Brings in ONNX runtime + HF transformers (~150MB across deps). Used by upcoming multilingual passage matcher to embed Indonesian thesis contexts against (mostly English) source PDF windows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(config): add passage.embedding_model enum config Widens the previously all-numeric config registry to accept enum values. Adds `passage.embedding_model` (none / minilm-l12-v2 / e5-small (default) / e5-base) with Indonesian labels, hints and per-model size/RAM tradeoff notes for the settings page. formatConfigForDisplay and parseConfigFromDisplay now take and return AnyConfigValue (union number | string) and dispatch on display kind. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(db): add source_window_embeddings table One row per (source_pdf, model, page, window). Stores the window text and its float32 embedding as bytea. The composite uniqueness constraint on (source_pdf_id, embedding_model, page_number, window_idx) lets us idempotently upsert when re-embedding a PDF. Keyed by embedding_model so changing the configured model in settings invalidates the old vectors lazily — the matcher checks the model and recomputes on miss. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(matcher): add embedder module wrapping HF transformers Lazy-loads the configured ONNX model (q8 quantized) per name. Singleton cache keyed by model so the same Embedder instance is reused for the lifetime of the process — model load is the expensive bit (2-7s cold), embedding inference is fast. Handles the e5 family's asymmetric "query: " / "passage: " prefix convention; MiniLM uses the same encoder for both. Smoke test on "Kotlin dapat bekerja..." (Indo) vs "Kotlin works seamlessly..." (Eng) gave cosine 0.89 with e5-small — the cross-lingual signal the existing lexical matcher cannot see. Buffer ↔ Float32Array helpers preserve bits exactly (verified roundtrip cosine = 1.0) so DB-stored vectors don't drift. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(matcher): two-stage semantic + lexical passage matching Rewrites matchPassage to be async and accept an optional Embedder (plus pre-computed window embeddings for the cached-vector path). Stage 1 — Semantic: cosine similarity between query and each window's embedding. Reject below absolute floor (0.78 for e5, 0.55 for MiniLM). Track top vs runner-up cosine and reject when they're indistinguishably close — that's the "topical noise" signature of a wrong-paper match. Stage 2 — Lexical guard: every accepted match must clear at least one anchor: exact 8-word n-gram, shared proper-noun, shared year/number, or token-overlap rate >= 0.2. Cross-lingual paraphrase naturally has zero overlap on content words but technical terms (Kotlin, Java) and numbers cross language boundaries. Lexical-only mode (model='none' or no embedder): same anchors, combined score = 0.6 * token_overlap_rate + 0.25 * exact + 0.1 * fuzzy + 0.05 * proper_noun_overlap. Honest "null" return when nothing clears the threshold instead of the old BM25-renormalized 0.6 false-confident output. Removes the buggy relative BM25 normalization that previously returned confidence=0.6 even when the source PDF was completely unrelated to the citation. Citation markers are stripped from the query (the surrounding text is what we want to match, not "(Hakim, 2023)" itself). Smoke results (single-page synthetic fixtures): Same-lang paraphrase: lex 0.54 ✓ | e5-small 0.51 ✓ Cross-lingual paraphrase: lex null | e5-small 0.62 ✓ Wrong source paper: lex null ✓ | e5-small null ✓ passages.ts now awaits the result; embedder + cache wiring lands in the next commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(track): hydrate passage matcher with cached embeddings matchPassagesForJob now fetches the configured embedder and, for each source PDF in the job, loads the precomputed window vectors from source_window_embeddings. Hits feed straight into matchPassage via cachedWindowEmbeddings. On miss (cold PDF, or a different model than what was previously cached) the helper computes embeddings for every window with the current embedder and persists them — the "lazy re-embed" promised in the settings copy. Subsequent matches against the same PDF in the same job reuse a per-call in-memory map so we never refetch from DB across citations of the same source. When the model is "none" (or no embedder is configured at all), the matcher runs in lexical-only mode and the embeddings table stays untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(settings): render Select dropdown for enum configs ConfigurationCard branches on CONFIG_DISPLAY when rendering its field: enum keys (currently only passage.embedding_model) get a shadcn Select wired to the TanStack Form field handler, with the per-option hint rendered underneath so the user can read the size/RAM tradeoff inline without diving into docs. Numeric inputs keep their existing Input + unit-suffix shape. Added a 'blush' card tone + 'pencocokan kutipan' group label for the new passage.* family. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(matcher): add 4-tier embedder comparison script Runs the matcher against thesis_example.pdf with each of the four configured model tiers (none / minilm-l12-v2 / e5-small / e5-base) and emits a quantitative report. Three measurements per model: A — Self-match recall: thesis used as its own source; each citation should re-find the page it was extracted from. Reports match rate, correct-page rate, average confidence. B — Wrong-paper precision: 14484.pdf (an unrelated Kubernetes/IoT paper) used as source for every thesis citation. A correct matcher should reject all. Reports rejection rate + sample false positives. C — Cross-lingual fixture: 6 handcrafted Indonesian thesis claims, each paired with one English paragraph that paraphrases the claim plus distractors. Reports top-1 accuracy (matched passage starts with the target sentence). Also captures cold-load and per-citation timings. Writes raw JSON to .claude/scripts/output/ (gitignored) and a human-readable summary to docs/track/embedder-comparison.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(matcher): precompute embeddings, corpus subset, commit results Script changes: - Precompute window embeddings once per (model, source PDF) and pass via cachedWindowEmbeddings. Earlier draft re-embedded the source on every citation; with ~1000 thesis windows × 44 citations that meant tens of thousands of redundant encodes per tier (pegged a laptop until OOM in the worst case). - MAX_THESIS_PAGES env var (default 30) caps the recall corpus to a slice of the thesis so the benchmark finishes in minutes on a laptop. Citations whose first occurrence falls outside the slice are excluded. - Report header now records corpus size and total. Results on first 30 pages of thesis_example.pdf (37 of 44 citations), 14484.pdf as the wrong-paper source, 6 handcrafted Indonesian↔English fixture pairs: none: recall 100% · precision 100% · cross-lingual 0/6 minilm-l12: recall 89% · precision 100% · cross-lingual 2/6 e5-small: recall 97% · precision 89% · cross-lingual 2/6 ← default e5-base: recall 57% · precision 97% · cross-lingual 1/6 Headline reading: - Lexical-only ("none") still wins same-language; correctly cannot bridge cross-lingual paraphrase. - e5-small is the right default — it picks up Indonesian↔English matches the lexical path can't see, at a cost of 4/37 false positives. Cross-lingual top-1 is lower than I'd hoped (2/6); the handcrafted fixture's distractors are intentionally topical, and the cosine floor of 0.78 is probably too tight. - e5-base recall collapses to 57% with the same floor. Almost certainly miscalibrated — e5-base's cosine distribution is not the same shape as e5-small's. Tune COSINE_FLOORS per-base in a follow-up. Until then e5-base is a regression, not an upgrade. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(matcher): chunk embedding inference to bound memory on long PDFs A 400-page thesis source produces ~3000-4000 sentence windows. Calling extractor(allWindows, ...) materialized the full attention matrix in one tensor — easily several GB on a laptop, and the trigger for the OOM-flavored slowdown we saw running the benchmark on the full thesis. Move the batch boundary inside Embedder: every embed call is now chunked at the spec-defined batch size (32 for 384-dim models, 16 for e5-base) regardless of how big the caller's array is. Callers don't need to know — passages.ts, the matcher's lazy-compute path, and the comparison script all stay one-line calls. Optional onChunk callback added for progress reporting; the production Track flow can wire it to job status later. Verified: 80 texts × batchSize 16 = 5 chunks, output count and dim match single-batch path, cross-mode cosine 0.997 on identical input (float-order noise, no semantic drift). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(track): refresh embedder benchmark after chunking fix Same 30-page corpus, same fixture, this time through the chunked embedder. Recall is identical across all four tiers (as expected — float reordering from smaller batches shouldn't change top-1 ranking). Precision is slightly better: e5-small: 89% (33/37) → 97% (36/37) 3 false positives gone e5-base: 97% (36/37) → 100% (37/37) 1 false positive gone The sub-percent cosine drift from smaller batches pushed a few borderline windows just below the absolute floor — they were already weak matches; chunking happens to filter them cleanly. e5-base cold load dropped 10s → 2s because the ONNX file is now cached locally on disk from the first run; first-ever cold load on a fresh machine is still ~10s while HF downloads ~280MB. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(track): address Gemini review on PR #3 Three medium-severity findings from gemini-code-assist: 1. passages.ts:52 — partial cache bug loadOrComputeWindowEmbeddings used "any existing rows = cache complete" as the exit condition. If a previous job died mid- insert (or was killed for memory), the table held a partial set of windows; we'd return the partial Map and the matcher would silently recompute the rest in memory on every call, never persisting. Now we always build the full window list first, fetch what's cached, then compute + persist only the missing ones. Cache eventually completes by itself. 2. passages.ts:73 — bulk insert / race on unique index A 400-page thesis can produce ~3500+ windows × 7 inserted columns = ~24500 placeholders. Comfortably under Postgres's 65535 bind-param ceiling but close enough to be brittle, and concurrent Track jobs hitting the same source_pdf_id would collide on the (sourcePdfId, model, page, windowIdx) unique index. Chunk inserts at 500 rows and use onConflictDoNothing targeting the composite unique columns. 3. passage-matcher.ts:40 — sentence splitter fragility The regex `(?<=[.!?])\s+(?=[A-Z0-9])` happily breaks inside "et al.", "Dr.", "e.g.", "Fig. 3", "tsb.", "Dr. Hakim". Fixed with a guard list of English + Indonesian abbreviations: after the regex splits, segments whose previous segment ends in a known abbreviation get merged back. Also handles single-letter dotted acronyms (U.S.A.). Stays in-house — no new dependency. Verified by smoke runs covering each abbreviation pattern and an end-to-end semantic match across the new splitter output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(track): refresh benchmark after splitter abbreviation guard Same 30-page corpus, same fixture. The new abbreviation-aware sentence splitter produces a slightly different window set: 14484.pdf goes from 206 → 200 windows (fewer abbreviation-induced fragments, slightly longer merged windows). Net effect on numbers: Recall: identical across all four tiers Cross-lingual: identical Precision: e5-small 36/37 → 34/37 (-2) e5-base 37/37 → 35/37 (-2) lexical & minilm unchanged Direction is honest: longer, less-fragmented windows carry slightly more topical content, so the cosine path picks up a few extra borderline matches against the unrelated source. These samples are right at the floor (0.44-0.56) and are the same class of "Semantic + token overlap" / "shared number" calls already in the report. The right durable fix is per-model floor calibration (already a follow-up); the splitter change makes the returned passages more readable end-to-end and was the higher- value win. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the design we landed on after the brainstorm: a loggedFetch wrapper writing to a new api_call_logs table, surfaced on a globally accessible /admin/api-logs route. Scoped to Track auto-fetch + Evaluation KBBI. Errors get the full response body up to 1MB, successes get a 2KB preview, binary PDF downloads get metadata only. FK cascade handles retention via the existing purge flow. Implementation will land on the feat/third-party-logs worktree. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ches Passage matching used to be one server fn that processed every matched citation in a single request and returned only when the whole thing finished. The user saw a static "Finding passages…" message for 30-60+ seconds with no signal that work was actually happening, and a mid-flight failure (one bad source PDF, transient network blip) wiped the entire run. Split the work into per-source-PDF batches that persist in a new passage_match_batches table: - enqueuePassageBatches partitions matched citations by source PDF and inserts one row per source (status, counters, attempts, error). No- source citations are returned inline so they render instantly. - processPassageBatch handles one source: loads embeddings (existing per-source cache still applies), runs matchPassage for each citation, writes passage_matches, updates the batch row's counters + status. Auto-retries once internally on failure, wiping any partial inserts between attempts. Idempotent — re-calling a done batch reconstructs results from the DB instead of re-running the matcher. - retryFailedPassageBatches flips failed rows back to pending for a manual retry from the UI. Client orchestrates the queue sequentially via the new handleMatchPassages: enqueue, then await each processPassageBatch in turn, updating the Zustand store after each. A new PassageBatchProgress panel renders during the matching-passages phase with a progress bar, per-source rows (filename, reference label, status icon, count of matched/total), elapsed timer, and a "Coba lagi sumber gagal" button that appears when at least one batch is in failed state. The phase only advances to review-passages once every batch is done; if any failed, the user stays on the matching screen with retry available. The drizzle-kit push --force in docker-entrypoint.sh picks up the new table on container start — no manual migration step. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…logs (#6) * feat(db): add api_call_logs table for 3rd-party request tracking Two nullable job FKs (track_job_id → jobs, eval_job_id → evaluation_jobs) with ON DELETE CASCADE so logs vanish when their originating job is purged — no extra plumbing in the existing purge sweep. Unattached logs (both FKs null) get cleaned up by a TTL pass added in a later commit. Indexes target the three query shapes the admin page actually needs: (created_at) for the default reverse-chrono list, (provider, created_at) for the provider filter, and per-FK lookups for job-scoped drill-in. Enum captures the four outcome shapes — success, http_error (got a non-2xx response), network_error (fetch threw before getting a response), timeout — so the UI can render them distinctly without parsing free-form error strings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(logs): add loggedFetch wrapper for 3rd-party HTTP Thin wrapper around fetch that records each call to api_call_logs. Returns the original Response so callers can consume .json() / .text() / .arrayBuffer() normally; the body is captured off a Response.clone() in a fire-and-forget background task so logging never adds latency. Capture rules: - success: first 2KB of body - non-2xx: full body up to 1MB - metadataOnly (binary PDF download): no body, only status + headers - network error / timeout / abort: outcome distinguished in the row Header subset captured: content-type, content-length, retry-after, all x-ratelimit-*. The full header set isn't useful for diagnostics and would balloon the JSONB column. Logging-side failures (DB unreachable, body read error) are silently swallowed — the caller never fails because of logging. Verified behavior with a stubbed fetch: small/large success bodies, non-2xx, and thrown network errors all return the right thing to the caller; truncation lands at the configured cap; the original Response remains consumable after the wrapper has cloned it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(track): route Track 3rd-party fetches through loggedFetch All 11 provider attempts in finder.ts (DOI resolver, Unpaywall, Semantic Scholar, CrossRef, OpenAlex doi/title, Europe PMC doi/title, PubMed eSearch, arXiv, Core) plus the PDF binary download in auto-fetch.ts now go through loggedFetch. The binary PDF download uses metadataOnly: true so we capture status + content-length + content-type without buffering the PDF body itself — keeps log rows small for the common case where the response is multi-megabyte binary. processReference wraps its work in withApiLogContext({ trackJobId }) so the trackJobId propagates to every loggedFetch call inside via AsyncLocalStorage. Saves threading a jobId argument through the 11 try* function signatures, and future fetches added inside the context inherit attribution automatically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(evaluation): route KBBI scrapes through loggedFetch cari.ts is the single fetch site for KBBI dictionary scraping across the four configured sources (kbbi.kemendikdasmen.go.id, kateglo, prpm-dbp, kateglo-mirror). Replaces the raw fetch with loggedFetch tagged as 'kbbi' provider. Orchestrator wraps runEvaluationAnalysis in withApiLogContext({ evalJobId }) so the evalJobId attaches to every KBBI scrape fired during the analysis pass — same AsyncLocalStorage pattern as Track. Cache warm-ups (warmKbbiCaches) run inside the context too, so even pre-flight lookups attribute to the job that triggered them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(logs): server fns listApiCallLogs + getApiCallLog listApiCallLogs accepts provider, outcome, trackJobId, evalJobId, limit, and a before cursor for reverse-chrono pagination. Returns one extra row past the limit to compute the nextCursor — standard keyset pagination over createdAt. getApiCallLog returns the full row including body_preview for the detail panel. Outcome filter collapses to three buckets the UI cares about: all / errors (any non-success outcome) / success. The four-state outcome enum on the schema is still observable per-row. No assertLocalOnly gate — the page is meant to be accessible to whoever runs the compose stack, per the local-first model. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(admin): /admin/api-logs page New globally accessible admin route — no isLocalEnv gate, matching the local-first model where operators of their own compose stack get access to their own diagnostic surface. Layout: - Mint header band with kicker + accent headline + 1-paragraph Indonesian explainer - Cream content band with filter bar above a compact 7-column table (expand icon, time, provider pill, status, duration, url, outcome badge) - Click a row to expand inline — fetches the full row via getApiCallLog and renders headers as a definition list plus the body in a max-h-[40vh] scroll pane. JSON bodies are pretty-printed; everything else is rendered as monospace text with whitespace preserved - Cursor pagination via useInfiniteQuery; "Muat lebih lama" pill loads the next page Filters: provider chip multiselect, outcome toggle (all / errors / success), uuid inputs for trackJobId + evalJobId. Local state for now (would migrate to route search-params if we want shareable links). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(purge): clean unattached api_call_logs by retention Logs attached to a track or evaluation job already cascade-delete with the job — the FK ON DELETE CASCADE picks them up for free. Logs with both FKs null (manual diagnostic calls, cache warm-ups that fired outside any job) don't cascade; they need a TTL sweep. purgeHistory now appends one DELETE FROM api_call_logs WHERE track_job_id IS NULL AND eval_job_id IS NULL AND created_at < retention_cutoff and surfaces the count as apiLogsDeleted on PurgeResult. pruneAll also drops all api_call_logs rows since cascade only covers the attached ones — the wipe-everything semantics require the unattached set to go too. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(logs): smoke script for loggedFetch + api_call_logs End-to-end script under .claude/scripts/ that exercises the three real outcome paths against a stubbed globalThis.fetch: 1. successful response (200 + JSON body) 2. HTTP error (403 + body capture under the 1MB cap) 3. network error (fetch throws — caller still sees the throw, wrapper still logs the outcome) Then waits 1s for the fire-and-forget inserts to flush and reads rows back from api_call_logs (filtered to provider='openalex' which is namespaced enough for repeated runs). Run with: NODE_ENV=test DATABASE_URL=... bun .claude/scripts/test-api-logs.ts Output landing in the table demonstrates the loggedFetch wrapper + schema + connection are wired correctly. The success case uses a fake trackJobId that won't pass the FK, so we expect that single insert to fail silently — by design, logging must never break the caller — and the other two cases (no jobId attached) land cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(logs): stream body capture so a huge error response can't OOM us readBodyWithCap previously called res.text() and then sliced the string. If a misbehaving provider returned a 100 MB HTML error page instead of JSON, the whole thing would land in memory before we ever applied the cap. Switch to a stream reader: pull chunks from res.body, decode incrementally, stop the read (and cancel the stream) once we cross the cap. totalBytes still tracks the full body size so the log row reports the true size even when truncated. Caller path is unchanged — we read off res.clone(), so the original Response stays consumable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(logs): inArray operators + stable keyset pagination on api-logs Two cleanups from the Gemini review: - Provider and outcome filters used or(...map(eq)) which generated a chain of OR equalities. Swapped to inArray — same SQL shape the database optimizer prefers, and reads cleaner. - Pagination ordered by createdAt alone. With concurrent inserts the millisecond timestamp can collide, and a pure-time cursor skips or duplicates rows across page boundaries when that happens. Added id as a secondary sort key (DESC, same direction as createdAt), and changed the cursor shape from a bare ISO string to { createdAt, id }. The cursor WHERE clause is now the full keyset predicate: (createdAt < t) OR (createdAt = t AND id < n) — the textbook fix for tied sort keys. Admin route bumped to feed the new cursor shape into useInfiniteQuery's pageParam. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
loggedFetch imports node:async_hooks for AsyncLocalStorage, which makes the whole module server-only. The client-side admin route was importing API_PROVIDERS and the ApiProvider type from that same file for the provider chip filter, which pulled node:async_hooks into the browser bundle and broke the Vite production build with: "AsyncLocalStorage" is not exported by "__vite-browser-external" Move API_PROVIDERS, ApiProvider, and ApiCallOutcome into a new src/services/logs/providers.ts that has no server-only imports. logged-fetch.ts and api-logs.ts (server) and admin/api-logs.tsx (client) all read from providers.ts. logged-fetch.ts still re-exports the same names so any external code keeps working. Verified by a full docker compose build app — the production Rollup bundle now passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Added "Log API" alongside Riwayat and Setelan in the local nav group. I built the route in PR #6 but skipped wiring it into the header — users had to type the URL by hand. It's reachable now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two tweaks the user flagged from the matching/results screen:
- PassageBatchProgress's "Berjalan {n}s" stat used to compute
elapsedSec at render time, so it only updated when batches
state changed (every batch completion, ~30-60s apart on slow
sources). A useElapsedSeconds hook now ticks every 1s via
setInterval. The interval clears once all batches have settled
(done | failed) so the timer freezes on the final value
instead of running forever.
- PassageResults now sorts rows: matched (by confidence DESC),
then no-match (N/A), then no-source (No PDF). Matches the
user's expectation that the high-signal rows are at the top
and the unactionable rows sink to the bottom.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The history card showed "3 DTK" for jobs whose real end-to-end runtime was several minutes. The duration is computed as jobs.updatedAt - jobs.createdAt, but the only thing that bumped updatedAt was the status transition to 'done' after PDF extraction (a few seconds in). Citation parsing, reference matching, source auto-fetch, and passage matching never touched the jobs row. After a batch lands in passage_match_batches with status='done', write a no-op UPDATE on jobs so $onUpdate refreshes updatedAt. Last batch wins, so the history duration now reflects when the final passage batch finished. Failed batches don't bump it — that path throws and the row keeps its older timestamp, which is the right thing: a job that died at extraction shouldn't get a 5-minute "duration" from later retry attempts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shadcn TableCell defaults to whitespace-nowrap. That made the source passage (and the thesis context) render on a single line that pushed the Citation column out past the viewport, with the table's own overflow-hidden container clipping any chance of scrolling to see it. Override with whitespace-pre-wrap + break-words on the inner <p>, and wrap each in an overflow-x-auto container so any exceptionally long unbreakable token (URL, hash) still scrolls within the box instead of bleeding into the next column. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Title pages were winning the cosine race because abstracts summarize the whole paper using every keyword the thesis paraphrases, while body windows get diluted by tables and figures. Skip past ISSN/ABSTRAK/ Keywords boilerplate when a clear body marker (PENDAHULUAN / INTRODUCTION / Latar Belakang) is present, so the matcher scores real prose instead of the cover block. When front-matter signals fire but no body marker is found (running header on a body page, or unusual layout), leave the page untouched rather than dropping it — preserving real body windows matters more than perfectly clean displayed passages. Also await the four pre-existing matchPassage tests that were silently failing since the matcher became async. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The "Mulai pencarian" and "Find Passages →" buttons both rendered as
coral on the same page, so users had to guess which one to click first.
They are sequential steps (fetch sources, then match passages), not
alternatives, but nothing on screen said so.
This pass makes the two phases visually and verbally distinct:
- Step 1 (in the auto-fetch card): kicker now reads "Langkah 1 ·
Sebelum mulai"; button label is "Cari PDF sumber otomatis" so the
verb names the action; helper line underneath explains what gets
downloaded.
- Step 2 (bottom nav): wrapped in a top-bordered band with "Langkah 2 ·
Cocokkan kutipan" kicker. Button renamed to "Cocokkan kutipan →"
(matches the rest of the Indonesian copy), paired with a "{N} sumber
siap" badge that goes mint when at least one PDF is paired. While
disabled the button drops to the outline variant so only one coral
CTA is lit at a time, and an italic helper line tells the user why
it's inactive.
Also translated the remaining English strings on this panel (drop-zone
prompts, source-list labels, "Pair with reference" / "Unassigned",
upload error fallbacks) to match the rest of the Indonesian UI. Added
an aria-label on the hidden file input to clear a pre-existing
jsx-a11y/control-has-associated-label warning that surfaced once the
file got re-linted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move the previously-stubbed `isLocalEnv` / `assertLocalOnly` pair from "always local" to actually driven by `PUBLIC_MODE=true`. Default stays false so `docker compose up` on a developer's machine keeps every feature visible — opt-in to public mode, not opt-out. Add the rest of the prod-deployment knobs as env vars with sane defaults so they can be tuned per-VPS without code changes: MAX_CONCURRENT_JOBS (default 1, safe for 2cpu/2gb), MAX_PDF_PAGES (default 250), JOB_RETENTION_DAYS (default 7), POLITE_POOL_EMAIL (empty by default, set for OpenAlex/CrossRef polite-pool access). Also gate the two `apiCallLogs` server functions behind `assertLocalOnly()` — they were previously reachable from public deployments even though `/admin/api-logs` was hidden from the nav. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a browser-scoped session UUID (Zustand + localStorage persist) and plumb it through the upload boundary so jobs/evaluationJobs can be filtered to "what this browser created". Local mode ignores the filter and shows everything, so existing single-user installs keep their current history view; public mode requires the session ID and 404s without one, giving anonymous public users a private job list without a login. Changes: - jobs/evaluation_jobs: nullable session_id column + index (migration 0009) - src/stores/sessionStore.ts: persisted Zustand store with auto-generated UUIDv4 - uploadThesis / uploadEvaluationThesis: read sessionId from the form, write it to the row; UI sites that call these now append the session ID to FormData - history server fn: optionally scoped by sessionId; throws "Not Found" in public mode when sessionId is absent - /history route: switched from SSR loader to client-side useQuery (so it can read sessionId from localStorage); ungated from isLocalEnv since the server handler now enforces the scope itself Also fixes two pre-existing jsx-a11y/control-has-associated-label warnings on the hidden file inputs surfaced by the re-lint pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a small in-process semaphore (env.MAX_CONCURRENT_JOBS, default 1) that gates the two RAM-heavy extraction handlers — processUpload and processEvaluationUpload. On a 2cpu/2gb VPS, where a single 400-page PDF can spike to ~500MB resident, two concurrent extractions would OOM-kill the container. Serialising them through one slot is the difference between "queued for a minute" and "everyone fails". Also rejects PDFs whose extracted page count exceeds env.MAX_PDF_PAGES (default 250) with a clear Indonesian error message pointing big-thesis users at a local install. The check runs after extraction since that's the first cheap place we know the real page count, and the job is marked failed via the existing error path so the UI surfaces it. withJobSlot is exported as a generic wrapper so it can later gate other heavy steps (passage-matching batches, embedder warmup) without re-introducing the semaphore plumbing each time. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three pieces: 1. Explicit pg.Pool in src/db/index.ts. Drizzle was previously letting node-postgres create an implicit pool with library defaults (max=10, no idle timeout), which is fine but opaque. Wiring it explicitly means we can tune max and timeouts here when we need to instead of grepping inside node_modules. Stays at max=10 for now, well under the new max_connections=20 on the db side. 2. Postgres command-line overrides in docker-compose. The alpine image's stock postgresql.conf assumes much more RAM headroom than a 2cpu/2gb host has — `shared_buffers=128MB`, `work_mem=4MB`, `max_connections=20`, `effective_cache_size=384MB`, `maintenance_work_mem=32MB`, `wal_compression=on`. Tuned for the smallest reasonable VPS; raise via a compose overlay on bigger hosts. 3. Daily retention sweep that deletes jobs and evaluation_jobs older than env.JOB_RETENTION_DAYS (default 7). FK cascades clear pages, citations, source_pdfs, etc. without needing per-table sweeps. Self-schedules from src/db/index.ts so any server fn that touches the db arms it on first import; runs once a minute after boot and then every 24h. setInterval is `.unref()`ed so it never holds process exit during graceful shutdowns. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ad analyser Three measurement-related pieces in one commit: 1. `api_call_logs.cache_hit` column (migration 0010) — defaults false, set true when a future cache layer serves a result without going out to the network. With it we can compute the cache hit rate directly from the same admin view, no inference required. 2. Polite-pool helper applied transparently inside loggedFetch. When POLITE_POOL_EMAIL is set, OpenAlex / Unpaywall calls get a `mailto=<email>` query param and CrossRef gets a `User-Agent: CiteTrack (mailto:<email>)` header — both routes us into the provider's much higher per-IP rate-limit bucket. No-op when the env var is empty so local dev stays anonymous. 3. `.claude/scripts/measure-api-load.ts` — analyses existing api_call_logs across a configurable window (WINDOW_HOURS, default 24) and reports: per-provider 429 rate, worst burst minute, per-job call cost (p50/p95), and cache hit potential (unique vs total URLs on lookup-style providers). Produces a verdict against the "A enough / borderline / insufficient" thresholds. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Switched the column separator from \t (which the surrounding shell escape couldn't deliver intact) to psql's default pipe. Pipe doesn't appear in our provider names or URL paths so it's safe.
Both badges were hand-rolling `bg-secondary/20 text-secondary-foreground` on the shadcn Badge — that pairs the *light* indigo foreground (designed for solid indigo) with a 20%-opacity indigo wash, so the text vanishes into the band. Route them through the project's Badge variants instead: needs-review → default (butter/warning), verified → secondary (sky/info), high confidence → secondary, mid confidence → default. Now matches the warning/info severity map already declared in the results dashboard. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both admin pages were rendering as flat colored Section bands with plain text — same shell as history / track / results, but none of the seasoning those pages use, so they read as generic admin chrome dropped into the app. Add the signature hero recipe to both: rounded white kicker pill with inline StarBurst, six scattered doodles (Sparkles, DottedArc, PaperPlane, Squiggle, StarBurst, Arrow), Underline below the lead. On settings, swap the headline AccentInk to Marker tone="green" so the accent matches the mint section tone, and move AccentInk down into the paragraph for a second beat the way track does. Alias lucide's Sparkles icon to SparklesIcon in settings so the doodle Sparkles import wins. Section gains `grid` and `relative` to anchor the absolute doodles. routeTree.gen.ts catches up on the /admin/api-logs route registration that was missing from the generated tree. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codifies CiteTrack's actual Vitest conventions rather than generic TDD advice. Triggers on "add tests", "this test is flaky", "review the test suite", "FP guard test". Covers four modes — validate, enhance, create unit, create integration — and locks naming (*.test.ts vs *.integration.test.ts), location (tests/ mirroring src/), the #/ alias, explicit timeouts for PDF/KBBI runs, refreshVocabularyCache in beforeAll, real Postgres via testcontainers (never mocks for DB), FP guards paired with positive controls, and descriptive expect(value, msg). Every code sample is lifted from a test that already lives in this repo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This reverts commit 2bfd45b. Reverting because the /history route switched from an SSR loader to a client-side useQuery, and the client-side fetch never resolves in the browser (loader hangs on "Memuat riwayat…" indefinitely even though the /_serverFn endpoint returns 200 with the data in <100ms via curl). Keep migration 0009 (session_id column) intact for now to avoid a broken-snapshot chain with later migrations; a follow-up commit will drop the column. Also keep the a11y aria-label fix on the hidden file inputs that 2bfd45b introduced — those were correct fixes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to the revert of 2bfd45b. The session_id columns on jobs and evaluation_jobs are no longer referenced by application code, so drop them along with their lookup indexes. Migration 0011 was applied to the live dev DB directly via psql because __drizzle_migrations is empty (this dev environment uses db:push, not db:migrate). The SQL file and journal entry stay authoritative for fresh installs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The global settings page now shows only the cross-cutting knobs — upload.max_file_size_bytes and the two purge windows — in a 2-col grid, plus the purge/prune actions and the dev preview toggle. Per-feature config moves to admin panels on the feature pages. Drops the tab nav, search box, and horizontal-scroll card row. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the local-env-only AdminSettingsPanel to the bottom of the Track page, surfacing the autofetch timeouts/concurrency and the passage embedding model where the feature lives. Hidden from public visitors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the local-env-only AdminSettingsPanel to the Evaluation landing page with the full KBBI knob set — local-only, dump bypass, Tor proxy, lookup budget/timeout, and the combined source toggles. Repoints the KBBI error link on /admin/api-logs from /settings?tab=kbbi to /evaluation where the config now lives. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The slimmed global page and per-feature admin panels no longer use the settings search box or tab nav, so drop their machinery: settingsSearchSchema (src/schemas/settings.ts), rankByQuery (src/lib/settings-search.ts) and its test, the CONFIG_KEYWORDS search catalog, and the route's validateSearch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Render the API-call stats panel even when no calls match the filter, instead of hiding it (which read as missing UI). At zero traffic the success/error/aborted hints say 'tidak ada panggilan' and the duration card shows '—' with 'belum ada data' rather than a misleading 0 ms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a "Setelan" pill next to "Cara kerja" in the hero eyebrow on the Evaluation and Track pages. It anchors to the inline admin settings panel below. Gated on isLocalEnv so it only shows when the panel is rendered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pty results jobs.status='done' is set the moment PDF extraction finishes, so history routed every extracted job to /results/$jobId — even ones stranded mid-pipeline (no sources uploaded, no passages matched), showing an empty report. Persist the pipeline checkpoint on jobs.phase and route from history by it: review-passages (terminal) opens the results report, anything earlier resumes /track at the exact step. Existing jobs are backfilled from their rows. - jobs.phase (pipeline_phase enum), default 'upload' - setJobPhase server fn, persisted on each resumable client transition - /track loader + hydration fall back to the persisted phase when the URL omits it - history surfaces phase; row label shows 'Lanjutkan' vs 'Selesai' accordingly Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ible PassageResults: the expanded "Konteks di skripsi" / "Kalimat sumber" text sat inside the narrow Sitasi cell, so in the auto-layout overflow-x table the column grew and the text ran off-screen instead of wrapping. Move the detail into a full-width row spanning the table (whitespace-pre-wrap + break-words) so every line wraps; also let the source-file/reference column wrap. status-indicators: the needs-review AlertTriangle used text-secondary-foreground (#ffffff), rendering white-on-cream. Use a marker-yellow fill with an ink outline — visible and consistent with the "perlu ditinjau" yellow marker. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reframe the landing page and nav around the language-evaluation feature while
keeping the CiteTrack name and the citation tracker as a secondary path.
- Nav: Evaluation now sits before Track
- Hero: subtext + primary CTA lead with "Periksa tulisan"; the sample-finding
quote card is now an EYD nitpick ("analisa" vs "analisis")
- Sections reordered (Pemeriksaan before Pelacak sitasi); evaluation gets the
coral primary CTA, track steps back to the indigo secondary CTA
- Closing CTA + meta description lead with the language check
- Copy rewritten through the humanizer: warmer, more specific, no forced
parallelisms or puffery
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ad route Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the db/fs/paths/docx orchestration into a server-only runner imported dynamically inside the server fn handler, so node built-ins no longer leak into the browser build. Register the download route in the generated route tree. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… into dropdown Replace the always-open "Terapkan perbaikan" panel with a button beside Bandingkan that opens it in a modal. Merge the XLSX and annotated-PDF download buttons into a single "Unduh" dropdown (Excel / PDF). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ion FPs
pdfjs inserts a single spurious space before punctuation from glyph advances
("Sumber :", "sendiri ." in captions), which is indistinguishable from a real
single-space typo on extracted text. Only flag a 2+ space gap — a genuine
spacing anomaly. Document-agnostic (no word whitelist); verified zero
space-before-punct findings on thesis_example_1/2/3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make eyd.foreign-not-italic findings eligible (keyed on the rule, no word list). The rebuild path emits the flagged word as an italic run, tracking italic ranges through text-replacement offset shifts. The patch path lists them in the change log to italicize by hand, rather than risk splitting runs in the student's own .docx. Unchecked by default (foreign detection has FPs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Show foreign-word italic recommendations as their own unchecked group, rendered as the word in italics rather than a text swap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a "Kecualikan halaman" input to the report filter row that hides findings on user-specified pages (e.g. "7, 10-12"). View-only session filter that feeds the existing ParsedFilter pipeline, so both lists, their counts, and the bulk-resolve action respect it. Shows how many findings are hidden with a clear button. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request introduces several enhancements to the CiteTrack codebase, including new scripts for KBBI data management, browser E2E testing runbooks, and improved configuration for API lookups. It also adds robust error handling and logging for third-party API calls, updates the documentation, and introduces a new React Doctor skill. My review identified two technical issues: a potential CSV parsing error due to line endings in the KBBI build script and a shell portability issue with ANSI-C quoting in the verification script. Both have been addressed with suggested fixes.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| 'https://raw.githubusercontent.com/aryakdaniswara/kbbi-dataset-kbbi-v/HEAD/csv/kbbi_v.csv' | ||
| const raw = await fetchText(url) | ||
| const lines = raw.split('\n') | ||
| const headerCells = parseCsvLine(lines[0]) |
There was a problem hiding this comment.
If the fetched CSV file has CRLF (\\r\\n) line endings, lines[0] will end with \\r. Since parseCsvLine splits by ,, the last cell will contain \\r (e.g., 'submakna\\r'). This causes headerCells.indexOf('submakna') to return -1, throwing an error and crashing the script.
To prevent this, trim each cell in headerCells using .map(c => c.trim()) to ensure robust parsing regardless of line endings.
| const headerCells = parseCsvLine(lines[0]) | |
| const headerCells = parseCsvLine(lines[0]).map((c) => c.trim()) |
| const raw = execSync( | ||
| `docker compose exec -T db psql -U postgres -d citetrack -tA -F $'\\t' -c "${sql}"`, | ||
| { encoding: 'utf8' }, | ||
| ) |
There was a problem hiding this comment.
The bash-specific ANSI-C quoting $'\\t' is used to pass a tab character to psql. However, execSync runs commands using /bin/sh by default, which on many systems (like Ubuntu/Debian) is dash and does not support $'...' syntax. This will cause psql to receive the literal string $\\t or \\t as the field separator, causing parsing or execution failures.
To make this portable across all shells, use a portable JS tab escape \\t inside the template literal, which expands to a literal tab character in the shell command.
| const raw = execSync( | |
| `docker compose exec -T db psql -U postgres -d citetrack -tA -F $'\\t' -c "${sql}"`, | |
| { encoding: 'utf8' }, | |
| ) | |
| const raw = execSync( | |
| `docker compose exec -T db psql -U postgres -d citetrack -tA -F '\\t' -c "${sql}"`, | |
| { encoding: 'utf8' }, | |
| ) |
No description provided.