Skip to content

Upgrade Harper to v5 and add integration tests - #8

Open
BboyAkers wants to merge 7 commits into
mainfrom
v5-upgrade
Open

Upgrade Harper to v5 and add integration tests#8
BboyAkers wants to merge 7 commits into
mainfrom
v5-upgrade

Conversation

@BboyAkers

Copy link
Copy Markdown

Summary

  • Dependency bump: added harper@^5.0.28 as an explicit dependency (was runtime-provided global). harperdb was never listed in package.json — this repo relied on the global being injected.
  • Migration fix: replaced non-standard table.update({ where, data }) call in the ModelFetchJobs retry handler with a v5-compatible table.put({ ...job, ...updates }) (frozen-record-safe spread).
  • Integration tests: added integrationTests/data-layer.test.ts using @harperfast/integration-testing@0.4.0. Tests cover the Harper data layer (Model, InferenceEvent, BenchmarkResult, Feature, ModelFetchJob table CRUD + REST resource endpoints + Status health check). AI inference endpoints (Predict/Personalize/Benchmark) require external model files, GPU, or running Ollama — these are exercised only for their data-path error handling in CI.
  • CI: added .github/workflows/integration-tests.yml with Node 22/24/26 matrix on ubuntu-latest. Sets MODEL_FETCH_WORKER=false to disable the background model-fetch worker during tests (avoids download attempts).
  • Lockfile: regenerated package-lock.json with --os=linux --cpu=x64 --include=optional to include bufferutil, utf-8-validate, and node-gyp-build for Linux CI.
  • TypeScript: added tsconfig.json targeting NodeNext with erasableSyntaxOnly for the integration test suite.
  • Branding: HarperDBHarper in README.md, CONTRIBUTING.md, ROADMAP.md, package.json description/author/keywords. Clone URL updated to HarperFast/edge-ai-ops.
  • Engines: bumped engines.node to >=22.0.0 (required by @harperfast/integration-testing).

Migration items applied

Item Status
harperdbharper import/dep N/A — runtime-provided global; no explicit import existed
wasLoadedFromSource() N/A — not used
blob.save() N/A — not used
Frozen records (no direct property mutation) Applied — table.update({where,data}) replaced with put({ ...job, ...updates })
Transaction context (getContext()) N/A — no explicit transaction usage
Table.get() return shape N/A — .wasLoadedFromSource() not called
Process spawning (allowedSpawnCommands) N/A — no child_process spawn calls

AI inference note

TensorFlow.js, ONNX Runtime, Transformers.js, and Ollama backends require external model files (hundreds of MB) or a running Ollama service. These are not available in CI. The integration tests cover the Harper data layer only, exercising:

  • Table CRUD (Model, InferenceEvent, Feature, BenchmarkResult, ModelFetchJob)
  • Custom resource endpoints (Status, Predict error path, WorkerControl, ModelList)

Known issues / flagged items

  • npm scope: package.json is named harper-edge-ai-example (unscoped). If this should move to the @harperfast/ npm scope, that migration is manual — flagged for human review per org process §11.1.
  • Local integration tests: will fail with EADDRNOTAVAIL on macOS (loopback aliases not configured). CI on ubuntu-latest is the test gate.
  • Lint warnings: pre-existing prettier/eslint warnings in scripts/ and examples/ are not introduced by this PR. ESLint is not blocking (errors are in scripts, not the Harper app code).

Test results

  • Local: blocked by EADDRNOTAVAIL (macOS loopback — environmental, not a code bug)
  • CI: running on push — see GitHub Actions for Node 22/24/26 results

🤖 Generated with Claude Code

- Add harper@5.0.28 as an explicit dependency (was runtime-provided global)
- Add @harperfast/integration-testing@0.4.0 + typescript devDeps
- Replace non-standard table.update({where,data}) with v5 put+spread in ModelFetchJobs retry handler
- Add integrationTests/data-layer.test.ts covering Harper table CRUD and REST resource endpoints (data layer only; AI inference skipped — requires external model files not available in CI)
- Add .github/workflows/integration-tests.yml with Node 22/24/26 matrix on ubuntu-latest
- Add tsconfig.json for TypeScript integration test compilation
- Regenerate package-lock.json with --os=linux --cpu=x64 --include=optional to include bufferutil/utf-8-validate/node-gyp-build for CI
- Update test:integration script to use harper-integration-test-run
- Bump engines.node to >=22.0.0 to match @harperfast/integration-testing requirement
- Branding: HarperDB → Harper in README, CONTRIBUTING.md, ROADMAP.md, package.json
- Fix repository URL and GitHub clone URL to HarperFast/edge-ai-ops

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the project for Harper v5, renaming references from HarperDB to Harper, bumping the minimum Node.js version to 22, adding a new integration test suite, and refactoring the model fetch job reset logic to avoid frozen-record mutations. Key feedback includes correcting a non-existent TypeScript version in package.json, utilizing native import.meta.resolve in tests, adding defensive checks for the test context, and strengthening test assertions for the /ModelList endpoint.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread package.json Outdated
Comment thread integrationTests/data-layer.test.ts Outdated
Comment thread integrationTests/data-layer.test.ts
Comment thread integrationTests/data-layer.test.ts
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Comment thread src/resources.js Outdated
Comment thread integrationTests/data-layer.test.ts
Comment thread integrationTests/data-layer.test.ts Outdated
Comment thread package.json Outdated
Comment thread .github/workflows/integration-tests.yml Outdated
- Coerce ISO-8601 date strings to integer timestamps before retry put in
  ModelFetchJobs.post, matching the pattern in ModelFetchWorker.updateJobStatus,
  to prevent FIFO sort failures and schema violations on Harper v5
- Guard teardownHarper in data-layer.test.ts after() so a startup failure
  surfaces the real HarperStartupError instead of a secondary TypeError
- Add test:integration:legacy script for tests/integration/**/*.test.js and
  include it in test:all to prevent those 9 files from being silently orphaned

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@BboyAkers

Copy link
Copy Markdown
Author

Review follow-up (autonomous agent): Fixed all 3 blocking findings: coerce ISO-8601 date strings to integer timestamps before retry put in ModelFetchJobs.post (matching ModelFetchWorker.updateJobStatus pattern); guard teardownHarper in after() to surface real startup errors; add test:integration:legacy script to include orphaned tests/integration/ files in CI.

BboyAkers and others added 4 commits August 10, 2026 12:35
Bump the harper dependency to ^5.2.1 and regenerate the lockfile.
Regenerated in full so the optional native deps (bufferutil,
utf-8-validate, segfault-handler) stay in the tree for Linux CI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous lockfile was generated with npm 11, which does not
auto-install the peer dependencies of an optional dependency. harper
5.2.1 pulls alasql, which optionally depends on react-native-fs, whose
peers (react-native, react) npm 12 installs and npm 11 does not. CI runs
npm 12 on Node 24/26, so npm ci failed there with those packages
"missing from lock file" while Node 22 (npm 11) passed.

Regenerated with npm 12 so the lockfile carries the full tree.
lockfileVersion stays 3; npm ci verified under both npm 11 and npm 12.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Use the native import.meta.resolve('harper') instead of a createRequire
  shim (package requires Node >=22, where it is stable).
- authFetch: throw a named error when ctx.harper.admin is missing, so a failed
  setupHarperWithFixture surfaces as a startup failure rather than as
  'Cannot read properties of undefined' in every test.
- GET /ModelList: assert the body is an array when the endpoint answers 200.
- Drop the unused deepStrictEqual import.
- Workflow: build the Node matrix into a variable and fail loudly if it ever
  resolves to []. An empty matrix would make fromJSON() yield zero jobs and
  report CI green having run no tests; the empty-NODE_VER path (every
  push/pull_request run) is now documented at the branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Migration item 1 (package/CLI rename) had been applied to the code and
package.json but not to the deploy tooling or docs, which still told users to
`npm install -g harperdb` and invoked `harperdb deploy/restart/status/logs/
system_information`. That binary does not exist in v5.

Renamed across deploy.sh, verify.sh, docs/DEPLOYMENT.md and docs/SCRIPTS.md
(15 sites). Deliberately untouched: the @harperdb/ npm scope, harperdb.io /
docs.harperdb.io URLs, and harperdb-config.yaml (v5 keeps that filename).
Shell syntax re-checked with bash -n.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant