Add incremental backup support by dudanogueira · Pull Request #459 · weaviate/typescript-client

dudanogueira · 2026-08-03T21:34:08Z

Exposes Weaviate's file-based incremental backups (server v1.37.0+) in the client:

const base = await client.backup.create({
  backupId: 'base-backup',
  backend: 'filesystem',
  waitForCompletion: true,
});

await client.backup.create({
  backupId: 'incremental-backup',
  backend: 'filesystem',
  incrementalBaseBackupId: base.id, // or any existing backup ID string
  waitForCompletion: true,
});

Unchanged files are not copied — they are restored from the base. Deleting a base backup therefore breaks every incremental built on top of it.

Changes

incrementalBaseBackupId on client.backup.create() and collection.backup.create(), plus .withIncrementalBaseBackupId() on the v2 builder.
Gated client-side by DbVersionSupport.supportsIncrementalBackups() — throws WeaviateUnsupportedFeatureError below 1.37.0, matching the Python client. The gate applies to the v2 builder too: Weaviate below 1.37.0 silently ignores the field and writes a full backup, which is worse than an error.
Validates that the base ID differs from the backup being created (the server otherwise returns a bare 500). The comparison is case-insensitive, and the ID is lowercased before it goes on the wire, because Weaviate treats backup IDs as case-insensitive.
The base backup ID is read back on getCreateStatus() and list().
parseStatus() now also surfaces size, which was declared on BackupStatusReturn but never populated.

When `incrementalBaseBackupId` comes back as `undefined`

This is a read-back with a narrower floor than the feature itself, so it is undefined in more cases than you might expect:

the backup is not incremental;
the caller is not an RBAC root user (Weaviate gates the field on isRequestFromRootUser);
the server is older than v1.37.6 — the create-side field landed much earlier, but the status/list read-back was added in 7a53058, whose earliest tag is v1.37.6;
the call is create() without waitForCompletion, since BackupCreateResponse carries no such field at all.

All four are documented on the type.

Notes for reviewers

backup(connection) now takes dbVersionSupport as a second argument — this applies to the v3 factory, backupCollection, and the v2 factory in src/backup/index.ts. All three are internal; only types are re-exported publicly. BackupCreator's new constructor argument is optional, so the public class stays source-compatible.
list() no longer returns the raw payload verbatim: incremental_base_backup_id is mapped to incrementalBaseBackupId. Everything else is passed through unchanged.
ci/docker-compose-rbac.yml gains backup-filesystem. It is the only CI instance with a root user (AUTHORIZATION_ADMIN_USERS populates the RBAC root-user list), and ci/docker-compose-backup.yml is anonymous — so the root-only read-back cannot be asserted anywhere else.

Testing

Mock tests cover the request payload, lowercasing, case-insensitive self-reference rejection, the collection-scoped path, the v2 builder, the version gate (1.36 vs 1.37) and response parsing including size.

Integration tests cover two instances:

the anonymous backup instance, which creates an incremental on top of a base, restores it, and asserts incrementalBaseBackupId is undefined — the honest contract for a non-root caller;
the RBAC instance, gated on 1.37.6, which asserts the base backup ID round-trips through create, getCreateStatus and list as a root user.

Verified against Weaviate 1.38.0 and 1.38.2.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Gc9nJTnKeYs41fidZAYWf6

Adds .withIncrementalBaseBackupId() to BackupCreator so the incremental_base_backup_id field is sent on backup creation, plus validation rejecting an empty base ID or one equal to the backup being created.

client.backup.create() and collection.backup.create() now accept incrementalBaseBackupId. The request is gated client-side on Weaviate >=1.37.0 via DbVersionSupport.supportsIncrementalBackups() and throws WeaviateUnsupportedFeatureError on older servers. The base backup ID is also surfaced on getCreateStatus() and list(), which Weaviate only returns to root users.

Mock tests assert the payload sent on create, the lowercasing of the base ID, the >=1.37.0 gate and the parsing of the base ID in list() and getCreateStatus(). Integration tests create an incremental backup on top of a base backup and restore it.

orca-security-eu

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0 0 0 0	View in Orca
Passed	SAST	0 0 0 0	View in Orca
Passed	Secrets	0 0 0 0	View in Orca
Passed	Vulnerabilities	0 0 0 0	View in Orca

g-despot

Tested this locally against 1.35.0, 1.36.10 and 1.38.9.

The feature works. I confirmed the incremental genuinely skips base segments on disk (1.7 MB vs 35 MB uncompressed, with the objects and property_text segments absent), and that restoring the incremental fails without the base present and succeeds with it.

Two things I'd like sorted before merge:

The 1.37.0 floor is too high. 1.36.10 accepts incremental_base_backup_id and produces a real incremental (345 KB vs 7.5 MB base, baseBackupId persisted in backup_config.json), but the client throws WeaviateUnsupportedFeatureError against it. A tag scan puts create at 1.35.13 and 1.36.3. Below that I checked 1.35.0, where the server silently ignores the field and you get a full backup, so the gate is worth keeping, just lower.
The read-back is a separate feature with its own floor. incremental_base_backup_id only comes back on status/list from ~1.37.6, and only for root users. On 1.37.0 to 1.37.5 we'd be promising a field the server never sends.

Rest inline. Nice work otherwise, the design is clean.

- Gate the v2 builder on the server version too. `weaviateV2` is public API and Weaviate below 1.37.0 silently writes a full backup when it sees `incremental_base_backup_id`, so the builder now throws WeaviateUnsupportedFeatureError instead. - Lowercase the base backup ID in `withIncrementalBaseBackupId` and compare IDs case-insensitively in `validateIncrementalBaseBackupId`, so `backupId: 'B1'` with base `'b1'` is rejected on both the v2 and v3 paths. - Surface `size` from `parseStatus`; it was declared on BackupStatusReturn but never populated. - Document that `incrementalBaseBackupId` is undefined for non-root callers, on servers below 1.37.6 (when the read-back was added), and on a create without `waitForCompletion`. - Collapse the duplicated `incrementalBaseBackupId` doc block to one line and derive BackupCollectionCreateArgs from BackupCreateArgs. - Make the integration coverage honest: assert the field is undefined on the anonymous instance, and add a root-user round-trip against the RBAC instance, which now enables backup-filesystem. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F65TAaZWcf7unm86QQ3FA8

dudanogueira · 2026-08-14T16:05:44Z

Pushed the review fixes in f493925. Summary of what changed per thread is inline; the PR description is rewritten too.

CI

Two legs are red, and I want to be upfront that I looked into whether I caused them. I do not believe I did:

tests-without-auth (22.x, 1.38) — collection.query > boost parameter > 'timeDecay', an ordering assertion. Unrelated to backups.
tests-without-auth (24.x, 1.37) — two test/users/integration.test.ts role assign/revoke 404s, plus a deleteMany "Stringified UUID is invalid".

The two users failures are a known flake: the identical pair failed on main on 2026-08-13 (run 31690118238, on the 1.34 leg), with no backup code in play.

The one leg that actually exercises the new root-user suite is tests-without-auth (24.x, 1.38), because requireAtLeast(1, 37, 6) skips it everywhere else and CI's WEAVIATE_137 is 1.37.5. That leg passed.

The only way my change could plausibly touch the users/roles suites is the new backup-filesystem module on ci/docker-compose-rbac.yml, so I ran both suites locally against that exact modified compose on 1.38.2: 39/39 pass.

I do not have rerun rights on this repo — if someone with admin could kick the two failed jobs, that should close it out.

bevzzz

"Incremental backup support" is 1 string field in the request payload and 1 string field in the response payload. It is surprising to see that the PR is over 500LOC.

We want to have a test or two to check that the values reach the server, which will add another 100-150 lines. But that's about it. Adding 300 lines of redundant comments / tests with each PR is simply unsustainable and it makes the PR that much more difficult to review.

Please take care to minimize the size of your contributions to changes that are strictly necessary.

bevzzz · 2026-08-17T10:13:58Z

+
+const BASE_BACKUP_ID = 'test-backup-base';
+
+/** Mocks the backup endpoints, recording the payload sent by the client on creation. */
+class IncrementalMock {
+  private grpc: GrpcServer;
+  private http: HttpServer;
+  static lastCreateRequest: BackupCreateRequest;
+
+  constructor(grpc: GrpcServer, http: HttpServer) {
+    this.grpc = grpc;
+    this.http = http;
+  }
+
+  public static use = async (version: string, httpPort: number, grpcPort: number) => {
+    const httpApp = express();
+    httpApp.use(express.json());
+    httpApp.get('/v1/meta', (req, res) => res.send({ version }));
+
+    httpApp.post(`/v1/backups/${BACKEND}`, (req, res: Response<BackupCreateResponse, any>) => {
+      IncrementalMock.lastCreateRequest = req.body;
+      res.send({
+        id: req.body.id,
+        backend: BACKEND,
+        classes: ['Article'],
+        path: 'path/to/backup',
+        status: 'STARTED',
+      });
+    });
+    httpApp.get(`/v1/backups/${BACKEND}/:id`, (req, res: Response<BackupCreateStatusResponse, any>) =>
+      res.send({
+        id: req.params.id,
+        backend: BACKEND,
+        path: 'path/to/backup',
+        status: 'SUCCESS',
+        size: 1.5,
+        incremental_base_backup_id: IncrementalMock.lastCreateRequest?.incremental_base_backup_id,
+      })
+    );
+    httpApp.get(`/v1/backups/${BACKEND}`, (req, res: Response<BackupListResponse, any>) =>
+      res.send([
+        { id: BASE_BACKUP_ID, classes: ['Article'], status: 'SUCCESS', incremental_base_backup_id: '' },
+        {
+          id: BACKUP_ID,
+          classes: ['Article'],
+          status: 'SUCCESS',
+          incremental_base_backup_id: BASE_BACKUP_ID,
+        },
+      ])
+    );
+
+    const healthMockImpl: HealthServiceImplementation = {
+      check: (request: HealthCheckRequest): Promise<HealthCheckResponse> =>
+        Promise.resolve(HealthCheckResponse.create({ status: HealthCheckResponse_ServingStatus.SERVING })),
+      watch: vi.fn(),
+    };
+
+    const grpc = createServer();
+    grpc.add(HealthDefinition, healthMockImpl);
+
+    httpApp.on('error', (error) => console.error('HTTP Server Error:', error));
+
+    await grpc.listen(`localhost:${grpcPort}`);
+    const http = await httpApp.listen(httpPort);
+    return new IncrementalMock(grpc, http);
+  };
+
+  public close = () => Promise.all([this.http.close(), this.grpc.shutdown()]);
+}
+
+describe('Mock testing of incremental backups', () => {
+  describe('with a supported Weaviate version', () => {
+    let client: WeaviateClient;
+    let mock: IncrementalMock;
+
+    beforeAll(async () => {
+      mock = await IncrementalMock.use('1.37.0', 8914, 8915);
+      client = await weaviate.connectToLocal({ port: 8914, grpcPort: 8915 });
+    });
+
+    it('should send the base backup ID when creating an incremental backup', async () => {
+      const res = await client.backup.create({
+        backupId: BACKUP_ID,
+        backend: BACKEND,
+        incrementalBaseBackupId: BASE_BACKUP_ID,
+        waitForCompletion: true,
+      });
+      expect(IncrementalMock.lastCreateRequest.incremental_base_backup_id).toBe(BASE_BACKUP_ID);
+      expect(res.status).toBe('SUCCESS');
+      expect(res.incrementalBaseBackupId).toBe(BASE_BACKUP_ID);
+    });
+
+    it('should lowercase the base backup ID', async () => {
+      await client.backup.create({
+        backupId: BACKUP_ID,
+        backend: BACKEND,
+        incrementalBaseBackupId: 'Test-Backup-BASE',
+        waitForCompletion: true,
+      });
+      expect(IncrementalMock.lastCreateRequest.incremental_base_backup_id).toBe(BASE_BACKUP_ID);
+    });
+
+    it('should not send the field for a regular backup', async () => {
+      await client.backup.create({ backupId: BACKUP_ID, backend: BACKEND });
+      expect(IncrementalMock.lastCreateRequest.incremental_base_backup_id).toBeUndefined();
+    });
+
+    it('should throw if the base backup is the backup being created', async () => {
+      const promise = client.backup.create({
+        backupId: BACKUP_ID,
+        backend: BACKEND,
+        incrementalBaseBackupId: BACKUP_ID,
+      });
+      await expect(promise).rejects.toThrow(WeaviateInvalidInputError);
+    });
+
+    it('should throw if the base backup only differs from the backup being created in case', async () => {
+      // Weaviate lowercases backup IDs, so these name the same backup.
+      const promise = client.backup.create({
+        backupId: BACKUP_ID.toUpperCase(),
+        backend: BACKEND,
+        incrementalBaseBackupId: BACKUP_ID,
+      });
+      await expect(promise).rejects.toThrow(WeaviateInvalidInputError);
+    });
+
+    it('should send the base backup ID when creating a collection-scoped backup', async () => {
+      await client.collections.use('Article').backup.create({
+        backupId: BACKUP_ID,
+        backend: BACKEND,
+        incrementalBaseBackupId: BASE_BACKUP_ID,
+        waitForCompletion: true,
+      });
+      expect(IncrementalMock.lastCreateRequest.incremental_base_backup_id).toBe(BASE_BACKUP_ID);
+      expect(IncrementalMock.lastCreateRequest.include).toEqual(['Article']);
+    });
+
+    it('should surface the base backup ID when listing backups', async () => {
+      const backups = await client.backup.list(BACKEND);
+      expect(backups[0].incrementalBaseBackupId).toBeUndefined();
+      expect(backups[1].incrementalBaseBackupId).toBe(BASE_BACKUP_ID);
+    });
+
+    it('should surface the base backup ID when getting the creation status', async () => {
+      await client.backup.create({ backupId: BACKUP_ID, backend: BACKEND }); // resets the recorded payload
+      const regular = await client.backup.getCreateStatus({ backupId: BACKUP_ID, backend: BACKEND });
+      expect(regular.incrementalBaseBackupId).toBeUndefined();
+
+      await client.backup.create({
+        backupId: BACKUP_ID,
+        backend: BACKEND,
+        incrementalBaseBackupId: BASE_BACKUP_ID,
+      });
+      const incremental = await client.backup.getCreateStatus({ backupId: BACKUP_ID, backend: BACKEND });
+      expect(incremental.incrementalBaseBackupId).toBe(BASE_BACKUP_ID);
+    });
+
+    it('should surface the backup size reported by Weaviate', async () => {
+      const status = await client.backup.getCreateStatus({ backupId: BACKUP_ID, backend: BACKEND });
+      expect(status.size).toBe(1.5);
+    });
+
+    afterAll(() => mock.close());
+  });
+
+  describe('with the v2 builder', () => {
+    let mock: IncrementalMock;
+    const clientV2 = weaviateV2.client({ scheme: 'http', host: 'localhost:8918' });
+
+    beforeAll(async () => {
+      mock = await IncrementalMock.use('1.37.0', 8918, 8919);
+    });
+
+    it('should lowercase the base backup ID', async () => {
+      await clientV2.backup
+        .creator()
+        .withBackupId(BACKUP_ID)
+        .withBackend(BACKEND)
+        .withIncrementalBaseBackupId('Test-Backup-BASE')
+        .do();
+      expect(IncrementalMock.lastCreateRequest.incremental_base_backup_id).toBe(BASE_BACKUP_ID);
+    });
+
+    it('should throw if the base backup only differs from the backup being created in case', async () => {
+      const promise = clientV2.backup
+        .creator()
+        .withBackupId(BACKUP_ID.toUpperCase())
+        .withBackend(BACKEND)
+        .withIncrementalBaseBackupId(BACKUP_ID)
+        .do();
+      await expect(promise).rejects.toThrow(WeaviateInvalidInputError);
+    });
+
+    afterAll(() => mock.close());
+  });
+
+  describe('with an unsupported Weaviate version', () => {
+    let client: WeaviateClient;
+    let mock: IncrementalMock;
+
+    beforeAll(async () => {
+      mock = await IncrementalMock.use('1.36.0', 8916, 8917);
+      client = await weaviate.connectToLocal({ port: 8916, grpcPort: 8917 });
+    });
+
+    it('should throw when requesting an incremental backup', async () => {
+      const promise = client.backup.create({
+        backupId: BACKUP_ID,
+        backend: BACKEND,
+        incrementalBaseBackupId: BASE_BACKUP_ID,
+      });
+      await expect(promise).rejects.toThrow(WeaviateUnsupportedFeatureError);
+    });
+
+    it('should still allow regular backups', async () => {
+      const res = await client.backup.create({ backupId: BACKUP_ID, backend: BACKEND });
+      expect(res.status).toBe('STARTED');
+    });
+
+    it('should throw from the v2 builder too', async () => {
+      const promise = weaviateV2
+        .client({ scheme: 'http', host: 'localhost:8916' })
+        .backup.creator()
+        .withBackupId(BACKUP_ID)
+        .withBackend(BACKEND)
+        .withIncrementalBaseBackupId(BASE_BACKUP_ID)
+        .do();
+      await expect(promise).rejects.toThrow(WeaviateUnsupportedFeatureError);
+    });
+
+    afterAll(() => mock.close());
+  });
+});


This PR already adds e2e test for this feature, so I really don't think there is a need for 250 lines of mock tests. Could we please remove these altogether?

bevzzz · 2026-08-17T10:17:48Z

+  // Weaviate treats backup IDs as case-insensitive, so 'B1' and 'b1' name the same backup.
+  if (incrementalBaseBackupId.toLowerCase() === backupId?.toLowerCase()) {
+    return ['incrementalBaseBackupId must be different from the ID of the backup being created'];
+  }
+  return [];


issue: This validation should not be in the client, it is entirely up to the server to resolve backup IDs and allow/forbid them.

bevzzz · 2026-08-17T10:21:31Z

+      // Restore responses carry neither of these fields; see BackupStatusReturn for when Weaviate
+      // omits `incremental_base_backup_id` from a create status.
+      size: 'size' in res ? res.size : undefined,
+      incrementalBaseBackupId:
+        'incremental_base_backup_id' in res ? res.incremental_base_backup_id || undefined : undefined,
    };


issue: these checks are entirely unnecessary. If a key is not present in the res object then the expression already invalidates to undefined.

See playground example.

bevzzz · 2026-08-17T10:24:01Z

+/** The arguments required to create a backup. */
+export type BackupCreateArgs = BackupArgs<BackupConfigCreate> & {
+  /** The ID of an existing backup to build a file-based incremental backup on. Files identical to the base are not copied and are restored from the base instead, so deleting a base backup breaks every incremental built on it. Requires Weaviate `v1.37.0` or higher. */
+  incrementalBaseBackupId?: string;
+};
+


issue: why is it necessary to declare a new type? BackupConfigCreate can just be extended with incrementalBaseBackupId.

bevzzz · 2026-08-17T10:26:33Z

+      // This instance is anonymous, so the caller is never a root user and Weaviate withholds the
+      // base backup ID. The root-user round-trip is covered by the RBAC suite below.


nit: please remove comment slop

bevzzz · 2026-08-17T10:28:36Z

+// Weaviate only returns incremental_base_backup_id to root users, and only from v1.37.6 onwards.
+// ci/docker-compose-rbac.yml is the only instance with a root user (AUTHORIZATION_ADMIN_USERS
+// populates the RBAC root user list), so the read-back can only be asserted there.
+requireAtLeast(1, 37, 6).describe('Integration testing of incremental backups as a root user', () => {
+  const clientPromise = weaviate.connectToLocal({
+    port: 8091,
+    grpcPort: 50062,
+    authCredentials: new ApiKey('admin-key'),
+  });
+
+  const collectionName = 'TestIncrementalBackupRoot';
+  const randomBackupId = () => 'backup-id-' + Math.floor(Math.random() * Number.MAX_SAFE_INTEGER);
+
+  afterAll(() => clientPromise.then((client) => client.collections.delete(collectionName)));
+
+  it('reports the base backup ID on create, status and list', async () => {
+    const client = await clientPromise;
+    const collection = await client.collections
+      .create({ name: collectionName })
+      .then((col) => col.data.insert().then(() => col));
+
+    const base = await client.backup.create({
+      backupId: randomBackupId(),
+      backend: 'filesystem',
+      includeCollections: [collection.name],
+      waitForCompletion: true,
+    });
+    expect(base.status).toBe('SUCCESS');
+    expect(base.incrementalBaseBackupId).toBeUndefined();
+
+    await collection.data.insert();
+
+    const incremental = await client.backup.create({
+      backupId: randomBackupId(),
+      backend: 'filesystem',
+      includeCollections: [collection.name],
+      incrementalBaseBackupId: base.id,
+      waitForCompletion: true,
+    });
+    expect(incremental.status).toBe('SUCCESS');
+    expect(incremental.incrementalBaseBackupId).toBe(base.id);
+
+    const status = await client.backup.getCreateStatus({
+      backupId: incremental.id,
+      backend: 'filesystem',
+    });
+    expect(status.incrementalBaseBackupId).toBe(base.id);
+
+    const listed = await client.backup.list('filesystem');
+    expect(listed.find((b) => b.id === incremental.id)?.incrementalBaseBackupId).toBe(base.id);
+    expect(listed.find((b) => b.id === base.id)?.incrementalBaseBackupId).toBeUndefined();
+  });
+});


issue: Weaviate only returns incremental_base_backup_id to root users, and only from v1.37.6 onwards is a property of Weaviate server and not of the client. Please remove this test.

bevzzz · 2026-08-17T10:32:43Z

+      # backup-filesystem is enabled here so that backup behaviour which Weaviate only exposes to
+      # root users (e.g. incremental_base_backup_id) can be tested; ci/docker-compose-backup.yml
+      # is anonymous and therefore never has a root caller.
+      ENABLE_MODULES: "generative-dummy,reranker-dummy,backup-filesystem"
+      BACKUP_FILESYSTEM_PATH: "/tmp/backups"


backup-filesystem is enabled here so that backup behaviour which Weaviate only exposes root users (e.g. incremental_base_backup_id) can be tested

This is not an acceptance test suite for the Weaviate server and is not aimed at testing the server's behaviour. Please revert this change and the associated describe block in test/collections/backup/integration.test.ts

dudanogueira added 3 commits August 3, 2026 18:27

feat(backup): support incremental base backup ID in the v2 creator

0ab941e

Adds .withIncrementalBaseBackupId() to BackupCreator so the incremental_base_backup_id field is sent on backup creation, plus validation rejecting an empty base ID or one equal to the backup being created.

orca-security-eu Bot reviewed Aug 3, 2026

View reviewed changes

g-despot requested changes Aug 14, 2026

View reviewed changes

dudanogueira requested a review from a team as a code owner August 14, 2026 15:55

bevzzz requested changes Aug 17, 2026

View reviewed changes

		// This instance is anonymous, so the caller is never a root user and Weaviate withholds the
		// base backup ID. The root-user round-trip is covered by the RBAC suite below.

Uh oh!

Conversation

dudanogueira commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

When incrementalBaseBackupId comes back as undefined

Notes for reviewers

Testing

Uh oh!

orca-security-eu Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Orca Security Scan Summary

Uh oh!

g-despot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dudanogueira commented Aug 14, 2026

CI

Uh oh!

bevzzz left a comment

Choose a reason for hiding this comment

Uh oh!

bevzzz Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

bevzzz Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

bevzzz Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

bevzzz Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

bevzzz Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

bevzzz Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

bevzzz Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dudanogueira commented Aug 3, 2026 •

edited

Loading

When `incrementalBaseBackupId` comes back as `undefined`

orca-security-eu Bot left a comment •

edited

Loading