Batch stream recovery closes the shared connection and can leave the client permanently closed

[jfrancoa]

Summary

The batch-stream recovery path closes the connection that is shared by every thread of the client and, when its 5 reconnect attempts fail, leaves the whole client permanently closed. After that:

• every query raises WeaviateClosedClientError, and
• every new collection.batch.stream() blocks for 60 seconds and then raises the misleading Batch stream was not started within 60 seconds. Please check your connection.

Nothing inside the client ever repairs this; the process needs a new client object. Observed on 4.23.0; the code is unchanged on current main.

How we hit it

A load generator (5 threads, each inserting through its own collection.batch.stream() context, sync client) ran against a 5-node Weaviate 1.39.0 cluster. One server node crashed mid-stream (server bug, crash-restarted). The client never recovered: it spent hours in the state above, while a freshly created client in the same pod connected and started a batch stream instantly over the same endpoints.

py-spy dump of the stuck process: all 5 worker threads sit in _start (sync.py) polling __all_threads_alive() until the 60 s timeout; no batch background threads and no connection event-loop thread exist any more.

The defects

All in weaviate/collections/batch/sync.py (permalinks to main @ 15e4aeb):

1. __reconnect() closes the shared connection. sync.py#L466-L490 calls result(self.__connection.close("sync")) then self.__connection.connect(force=True). self.__connection is the client-wide connection: while one stream recovers, every other thread's operations fail with WeaviateClosedClientError.

2. Concurrent recoveries are uncoordinated. Each active batch stream runs its own recv_wrapper (sync.py#L499-L522); a node crash hangs up several streams at once, so several threads interleave close("sync") / connect(force=True) on the same connection. One thread's close races another thread's just-completed reconnect.

3. Permanent give-up. After 5 failed attempts __reconnect sets self.__bg_exception = e and returns (sync.py#L488-L490) — with the connection left closed. No later operation reopens it.

4. _start hides the real error. sync.py#L123-L132 polls __all_threads_alive() for 60 s and never checks __bg_exception. On a closed connection the bg threads die immediately, so every batch.stream() enter burns 60 s and then reports a connection hint instead of the stored exception.

Reproduction sketch

1. Cluster with ≥2 nodes, any collection.
2. N threads, each looping with collection.batch.stream() as b: b.add_object(...) on one shared client.
3. kill -9 the weaviate node the gRPC channel is connected to (crash, not graceful shutdown — graceful takes the shutting_down path instead).
4. Depending on the race outcome the client either recovers or enters the permanent state above; repeat a few crashes to hit it. Once wedged: queries raise WeaviateClosedClientError, batch streams time out after 60 s each, forever.

Expected behavior

• A batch stream's recovery must not tear down the connection other threads are using, or must coordinate so that close/connect pairs cannot interleave.
• _start should raise __bg_exception as soon as the background threads die instead of waiting 60 s and blaming the network.
• If recovery gives up, the client should either keep retrying with backoff or fail every subsequent call fast with the stored root cause — not a mix of 60 s timeouts and closed-client errors.

Environment

• weaviate-python-client 4.23.0 (sync API), python 3.10
• weaviate server 1.39.0, 5 nodes, gRPC through a TLS endpoint