Add endpoint-keyed flush for batching targets

[royischoss]

Summary

Add an opt-in, endpoint-keyed flush fence for Storey batching targets. This lets MLRun wait until buffered and already-in-flight Parquet writes for one model endpoint are durable before acknowledging the event, without flushing or blocking unrelated endpoints.

Changes Made

• Add flush_key_field and public async flush(flush_key) support to batching targets.
• Track logical flush keys independently from physical partition keys, including mixed-key batches and endpoints spanning multiple partitions.
• Serialize same-key fences, shield writes from waiter cancellation, retain write failures, and coordinate outstanding writes with termination.
• Expose the keyed flush configuration on ParquetTarget, preserve monotonic last-written metadata, and reject unsupported single-file mode.
• Add deterministic coverage for buffered and in-flight writes, key isolation, concurrent fences, multiple partitions, failures, cancellation, timer/max-event races, and termination.

Testing

• make lint
• make test — 469 passed, 1 skipped
• pytest tests/test_flow.py -q — 275 passed

Related Issues

• Related to ML-12996

Additional Notes

• Keyed tracking is disabled by default, preserving existing target behavior.
• MLRun controller event semantics and retry policy remain outside Storey and will be implemented separately.

Made with Cursor