Skip to content

[ML] Periodically report pytorch_inference RSS - #3160

Draft
edsavage wants to merge 2 commits into
elastic:mainfrom
edsavage:pytorch-periodic-rss-reporting
Draft

[ML] Periodically report pytorch_inference RSS#3160
edsavage wants to merge 2 commits into
elastic:mainfrom
edsavage:pytorch-periodic-rss-reporting

Conversation

@edsavage

Copy link
Copy Markdown
Contributor

Summary

Adds a periodic memory reporter to pytorch_inference so Elasticsearch can track the process's real resident set size (RSS) for a trained model deployment, independent of inference traffic.

  • A dedicated timer thread emits process stats every 10s (a single RSS read; negligible cost), so memory is observed even when a deployment is idle or between requests.
  • Each report carries both the current RSS (memory_rss) and the OS peak / high-water mark (memory_max_rss). The peak is transmitted explicitly rather than derived on the ES side from the sample stream, so transient spikes between reports are not lost — this is the signal used to keep model assignment and adaptive scaling OOM-safe.
  • The process-stats document key is renamed from process_stats to stats to match the Elasticsearch PyTorchResult / InferenceProcessStats parser.
  • The reporter thread is shut down promptly (condition-variable signalled) before the rest of the process tears down, so it can never write to a closing stream.

Relates #2885

Release-ordering note (important)

ES's InferenceProcessStats parser is strict and treats an unknown field as a fatal parse error for the result stream. Because this PR (re)introduces memory_max_rss, the Elasticsearch-side change that accepts memory_max_rss as an optional field must be released before the ml-cpp version emitting it is bumped into Elasticsearch. See the companion Elasticsearch PR. Merging this PR in the ml-cpp repo is safe on its own; the constraint applies to the ml-cpp version bump inside ES.

Test plan

  • CResultWriterTest.testWriteProcessStats updated for the stats schema + memory_max_rss; passes.
  • Manual: run pytorch_inference with a model and confirm periodic stats docs are emitted ~every 10s with memory_rss and memory_max_rss.
  • Confirm prompt shutdown (no stray writes / hangs on exit).

Made with Cursor

Emit the pytorch_inference resident set size on a fixed 10s
interval, independent of inference requests, reporting both the
current RSS (memory_rss) and the OS peak (memory_max_rss). This
lets Elasticsearch track real native memory use per trained
model deployment and keep assignment and adaptive scaling
OOM-safe rather than relying on an a priori estimate.

The process-stats field is renamed to "stats" to match the
Elasticsearch PyTorchResult parser.

Relates elastic#2885
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants