[ML] Periodically report pytorch_inference RSS - #3160
Draft
edsavage wants to merge 2 commits into
Draft
Conversation
Emit the pytorch_inference resident set size on a fixed 10s interval, independent of inference requests, reporting both the current RSS (memory_rss) and the OS peak (memory_max_rss). This lets Elasticsearch track real native memory use per trained model deployment and keep assignment and adaptive scaling OOM-safe rather than relying on an a priori estimate. The process-stats field is renamed to "stats" to match the Elasticsearch PyTorchResult parser. Relates elastic#2885
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a periodic memory reporter to
pytorch_inferenceso Elasticsearch can track the process's real resident set size (RSS) for a trained model deployment, independent of inference traffic.memory_rss) and the OS peak / high-water mark (memory_max_rss). The peak is transmitted explicitly rather than derived on the ES side from the sample stream, so transient spikes between reports are not lost — this is the signal used to keep model assignment and adaptive scaling OOM-safe.process_statstostatsto match the ElasticsearchPyTorchResult/InferenceProcessStatsparser.Relates #2885
Release-ordering note (important)
ES's
InferenceProcessStatsparser is strict and treats an unknown field as a fatal parse error for the result stream. Because this PR (re)introducesmemory_max_rss, the Elasticsearch-side change that acceptsmemory_max_rssas an optional field must be released before the ml-cpp version emitting it is bumped into Elasticsearch. See the companion Elasticsearch PR. Merging this PR in the ml-cpp repo is safe on its own; the constraint applies to the ml-cpp version bump inside ES.Test plan
CResultWriterTest.testWriteProcessStatsupdated for thestatsschema +memory_max_rss; passes.pytorch_inferencewith a model and confirm periodicstatsdocs are emitted ~every 10s withmemory_rssandmemory_max_rss.Made with Cursor