perf: verify shuffle IPC context-reuse allocation savings with Arrow 59.1+

[peterxcli]

What is the problem the feature request solves?

PR #5038 reuses the Arrow IPC write context across shuffle blocks, but Comet currently pins Arrow 58.4.0. In that version, CompressionContext only owns an optional zstd compressor. Comet does not enable Arrow IPC buffer compression, and the FlatBufferBuilder is still allocated separately for every batch, so the merged lifetime change provides no allocation savings on current main.

Starting with Arrow 59.1, IpcWriteContext owns the reusable FlatBufferBuilder, making the context reuse introduced by #5038 effective.

The broader DataFusion/Arrow dependency upgrade is already tracked by #4865 and draft PR #5262. This issue tracks validating the deferred shuffle performance benefit after that upgrade lands.

Describe the potential solution

Close this issue when:

• Comet uses Arrow 59.1 or newer through a compatible DataFusion upgrade.
• The shuffle writer paths retain the per-writer and row-conversion-loop context lifetimes introduced by perf: reuse Arrow IPC compression context across shuffle blocks #5038 and use the Arrow 59.1+ write-context API.
• Focused, repeated same-machine measurements demonstrate fewer IPC metadata allocations on the production shuffle paths, with the results recorded.
• Existing shuffle IPC round-trip and writer tests pass.

Comet's outer per-block compression encoders are independent and outside this issue's scope.

Additional context

• Follow-up comment: perf: reuse Arrow IPC compression context across shuffle blocks #5038 (comment)
• Context-reuse implementation: perf: reuse Arrow IPC compression context across shuffle blocks #5038
• Benchmark evidence: perf: reuse Arrow IPC compression context across shuffle blocks #5038 (comment)
• Original implementation issue: perf: reuse Arrow IPC CompressionContext across shuffle blocks instead of per-batch allocation #5008
• DataFusion 55 tracker: chore: DataFusion 55.0.0 #4865
• Current upgrade PR: deps: [DO NOT MERGE] bump DataFusion 55.0 and Arrow/Parqet 59.2 #5262

[peterxcli]

take

[comphead]

Thanks @peterxcli love to investigate the results