Currently cuDF is using cudaMemcpyFlagPreferOverlapWithCompute in its internal copies (source). This decision was made to ensure that SMs can be free for computation while copies occur on the copy engine. However, @felipeblazing ran an experiment that prompted me to investigate and I have discovered that this is not optimal in many cases.
As of CUDA 13.1 (see Release Notes), this flag tells the driver to use copy engines (CE) instead of SMs for copies. For optimal performance, we need to use cudaMemcpyFlagPreferOverlapWithCompute only for small transfers. Data I collected agrees with internal data from others: CE can be lower latency for small copies, and won't consume SMs. However, only SM copies can reach full memory bandwidth.
I ran a study on a few different systems that I have with CUDA 13.1 or newer, and wrote a brief technical report. https://gist.github.com/bdice/b448671fa0b9768fab22cc5725bd619c#file-report-md
The decision I made from that study is that we should use cudaMemcpyFlagPreferOverlapWithCompute only for copies 128 KiB or smaller.
We will need to fix this in cuDF, rapidsmpf, and RMM, since all these libraries use (or plan to use) similar wrappers for cudaMemcpyBatchAsync.
Currently cuDF is using
cudaMemcpyFlagPreferOverlapWithComputein its internal copies (source). This decision was made to ensure that SMs can be free for computation while copies occur on the copy engine. However, @felipeblazing ran an experiment that prompted me to investigate and I have discovered that this is not optimal in many cases.As of CUDA 13.1 (see Release Notes), this flag tells the driver to use copy engines (CE) instead of SMs for copies. For optimal performance, we need to use
cudaMemcpyFlagPreferOverlapWithComputeonly for small transfers. Data I collected agrees with internal data from others: CE can be lower latency for small copies, and won't consume SMs. However, only SM copies can reach full memory bandwidth.I ran a study on a few different systems that I have with CUDA 13.1 or newer, and wrote a brief technical report. https://gist.github.com/bdice/b448671fa0b9768fab22cc5725bd619c#file-report-md
The decision I made from that study is that we should use
cudaMemcpyFlagPreferOverlapWithComputeonly for copies 128 KiB or smaller.We will need to fix this in cuDF, rapidsmpf, and RMM, since all these libraries use (or plan to use) similar wrappers for
cudaMemcpyBatchAsync.