Develop an Nsight System plugin to profile network bandwidth - #999
Develop an Nsight System plugin to profile network bandwidth#999kingcrimsontianyu wants to merge 22 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
kingcrimsontianyu
added
improvement
Sample UIBelow is an example of the NIC bandwidth profile in nsys UI. 
|
Sample documentation
|
wence-
left a comment
wence-
left a comment
There was a problem hiding this comment.
flushing some small comments
| // Set from a signal handler for a clean shutdown. | ||
| volatile std::sig_atomic_t g_stop = 0; | ||
|
|
There was a problem hiding this comment.
I think this can be:
std::atomic_flag g_stop = ATOMIC_FLAG_INIT;
There was a problem hiding this comment.
Ref says the following. So I've left it default constructed.
This macro is no longer needed since default constructor of std::atomic_flag initializes it to clear state. It is kept for the compatibility with C.
There was a problem hiding this comment.
Great to learn about std::atomic_flag.
| // loop breaks and the process exits cleanly with code 0 instead of an abnormal termination. Also | ||
| // catch SIGINT to have the same clean exit for Ctrl-C. | ||
| extern "C" { | ||
| static void kvikio_nic_handle_signal(int /*signum*/) { g_stop = 1; } |
There was a problem hiding this comment.
And this:
g_stop.test_and_set();
There was a problem hiding this comment.
Done. Passed the relaxed memory ordering.
| auto prev_time = clock::now(); | ||
| auto next_deadline = prev_time; | ||
|
|
||
| while (stop == 0) { |
There was a problem hiding this comment.
Done, and passed the relaxed memory ordering.
|
@wence- Thanks! These are great suggestions. I didn't know the modern C++ way of doing signal handling, so this is a good learning experience. |
madsbk
left a comment
madsbk
left a comment
There was a problem hiding this comment.
LGTM, thanks @kingcrimsontianyu
| option(KvikIO_BUILD_EXAMPLES "Configure CMake to build examples" ON) | ||
| option(KvikIO_BUILD_TESTS "Configure CMake to build tests" ON) | ||
| option(KvikIO_REMOTE_SUPPORT "Configure CMake to build with remote IO support" ON) | ||
| option(KvikIO_BUILD_NSYS_PLUGIN "Configure CMake to build the Nsight Systems NIC plugin" ON) |
There was a problem hiding this comment.
Do we want to build this by default? I guess, yes?
There was a problem hiding this comment.
Yes. I hope to have it built by default
5 participants
When developing and optimizing remote I/O, a commonly asked question is: how do I know the achieved bandwidth of my application? While nsys does offer network-related features, limitations exist:
--nic-metrics: Requires OFED/InfiniBand dependency that cannot be fulfilled on many systems, includingg7e.8xlarge.--enable=network_interface: Shows raw counter instead of bandwidth value. Does not register signals and therefore always returns non-zero exit code.This PR develops an nsys plugin to allow flexible, generic NIC bandwidth profiling without the above limitations.