wPerf: Generic Off-CPU Analysis to Identify Bottleneck Waiting Events

Notice

This repo has been deprecated and will be reimplemented using eBPF. New repo will be updated later.

Intro

wPerf is designed to identify bottlenecks caused by all kinds of waiting events. To identify waiting events that limit the application’s throughput, wPerf uses cascaded re-distribution to compute the local impact of a waiting event and uses wait-for graph to compute whether such impact can reach other threads.

Check the paper for more details: wPerf: Generic Off-CPU Analysis to Identify Bottleneck Waiting Events, OSDI 2018.

Requirements

• Kernel: Only tested on CentOS Linux release 7.5.1804 Please make sure the KProbe and CONFIG_SCHEDSTATS features are enabled.
• libuv Tested with v1.x

Deploy

git clone https://github.com/ethercflow/wPerf
cd wPerf
./deploy.sh

Tracing

Example: The target pid is 1234, which has three worker threads A,B,C, default the output dir is /tmp/wperf, record period is 90s.

Run

cd bin
sudo ./recorder.py -T A,B,C -p 1234
./encorder.py -i /tmp/wperf/softirq/output -o softirq
./encorder.py -i /tmp/wperf/switch/output -o switch
cp -a /tmp/wperf/cpufreq .
cp -a /tmp/wperf/pidlist .
./analyzer -cpufreq=cpufreq -pids=pidlist -switch=switch -softirq=softirq

After analyzer, you can get a waitfor file, open it to see if there is a knot. In the future, I'll graphs the results. Currently, you can update waitfor to online graph explorer with your result file and check out the bottleneck in that knot. (You'd better filter kworker info before upload)