CMP 50

[olegk0]

I ran a test at CMP 50, but I don't quite understand how to interpret the results.

==> Using /usr/bin/nvcc                                                                                                                                                   
==> Detecting GPU 1 ...                                                                                                                                                   
    GPU: NVIDIA CMP 50HX  (sm_75, 56 SMs, 5120 KB L2)                                                                                                                     
==> Building benchmarks for sm_75 (this can take a minute) ...                                                                                                            
    build OK                                                                                                                                                              
========================================================================                                                                                                  
 gpu-micro-bench results                                                                                                                                                  
 gpu_name=NVIDIA CMP 50HX  arch=sm_75  n_sms=56  l2_kb=5120                                                                                                               
 gpu_id=1                                                                                                                                                                 
========================================================================                                                                                                  
                                                                                                                                                                          
==> [1/4] memtest (memory subsystem) ...                                                                                                                                  
===== BEGIN MEMTEST =====                                                                                                                                                 
GPU,NVIDIA CMP 50HX,sm_75,56,5120                                                                                                                                         
LAT_WARM,2048,20.96,119.77,20.82,,8.1                                                                                                                                     
LAT_WARM,4096,20.85,116.48,20.84,,7.9                                                                                                                                     
LAT_WARM,8192,20.92,117.75,20.88,,8.0                                                                                                                                     
LAT_WARM,16384,21.02,117.39,20.98,,8.0                                                                                                                                    
LAT_WARM,32768,21.26,117.27,21.25,,8.0                                                                                                                                    
LAT_WARM,65536,58.09,117.58,45.56,251.84,11.7                                                                                                                             
LAT_WARM,131072,117.43,117.43,83.84,246.61,17.2                                                                                                                           
LAT_WARM,262144,117.46,117.46,102.17,243.30,19.3                                                                                                                          
LAT_WARM,524288,117.43,117.41,111.63,242.46,22.3                                                                                                                          
LAT_WARM,1048576,117.46,117.46,114.65,241.33,27.4                                                                                                                         
LAT_WARM,2097152,117.43,117.45,116.62,241.09,42.8                                                                                                                         
LAT_WARM,4194304,117.39,117.77,117.48,241.33,85.8                                                                                                                         
LAT_WARM,8388608,240.78,240.80,240.42,240.85,268.2                                                                                                                        
LAT_WARM,16777216,240.75,240.76,240.63,240.80,536.4                                                                                                                       
LAT_WARM,33554432,240.77,240.76,240.72,240.79,1072.9
LAT_WARM,67108864,242.96,243.08,242.96,242.99,2165.8
LAT_WARM,134217728,244.16,244.15,244.20,244.16,4352.5
LAT_MULTI,2048,21.00,124.46,20.96,,3.3
LAT_MULTI,4096,21.00,123.28,21.05,,3.3
LAT_MULTI,8192,21.20,122.68,21.13,,3.3
LAT_MULTI,16384,21.50,122.39,21.45,,3.3
LAT_MULTI,32768,22.17,123.21,22.12,,3.4
LAT_MULTI,65536,72.28,122.86,48.13,137.89,5.2
LAT_MULTI,131072,122.86,122.90,88.46,133.27,7.4
LAT_MULTI,262144,122.86,122.85,107.72,129.02,8.4                                                                                                                          
LAT_MULTI,524288,122.88,122.88,117.17,128.34,9.9
LAT_MULTI,1048576,122.90,122.82,120.42,128.86,12.6
LAT_MULTI,2097152,122.82,122.89,121.82,128.16,17.8
LAT_MULTI,4194304,122.89,122.81,122.88,125.55,27.9
LAT_MULTI,8388608,234.82,233.99,237.88,233.72,90.7
LAT_MULTI,16777216,246.96,246.25,246.27,243.24,174.2
LAT_MULTI,33554432,247.17,247.13,247.19,245.73,336.9
LAT_MULTI,67108864,250.43,250.33,250.16,248.76,667.1
LAT_MULTI,134217728,251.55,251.41,251.60,249.95,1325.5
BW_WARM,65536,32.00,18.16,32.20,11.74,0.1
BW_WARM,131072,72.62,32.00,70.62,23.38,0.1 
BW_WARM,262144,141.98,75.29,136.53,46.18,0.1
BW_WARM,524288,251.67,271.26,256.00,83.68,0.1
BW_WARM,1048576,459.58,454.48,436.91,142.47,0.1
BW_WARM,2097152,685.52,682.67,468.45,225.05,0.2
BW_WARM,4194304,930.91,791.98,930.91,312.22,0.2
BW_WARM,8388608,421.66,440.73,466.45,396.83,0.7
BW_WARM,16777216,467.32,486.44,490.31,446.51,1.2
BW_WARM,33554432,496.56,499.75,498.51,473.01,2.4
BW_WARM,67108864,502.58,503.35,503.16,487.57,4.7
BW_WARM,134217728,495.62,495.14,495.36,489.01,9.5
BLOCK_COST,56,1,2.728,2.728,0.1
BLOCK_COST,112,2,2.771,1.385,0.1
BLOCK_COST,224,4,2.877,0.719,0.1
BLOCK_COST,448,8,3.248,0.406,0.2
BLOCK_COST,896,16,4.929,0.308,0.2
BLOCK_COST,1792,32,7.583,0.237,0.4
BLOCK_COST,3584,64,12.653,0.198,0.6
CONC_WARP,1,243.07,4.1140,12.2
CONC_WARP,2,243.42,8.2164,12.2
CONC_WARP,4,244.09,16.3874,12.2
CONC_WARP,8,245.97,32.5242,12.3
CONC_WARP,16,248.30,64.4370,12.4
CONC_WARP,32,253.22,126.3724,12.7
CONC_BLOCK,1,243.06,4.1142,12.2                                                                                                                                           
CONC_BLOCK,2,248.43,8.0506,12.4
CONC_BLOCK,4,248.77,16.0790,12.4
CONC_BLOCK,8,248.89,32.1430,12.4
CONC_BLOCK,16,248.98,64.2611,12.4
CONC_BLOCK,32,249.01,128.5086,12.5
CONC_BLOCK,56,249.79,224.1842,12.5
CONC_BLOCK,112,251.90,444.6285,12.6
CONC_BLOCK,224,259.07,864.6288,13.0
CONC_BLOCK,448,282.56,1585.5039,14.1
CONC_BLOCK,896,354.13,2530.1221,17.7
L2_PERSIST,65536,9.94,20.48,0.0
L2_PERSIST,131072,19.60,28.60,0.0
L2_PERSIST,262144,42.67,62.96,0.0
L2_PERSIST,524288,73.47,112.48,0.0
L2_PERSIST,1048576,117.87,271.31,0.0
L2_PERSIST,2097152,233.22,444.14,0.1
L2_PERSIST,4194304,280.67,707.22,0.1
L2_PERSIST,8388608,368.70,432.74,0.2
L2_PERSIST,16777216,441.69,469.61,0.4
L2_PERSIST,33554432,473.61,491.52,0.7
L2_PERSIST,67108864,487.26,492.23,1.4
L2_PERSIST,134217728,489.07,491.52,2.7
===== END MEMTEST =====

==> [2/4] arithmtest (instruction throughput/latency) ...                                                                                               09:39:07 [170/280]
===== BEGIN ARITHM =====
GPU,NVIDIA CMP 50HX,sm_75,56,5120
ARITH,f32_add,tput,100000,229916.8,0.287,3.4795,11.5
ARITH,f32_add,lat,100000,1736019.7,2.170,0.4608,86.8
ARITH,f32_sub,tput,100000,229450.9,0.287,3.4866,11.5
ARITH,f32_sub,lat,100000,1734665.6,2.168,0.4612,86.7
ARITH,f32_mul,tput,100000,229487.3,0.287,3.4860,11.5
ARITH,f32_mul,lat,100000,1734703.7,2.168,0.4612,86.7
ARITH,f32_fma,tput,100000,3311071.2,4.139,0.2416,165.6
ARITH,f32_fma,lat,100000,26463707.0,33.080,0.0302,1323.2
ARITH,f32_min,tput,100000,229480.3,0.287,3.4861,11.5
ARITH,f32_min,lat,100000,1734628.2,2.168,0.4612,86.7
ARITH,f32_max,tput,100000,229409.2,0.287,3.4872,11.5
ARITH,f32_max,lat,100000,1734719.4,2.168,0.4612,86.7
ARITH,f32_div_approx,tput,100000,84118.4,0.105,9.5104,4.2
ARITH,f32_div_approx,lat,100000,1734647.0,2.168,0.4612,86.7
ARITH,f32_div_rn,tput,100000,132312133.8,165.390,0.0060,6615.6
ARITH,f32_div_rn,lat,100000,132308914.2,165.386,0.0060,6615.4
ARITH,f32_neg,tput,100000,3697.3,0.005,216.3753,0.2
ARITH,f32_neg,lat,100000,3597.4,0.004,222.3804,0.2
ARITH,f32_abs,tput,100000,84089.0,0.105,9.5137,4.2
ARITH,f32_abs,lat,100000,84078.1,0.105,9.5150,4.2
ARITH,f32_rcp_approx,tput,100000,2096369.3,2.620,0.3816,104.8
ARITH,f32_rcp_approx,lat,100000,16334384.9,20.418,0.0490,816.7
ARITH,f32_rcp_rn,tput,100000,53047298.4,66.309,0.0151,2652.4
ARITH,f32_rcp_rn,lat,100000,53067836.8,66.335,0.0151,2653.4
ARITH,f32_sqrt_approx,tput,100000,2031885.6,2.540,0.3937,101.6
ARITH,f32_sqrt_approx,lat,100000,15946641.0,19.933,0.0502,797.3
ARITH,f32_sqrt_rn,tput,100000,52928165.4,66.160,0.0151,2646.4
ARITH,f32_sqrt_rn,lat,100000,52924007.4,66.155,0.0151,2646.2
ARITH,f32_rsqrt,tput,100000,2031783.3,2.540,0.3937,101.6
ARITH,f32_rsqrt,lat,100000,15946576.1,19.933,0.0502,797.3
ARITH,f32_sin,tput,100000,1088652.4,1.361,0.7349,54.4
ARITH,f32_sin,lat,100000,8685668.0,10.857,0.0921,434.3
ARITH,f32_cos,tput,100000,1088680.4,1.361,0.7348,54.4
ARITH,f32_cos,lat,100000,8685711.9,10.857,0.0921,434.3
ARITH,f32_lg2,tput,100000,2031839.8,2.540,0.3937,101.6
ARITH,f32_lg2,lat,100000,15951076.5,19.939,0.0502,797.6
ARITH,f32_ex2,tput,100000,1153266.4,1.442,0.6937,57.7
ARITH,f32_ex2,lat,100000,6579763.4,8.225,0.1216,329.0
ARITH,f32_tanh,tput,100000,778532.6,0.973,1.0276,38.9
ARITH,f32_tanh,lat,100000,6205090.5,7.756,0.1289,310.3
ARITH,f16x2_add,tput,100000,326401.9,0.408,2.4510,16.3
ARITH,f16x2_add,lat,100000,2535680.8,3.170,0.3155,126.8
ARITH,f16x2_sub,tput,100000,326399.4,0.408,2.4510,16.3
ARITH,f16x2_sub,lat,100000,2535735.4,3.170,0.3155,126.8
ARITH,f16x2_mul,tput,100000,326431.4,0.408,2.4507,16.3
ARITH,f16x2_mul,lat,100000,2535645.5,3.170,0.3155,126.8
ARITH,f16x2_fma,tput,100000,326397.5,0.408,2.4510,16.3
ARITH,f16x2_fma,lat,100000,2535764.2,3.170,0.3155,126.8
ARITH,f16x2_neg,tput,100000,84021.1,0.105,9.5214,4.2
ARITH,f16x2_neg,lat,100000,326303.4,0.408,2.4517,16.3
ARITH,f16x2_abs,tput,100000,84199.7,0.105,9.5012,4.2
ARITH,f16x2_abs,lat,100000,326215.7,0.408,2.4524,16.3
ARITH,f16x2_tanh,tput,100000,885132.1,1.106,0.9038,44.3
ARITH,f16x2_tanh,lat,100000,6838143.3,8.548,0.1170,341.9
ARITH,f16x2_ex2,tput,100000,885117.9,1.106,0.9038,44.3
ARITH,f16x2_ex2,lat,100000,6838189.1,8.548,0.1170,341.9
ARITH,f16_add,tput,100000,326330.2,0.408,2.4515,16.3
ARITH,f16_add,lat,100000,2535681.5,3.170,0.3155,126.8
ARITH,f16_sub,tput,100000,326316.8,0.408,2.4516,16.3
ARITH,f16_sub,lat,100000,2535727.3,3.170,0.3155,126.8
ARITH,f16_mul,tput,100000,326386.6,0.408,2.4511,16.3
ARITH,f16_mul,lat,100000,2535684.6,3.170,0.3155,126.8
ARITH,f16_fma,tput,100000,326304.0,0.408,2.4517,16.3
ARITH,f16_fma,lat,100000,2535640.0,3.170,0.3155,126.8
ARITH,f16_neg,tput,100000,3706.2,0.005,215.8522,0.2
ARITH,f16_neg,lat,100000,3728.6,0.005,214.5554,0.2
ARITH,f16_abs,tput,100000,84087.0,0.105,9.5140,4.2
ARITH,f16_abs,lat,100000,84148.5,0.105,9.5070,4.2
ARITH,f16_tanh,tput,100000,778541.6,0.973,1.0276,38.9
ARITH,f16_tanh,lat,100000,6205054.8,7.756,0.1289,310.3
ARITH,f16_ex2,tput,100000,778512.7,0.973,1.0276,38.9
ARITH,f16_ex2,lat,100000,6205083.8,7.756,0.1289,310.3
ARITH,s32_add,tput,100000,251963.5,0.315,3.1751,12.6                                                                                                                      
ARITH,s32_add,lat,100000,352153.6,0.440,2.2717,17.6
ARITH,s32_sub,tput,100000,84224.7,0.105,9.4984,4.2
ARITH,s32_sub,lat,100000,326289.3,0.408,2.4518,16.3
ARITH,s32_mul_lo,tput,100000,84083.2,0.105,9.5144,4.2
ARITH,s32_mul_lo,lat,100000,352189.4,0.440,2.2715,17.6
ARITH,s32_mul_hi,tput,100000,743054.0,0.929,1.0766,37.2
ARITH,s32_mul_hi,lat,100000,3375690.7,4.220,0.2370,168.8
ARITH,s32_mad_lo,tput,100000,229475.8,0.287,3.4862,11.5
ARITH,s32_mad_lo,lat,100000,907770.8,1.135,0.8813,45.4
ARITH,s32_mad_hi,tput,100000,484617.5,0.606,1.6508,24.2
ARITH,s32_mad_hi,lat,100000,3814884.9,4.769,0.2097,190.7
ARITH,s32_min,tput,100000,229447.6,0.287,3.4866,11.5
ARITH,s32_min,lat,100000,1734687.1,2.168,0.4612,86.7
ARITH,s32_max,tput,100000,229456.6,0.287,3.4865,11.5
ARITH,s32_max,lat,100000,1747543.7,2.184,0.4578,87.4
ARITH,s32_abs,tput,100000,84073.0,0.105,9.5155,4.2
ARITH,s32_abs,lat,100000,84009.0,0.105,9.5228,4.2
ARITH,s32_neg,tput,100000,4245.1,0.005,188.4517,0.2
ARITH,s32_neg,lat,100000,3361.9,0.004,237.9593,0.2
ARITH,s32_div,tput,100000,6799346.9,8.499,0.1177,340.0
ARITH,s32_div,lat,100000,22923620.2,28.655,0.0349,1146.2
ARITH,s32_rem,tput,100000,6700335.5,8.375,0.1194,335.0
ARITH,s32_rem,lat,100000,23026943.2,28.784,0.0347,1151.3
ARITH,s32_mul24_lo,tput,100000,536411.5,0.671,1.4914,26.8
ARITH,s32_mul24_lo,lat,100000,4202407.4,5.253,0.1904,210.1
ARITH,s32_mad24_lo,tput,100000,536215.6,0.670,1.4919,26.8
ARITH,s32_mad24_lo,lat,100000,4202435.5,5.253,0.1904,210.1
ARITH,s32_sad,tput,100000,229375.3,0.287,3.4877,11.5
ARITH,s32_sad,lat,100000,1734621.8,2.168,0.4612,86.7
ARITH,u32_add,tput,100000,252043.5,0.315,3.1741,12.6
ARITH,u32_add,lat,100000,352126.7,0.440,2.2719,17.6
ARITH,u32_sub,tput,100000,84042.9,0.105,9.5189,4.2
ARITH,u32_sub,lat,100000,326213.7,0.408,2.4524,16.3
ARITH,u32_mul_lo,tput,100000,84035.2,0.105,9.5198,4.2
ARITH,u32_mul_lo,lat,100000,352063.3,0.440,2.2723,17.6                                                                                                                    
ARITH,u32_mul_hi,tput,100000,743003.5,0.929,1.0767,37.2
ARITH,u32_mul_hi,lat,100000,3375462.1,4.219,0.2370,168.8
ARITH,u32_mad_lo,tput,100000,229422.0,0.287,3.4870,11.5
ARITH,u32_mad_lo,lat,100000,907675.1,1.135,0.8814,45.4
ARITH,u32_min,tput,100000,229373.4,0.287,3.4878,11.5
ARITH,u32_min,lat,100000,1734651.1,2.168,0.4612,86.7
ARITH,u32_max,tput,100000,229408.6,0.287,3.4872,11.5
ARITH,u32_max,lat,100000,1747510.6,2.184,0.4578,87.4
ARITH,u32_div,tput,100000,2716934.9,3.396,0.2944,135.8
ARITH,u32_div,lat,100000,18349855.4,22.937,0.0436,917.5
ARITH,u32_rem,tput,100000,2652164.7,3.315,0.3016,132.6
ARITH,u32_rem,lat,100000,18324144.4,22.905,0.0437,916.2
ARITH,b32_and,tput,100000,84055.0,0.105,9.5176,4.2
ARITH,b32_and,lat,100000,326214.4,0.408,2.4524,16.3
ARITH,b32_or,tput,100000,84061.4,0.105,9.5168,4.2
ARITH,b32_or,lat,100000,326210.0,0.408,2.4524,16.3
ARITH,b32_xor,tput,100000,84509.4,0.106,9.4664,4.2
ARITH,b32_xor,lat,100000,326188.2,0.408,2.4526,16.3
ARITH,b32_not,tput,100000,84068.5,0.105,9.5161,4.2
ARITH,b32_not,lat,100000,326225.3,0.408,2.4523,16.3
ARITH,b32_shl,tput,100000,332731.5,0.416,2.4043,16.6
ARITH,b32_shl,lat,100000,3809.3,0.005,210.0134,0.2
ARITH,b32_shr,tput,100000,84074.2,0.105,9.5154,4.2
ARITH,b32_shr,lat,100000,4013.4,0.005,199.3302,0.2
ARITH,s32_shr,tput,100000,84364.2,0.105,9.4827,4.2
ARITH,s32_shr,lat,100000,326259.9,0.408,2.4520,16.3
ARITH,b32_cnot,tput,100000,84027.5,0.105,9.5207,4.2
ARITH,b32_cnot,lat,100000,326316.8,0.408,2.4516,16.3
ARITH,b32_popc,tput,100000,778591.9,0.973,1.0275,38.9
ARITH,b32_popc,lat,100000,6205071.4,7.756,0.1289,310.3
ARITH,b32_clz,tput,100000,1101483.7,1.377,0.7263,55.1
ARITH,b32_clz,lat,100000,8737253.2,10.922,0.0916,436.9
ARITH,s32_bfind,tput,100000,778626.7,0.973,1.0275,38.9
ARITH,s32_bfind,lat,100000,6205072.4,7.756,0.1289,310.3
ARITH,b32_brev,tput,100000,778523.6,0.973,1.0276,38.9
ARITH,b32_brev,lat,100000,6205008.0,7.756,0.1289,310.3
ARITH,s32_bfe,tput,100000,281174.4,0.351,2.8452,14.1
ARITH,s32_bfe,lat,100000,3388532.2,4.236,0.2361,169.4
ARITH,u32_bfe,tput,100000,281084.8,0.351,2.8461,14.1
ARITH,u32_bfe,lat,100000,3388664.0,4.236,0.2361,169.4
ARITH,b32_bfi,tput,100000,93719.7,0.117,8.5361,4.7
ARITH,b32_bfi,lat,100000,1850923.9,2.314,0.4322,92.5
ARITH,b32_lop3,tput,100000,229454.1,0.287,3.4865,11.5
ARITH,b32_lop3,lat,100000,1798857.9,2.249,0.4447,89.9
ARITH,b32_prmt,tput,100000,229393.9,0.287,3.4875,11.5
ARITH,b32_prmt,lat,100000,1799279.5,2.249,0.4446,90.0
ARITH,dp4a_ss,tput,100000,3311090.9,4.139,0.2416,165.6
ARITH,dp4a_ss,lat,100000,26463760.4,33.080,0.0302,1323.2
ARITH,dp4a_uu,tput,100000,3311130.8,4.139,0.2416,165.6
ARITH,dp4a_uu,lat,100000,26463722.2,33.080,0.0302,1323.2
ARITH,dp4a_su,tput,100000,3311207.3,4.139,0.2416,165.6
ARITH,dp4a_su,lat,100000,26463998.8,33.080,0.0302,1323.2
ARITH,dp4a_us,tput,100000,3311171.1,4.139,0.2416,165.6
ARITH,dp4a_us,lat,100000,26468095.8,33.085,0.0302,1323.4
ARITH,dp2a_lo_ss,tput,100000,229478.4,0.287,3.4862,11.5
ARITH,dp2a_lo_ss,lat,100000,1799196.7,2.249,0.4446,90.0
ARITH,dp2a_hi_ss,tput,100000,229397.7,0.287,3.4874,11.5
ARITH,dp2a_hi_ss,lat,100000,1799089.8,2.249,0.4447,90.0
ARITH,s32_add_cc,tput,100000,252039.6,0.315,3.1741,12.6
ARITH,s32_add_cc,lat,100000,352035.9,0.440,2.2725,17.6
ARITH,s32_sub_cc,tput,100000,84106.2,0.105,9.5118,4.2
ARITH,s32_sub_cc,lat,100000,326295.7,0.408,2.4518,16.3
ARITH,s32_addcc_addc,tput,100000,597683.2,0.747,1.3385,29.9
ARITH,s32_addcc_addc,lat,100000,3401379.1,4.252,0.2352,170.1
===== END ARITHM =====

==> [3/4] mmvq (Qwen3-8B Q1_0, baseline DP4A) ...
===== BEGIN MMVQ_BASE =====
GPU,NVIDIA CMP 50HX,sm_75,56,5120
LAYER,q_proj,4096,4096,32,807.8,25.24,2.25,87.05
LAYER,k_proj,1024,4096,32,264.6,8.27,0.56,66.43
LAYER,v_proj,1024,4096,32,264.6,8.27,0.56,66.42
LAYER,o_proj,4096,4096,32,806.4,25.20,2.25,87.19
LAYER,gate_proj,12288,4096,32,2290.9,71.59,6.75,92.07
LAYER,up_proj,12288,4096,32,2289.0,71.53,6.75,92.15
LAYER,down_proj,4096,12288,32,2300.4,71.89,6.75,91.70
LAYER,lm_head,151669,4096,1,851.1,851.10,83.31,95.60
SUMMARY,9874.8,101.27,911.31
===== END MMVQ_BASE =====

==> [4/4] mmvq (Qwen3-8B Q1_0, DP2A+PRMT replacement) ...
===== BEGIN MMVQ_DP2A_PRMT =====
GPU,NVIDIA CMP 50HX,sm_75,56,5120
LAYER,q_proj,4096,4096,32,495.7,15.49,2.25,141.86
LAYER,k_proj,1024,4096,32,178.1,5.57,0.56,98.69
LAYER,v_proj,1024,4096,32,175.6,5.49,0.56,100.12
LAYER,o_proj,4096,4096,32,501.1,15.66,2.25,140.30
LAYER,gate_proj,12288,4096,32,1491.5,46.61,6.75,141.43
LAYER,up_proj,12288,4096,32,1496.7,46.77,6.75,140.93
LAYER,down_proj,4096,12288,32,1394.0,43.56,6.75,151.31
LAYER,lm_head,151669,4096,1,549.1,549.09,83.31,148.18
SUMMARY,6281.8,159.19,911.31
===== END MMVQ_DP2A_PRMT =====

========================================================================
 DONE — copy everything from the first '====' line above to here.

[olegk0]

and if I put it into practice, I get the following results...
gemma-4-12B-it-heretic-Q4_K_M.gguf

mod	size	params	backend	ngl	fa	test	t/s
stock	6.86 GiB	11.91 B	CUDA	-1	1	pp4096	320.93 ± 0.02
-	6.86 GiB	11.91 B	CUDA	-1	1	tg128	18.96 ± 0.00
fix ndp4a mad.lo	6.86 GiB	11.91 B	CUDA	-1	1	pp4096	321.49 ± 0.31
-	6.86 GiB	11.91 B	CUDA	-1	1	tg128	35.07 ± 0.04
fix ndp4a mul.lo.s32+sum	6.86 GiB	11.91 B	CUDA	-1	1	pp4096	320.85 ± 0.02
-	6.86 GiB	11.91 B	CUDA	-1	1	tg128	34.31 ± 0.03
fix ndp4a prmt+dp2a	6.86 GiB	11.91 B	CUDA	-1	1	pp4096	320.94 ± 0.01
-	6.86 GiB	11.91 B	CUDA	-1	1	tg128	33.72 ± 0.01
fix ndp4a mad.lo and --fmad=false	6.86 GiB	11.91 B	CUDA	-1	1	pp4096	399.24 ± 0.01
-	6.86 GiB	11.91 B	CUDA	-1	1	tg128	47.36 ± 0.01

This is the fastest option for me.

int result = c;

  int a0 = (int)(int8_t)(a);                                        
  int b0 = (int)(int8_t)(b);                                                             
  asm("mad.lo.s32 %0, %1, %2, %0;" : "+r"(result) : "r"(a0), "r"(b0));
                                                                                                                                                                        
  int a1 = (int)(int8_t)(a >> 8);                            
  int b1 = (int)(int8_t)(b >> 8);                                       
  asm("mad.lo.s32 %0, %1, %2, %0;" : "+r"(result) : "r"(a1), "r"(b1));    
                                                                                                                                                                                                                                                                                                    
  int a2 = (int)(int8_t)(a >> 16);
  int b2 = (int)(int8_t)(b >> 16);
  asm("mad.lo.s32 %0, %1, %2, %0;" : "+r"(result) : "r"(a2), "r"(b2));
                                                                                                                                                                                                                                                                                                   
  int a3 = (int)(int8_t)(a >> 24);
  int b3 = (int)(int8_t)(b >> 24);
  asm("mad.lo.s32 %0, %1, %2, %0;" : "+r"(result) : "r"(a3), "r"(b3));
                                                                                      
  return result;

[arabel1a]

Thank you for these numbers! That is pretty interesting.

1. from what I see on the benchmark, cmp50hx follows exactly the same throttling mechanism as cmp170hx: 16x slower dp4a and fmad (and not all fp32 compute like cmp90hx).

2. Do I understand it right, that "fix ndp4a prmt+dp2a" means my dp2a patch but WITHOUT --noffma flag?

Note that you stream 6GB of weights under ~500GB/s bandwidth. The theoretical peak is therefore ~80tok/s, and you reach 47 of that. This is fine ration for Turing: 2080ti with 616 GB/s bandwidth reaches 107 tok/s on 3.5GB weights --> 107tps / 176 peak, more-or-less your ratio). It seem to me that 35.07/34.31/33.72 differ mostly by noise, despite 7 vs 4 instructions, and your card is already in memory bandwidth bound regime. We can vaerify it by runnig with higher concurrency -- I expect that once we flip back to compute-bounded decode, we get DP2A working ~30% faster than MAD implementation.

[olegk0]

2. Do I understand it right, that "fix ndp4a prmt+dp2a" means my dp2a patch but WITHOUT --noffma flag?

Yes

It seem to me that 35.07/34.31/33.72 differ mostly by noise, despite 7 vs 4 instructions, and your card is already in memory bandwidth bound regime.

Yes, I think so too.

[olegk0]

The theoretical peak is therefore ~80tok/s, and you reach 47 of that.

I didn't specify earlier, but I ran this on two cmp 50 cards.

This is fine ration for Turing: 2080ti with 616 GB/s bandwidth reaches 107 tok/s

I tried running it on a similar model I had on hand and now it only runs on one video card.

model	size	params	backend	ngl	test	t/s
gemma3 4B Q5_K - Medium	2.63 GiB	3.88 B	CUDA	-1	pp512	896.66 ± 0.47
gemma3 4B Q5_K - Medium	2.63 GiB	3.88 B	CUDA	-1	tg128	110.04 ± 0.15

[arabel1a]

Can you pls try batched bench (shipped with llama.cpp)? https://github.com/ggml-org/llama.cpp/tree/master/tools/batched-bench

The aggregate tokens/s vs with concurrency curve will clarify whether we are compute, latency, or memory-bandwidth bound.

170HX shows this (model ad hand):

[olegk0]

Can you pls try batched bench (shipped with llama.cpp)? https://github.com/ggml-org/llama.cpp/tree/master/tools/batched-bench

i run this on one cmp50 (gemma-3-4b-it-Q5_K_M.gguf), with params
-c 2048 -b 2048 -ub 512 -npp 128,256,512 -ntg 128,256 -npl 1,2,4,8,16,32

llama_batched_bench: n_kv_max = 8192, n_batch = 2048, n_ubatch = 512, flash_attn = -1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 14, n_threads_batch = 14

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
128	128	1	256	0.184	695.85	1.162	110.20	1.346	190.26
128	128	2	512	0.306	837.14	1.592	160.77	1.898	269.73
128	128	4	1024	0.588	870.97	2.601	196.83	3.189	321.10
128	128	8	2048	1.147	892.67	4.342	235.83	5.489	373.10
128	128	16	4096	2.241	913.97	5.282	387.70	7.523	544.44
128	128	32	8192	4.470	916.29	9.525	430.01	13.996	585.33

W decode: failed to find a memory slot for batch of size 1