enable triton kernel on XPU

[jiqing-feng]

The triton kernel also works on Intel XPU and could bring 10x speed-up compared to pytorch implementation.

[jiqing-feng]

Hi @Godofnothing . Would you please review this PR? Thanks!

[jiqing-feng]

BTW, I don't understand why it has this restriction for gemv. I removed this restriction and found triton kernel still works and can give the same outputs as PyTorch Gemm.

[BlackSamorez]

BTW, I don't understand why it has this restriction for gemv

Repeated GEMV is faster when bs<=6 and dequant+GEMM is faster when bs>6.

[jiqing-feng]

BTW, I don't understand why it has this restriction for gemv

Repeated GEMV is faster when bs<=6 and dequant+GEMM is faster when bs>6.

I see, that works for cuda kernel. But for triton, it still faster than pytorch. I removed the training check for triton because it will faster than pytorch anyway.

Hi @BlackSamorez , would you please review this change? Thanks!

[jiqing-feng]

Hi @BlackSamorez @Vahe1994 . Would you please review this PR? Thanks!

[BlackSamorez]

Dequantization is still faster than GEMV starting with some batch sizes. Please benchmark the two and choose a threahold.

[jiqing-feng]

Dequantization is still faster than GEMV starting with some batch sizes. Please benchmark the two and choose a threahold.

The restriction is for batch_size * sequence_length actually, so I tested the prefill and decode latency for triton and torch.

Here is the triton speed-up compared to pytorch on XPU.
input length = 32
| batch_size | prefill | decode |
| 1 | 6.0x | 19x |
| 4 | 1.7x | 19x |
| 8 | 1.0x | 15x |
| 32 | 0.2x | 6x |

You were right, the threshold should be 8*32=256 on XPU (because pytorch and triton show almost the same performance when batch size is 8 and input length is 32).

Hi @BlackSamorez . Thanks for your reminder, I have figured out the threshold for using gemv on XPU. Please review the changes. 😄

[jiqing-feng]

Hi @BlackSamorez , do you mind taking a look at this PR? Thanks!

[jiqing-feng]

Hi @BlackSamorez . Do you have time to take a look at this PR? Thanks!