[v7.0.0] AssertionError: Tensor-likes are not equal on Pytorch built with dpclang v7.0.0

[dvrogozh]

Describe the bug

Some pytorch tests are failing with on comparing output tensors with the references:

AssertionError: Tensor-likes are not equal!

To reproduce

Below assumes that dpclang is installed and configured. For example:

export CMAKE_PREFIX_PATH=~/git/intel-llvm/build/install/
export PKG_CONFIG_PATH=$HOME/git/intel-llvm/build/install/lib/pkgconfig:$PKG_CONFIG_PATH
export LD_LIBRARY_PATH=$HOME/git/intel-llvm/build/install/lib:$LD_LIBRARY_PATH
export PATH=$HOME/git/intel-llvm/build/install/bin:$PATH

Using pre-built pytorch wheels:

1. Work in python 3.10 virtual environment:

uv venv --python 3.10
source .venv/bin/activate

2. Install pytorch pre-built with dpclang v7.0.0 by CI job (download and install Torch-XPU-Wheel-4789-31020166365-1-dpclang archive)

uv pip install torch-2.14.0a0+git99731ed-cp310-cp310-linux_x86_64.whl

3. Run the following python script:

import torch

input = torch.tensor([
  [  0,   0, 233,   0,   0],
  [  0, 125,   0, 233,  81],
  [  0,   0, 153,  13,   0],
  [ 14,   0,  65,   0,   0],
  [  0,   0,  89,  31,   0]], device='xpu:0', dtype=torch.uint8)
src = torch.tensor([
  [120, 185,   0, 151,   0],
  [143, 106,  24,   6,   0],
  [109,  19, 245, 117,   0],
  [  0, 123,   0,   0, 203],
  [  0,   0,   0, 118,   0]], device='xpu:0', dtype=torch.uint8)

index = torch.tensor(
  [3, 2, 1, 4, 0], device='xpu:0')

# NOTE: This will actually work correctly:
# print(input.index_add(-1, index, src, alpha=1))

input_bool = input.view(torch.bool)
src_bool = src.view(torch.bool)
print(input_bool.index_add(-1, index, src_bool, alpha=1))

With dpclang v7.0.0 output will be:

tensor([[False, False,  True, False,  True],
        [False,  True, False,  True,  True],
        [False,  True,  True,  True,  True],
        [ True, False,  True, False, False],
        [False, False,  True,  True, False]], device='xpu:0')

Expected output is (with pytorch built with oneAPI 2026.0):

tensor([[False, False,  True,  True,  True],
        [False,  True,  True,  True,  True],
        [False,  True,  True,  True,  True],
        [ True, False,  True, False, False],
        [False, False,  True,  True,  True]], device='xpu:0')

To build pytorch from sources:

git clone https://github.com/pytorch/pytorch.git && cd pytorch
git reset --hard 240f31ace12a3bbbb34d10666780e1c365cfbb6e
git submodule update --init --recursive
pip install -r requirements.txt
XPU_SYCL_COMPILER=dpclang TORCH_XPU_ARCH_LIST=pvc USE_KINETO=0 USE_ONEMKL_XPU=0 python -m pip install --no-build-isolation -v -e .

Environment

• OS: Linux
• Target device and vendor: Pytorch CI was running on PVC, other devices not tried
• DPC++ version: https://github.com/intel/llvm/releases/tag/v7.0.0
• Dependencies version:

$ sycl-ls --verbose
<LOADER>[INFO]: failed to load adapter 'libur_adapter_cuda.so.0' with error: libur_adapter_cuda.so.0: cannot open shared object file: No such file or directory
<LOADER>[INFO]: failed to load adapter '/home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_cuda.so.0' with error: /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_cuda.so.0: cannot open shared object file: No such file or directory
<LOADER>[INFO]: failed to load adapter 'libur_adapter_hip.so.0' with error: libur_adapter_hip.so.0: cannot open shared object file: No such file or directory
<LOADER>[INFO]: failed to load adapter '/home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_hip.so.0' with error: /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_hip.so.0: cannot open shared object file: No such file or directory
<LOADER>[INFO]: loaded adapter 0x0x608a6aae5210 (libur_adapter_level_zero.so.0) from /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_level_zero.so.0
<LOADER>[INFO]: loaded adapter 0x0x608a6aae82f0 (libur_adapter_level_zero_v2.so.0) from /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_level_zero_v2.so.0
<LOADER>[INFO]: failed to load adapter 'libur_adapter_native_cpu.so.0' with error: libur_adapter_native_cpu.so.0: cannot open shared object file: No such file or directory
<LOADER>[INFO]: failed to load adapter '/home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_native_cpu.so.0' with error: /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_native_cpu.so.0: cannot open shared object file: No such file or directory
<LOADER>[INFO]: failed to load adapter 'libur_adapter_offload.so.0' with error: libur_adapter_offload.so.0: cannot open shared object file: No such file or directory
<LOADER>[INFO]: failed to load adapter '/home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_offload.so.0' with error: /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_offload.so.0: cannot open shared object file: No such file or directory
<LOADER>[INFO]: loaded adapter 0x0x608a6aaeb6c0 (libur_adapter_opencl.so.0) from /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_opencl.so.0
[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
[level_zero:gpu][level_zero:1] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
[level_zero:gpu][level_zero:2] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
[level_zero:gpu][level_zero:3] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
[opencl:gpu][opencl:0] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO  [25.18.33578]
[opencl:gpu][opencl:1] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO  [25.18.33578]
[opencl:gpu][opencl:2] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO  [25.18.33578]
[opencl:gpu][opencl:3] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO  [25.18.33578]

Platforms: 2
Platform [#1]:
    Version  : 1.6
    Name     : Intel(R) oneAPI Unified Runtime over Level-Zero
    Vendor   : Intel(R) Corporation
    Devices  : 4
        Device [#0]:
        Type              : gpu
        Version           : 12.60.7
        Name              : Intel(R) Data Center GPU Max 1550
        Vendor            : Intel(R) Corporation
        Driver            : 1.6.33578+42
        UUID              : 8680d50b-2f00-0000-4d00-000000000001
        DeviceID          : 3029
        Num SubDevices    : 0
        Num SubSubDevices : 0
        Aspects           : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_native_assert ext_intel_gpu_hw_threads_per_eu ext_intel_free_memory ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_external_memory_import ext_oneapi_external_semaphore_import ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_bindless_sampled_image_fetch_1d_usm ext_oneapi_bindless_sampled_image_fetch_1d ext_oneapi_bindless_sampled_image_fetch_2d_usm ext_oneapi_bindless_sampled_image_fetch_2d ext_oneapi_bindless_sampled_image_fetch_3d ext_oneapi_queue_profiling_tag ext_oneapi_virtual_mem ext_oneapi_image_array ext_oneapi_virtual_functions ext_intel_spill_memory_size ext_intel_current_clock_throttle_reasons ext_intel_power_limits ext_oneapi_async_memory_alloc ext_oneapi_exportable_device_mem ext_oneapi_ipc_memory
        info::device::sub_group_sizes: 16 32
        Architecture: intel_gpu_pvc
        Device [#1]:
        Type              : gpu
        Version           : 12.60.7
        Name              : Intel(R) Data Center GPU Max 1550
        Vendor            : Intel(R) Corporation
        Driver            : 1.6.33578+42
        UUID              : 8680d50b-2f00-0000-4d00-000000000002
        DeviceID          : 3029
        Num SubDevices    : 0
        Num SubSubDevices : 0
        Aspects           : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_native_assert ext_intel_gpu_hw_threads_per_eu ext_intel_free_memory ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_external_memory_import ext_oneapi_external_semaphore_import ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_bindless_sampled_image_fetch_1d_usm ext_oneapi_bindless_sampled_image_fetch_1d ext_oneapi_bindless_sampled_image_fetch_2d_usm ext_oneapi_bindless_sampled_image_fetch_2d ext_oneapi_bindless_sampled_image_fetch_3d ext_oneapi_queue_profiling_tag ext_oneapi_virtual_mem ext_oneapi_image_array ext_oneapi_virtual_functions ext_intel_spill_memory_size ext_intel_current_clock_throttle_reasons ext_intel_power_limits ext_oneapi_async_memory_alloc ext_oneapi_exportable_device_mem ext_oneapi_ipc_memory
        info::device::sub_group_sizes: 16 32
        Architecture: intel_gpu_pvc
        Device [#2]:
        Type              : gpu
        Version           : 12.60.7
        Name              : Intel(R) Data Center GPU Max 1550
        Vendor            : Intel(R) Corporation
        Driver            : 1.6.33578+42
        UUID              : 8680d50b-2f00-0000-9a00-000000000001
        DeviceID          : 3029
        Num SubDevices    : 0
        Num SubSubDevices : 0
        Aspects           : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_native_assert ext_intel_gpu_hw_threads_per_eu ext_intel_free_memory ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_external_memory_import ext_oneapi_external_semaphore_import ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_bindless_sampled_image_fetch_1d_usm ext_oneapi_bindless_sampled_image_fetch_1d ext_oneapi_bindless_sampled_image_fetch_2d_usm ext_oneapi_bindless_sampled_image_fetch_2d ext_oneapi_bindless_sampled_image_fetch_3d ext_oneapi_queue_profiling_tag ext_oneapi_virtual_mem ext_oneapi_image_array ext_oneapi_virtual_functions ext_intel_spill_memory_size ext_intel_current_clock_throttle_reasons ext_intel_power_limits ext_oneapi_async_memory_alloc ext_oneapi_exportable_device_mem ext_oneapi_ipc_memory
        info::device::sub_group_sizes: 16 32
        Architecture: intel_gpu_pvc
        Device [#3]:
        Type              : gpu
        Version           : 12.60.7
        Name              : Intel(R) Data Center GPU Max 1550
        Vendor            : Intel(R) Corporation
        Driver            : 1.6.33578+42
        UUID              : 8680d50b-2f00-0000-9a00-000000000002
        DeviceID          : 3029
        Num SubDevices    : 0
        Num SubSubDevices : 0
        Aspects           : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_native_assert ext_intel_gpu_hw_threads_per_eu ext_intel_free_memory ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_external_memory_import ext_oneapi_external_semaphore_import ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_bindless_sampled_image_fetch_1d_usm ext_oneapi_bindless_sampled_image_fetch_1d ext_oneapi_bindless_sampled_image_fetch_2d_usm ext_oneapi_bindless_sampled_image_fetch_2d ext_oneapi_bindless_sampled_image_fetch_3d ext_oneapi_queue_profiling_tag ext_oneapi_virtual_mem ext_oneapi_image_array ext_oneapi_virtual_functions ext_intel_spill_memory_size ext_intel_current_clock_throttle_reasons ext_intel_power_limits ext_oneapi_async_memory_alloc ext_oneapi_exportable_device_mem ext_oneapi_ipc_memory
        info::device::sub_group_sizes: 16 32
        Architecture: intel_gpu_pvc
Platform [#2]:
    Version  : OpenCL 3.0
    Name     : Intel(R) OpenCL Graphics
    Vendor   : Intel(R) Corporation
    Devices  : 4
        Device [#0]:
        Type              : gpu
        Version           : OpenCL 3.0 NEO
        Name              : Intel(R) Data Center GPU Max 1550
        Vendor            : Intel(R) Corporation
        Driver            : 25.18.33578
        UUID              : 8680d50b-2f00-0000-4d00-000000000001
        DeviceID          : 3029
        Num SubDevices    : 0
        Num SubSubDevices : 0
        Aspects           : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_srgb ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_private_alloca ext_oneapi_atomic16 ext_oneapi_virtual_functions ext_intel_device_info_node_mask
        info::device::sub_group_sizes: 16 32
        Architecture: intel_gpu_pvc
        Device [#1]:
        Type              : gpu
        Version           : OpenCL 3.0 NEO
        Name              : Intel(R) Data Center GPU Max 1550
        Vendor            : Intel(R) Corporation
        Driver            : 25.18.33578
        UUID              : 8680d50b-2f00-0000-4d00-000000000002
        DeviceID          : 3029
        Num SubDevices    : 0
        Num SubSubDevices : 0
        Aspects           : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_srgb ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_private_alloca ext_oneapi_atomic16 ext_oneapi_virtual_functions ext_intel_device_info_node_mask
        info::device::sub_group_sizes: 16 32
        Architecture: intel_gpu_pvc
        Device [#2]:
        Type              : gpu
        Version           : OpenCL 3.0 NEO
        Name              : Intel(R) Data Center GPU Max 1550
        Vendor            : Intel(R) Corporation
        Driver            : 25.18.33578
        UUID              : 8680d50b-2f00-0000-9a00-000000000001
        DeviceID          : 3029
        Num SubDevices    : 0
        Num SubSubDevices : 0
        Aspects           : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_srgb ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_private_alloca ext_oneapi_atomic16 ext_oneapi_virtual_functions ext_intel_device_info_node_mask
        info::device::sub_group_sizes: 16 32
        Architecture: intel_gpu_pvc
        Device [#3]:
        Type              : gpu
        Version           : OpenCL 3.0 NEO
        Name              : Intel(R) Data Center GPU Max 1550
        Vendor            : Intel(R) Corporation
        Driver            : 25.18.33578
        UUID              : 8680d50b-2f00-0000-9a00-000000000002
        DeviceID          : 3029
        Num SubDevices    : 0
        Num SubSubDevices : 0
        Aspects           : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_srgb ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_private_alloca ext_oneapi_atomic16 ext_oneapi_virtual_functions ext_intel_device_info_node_mask
        info::device::sub_group_sizes: 16 32
        Architecture: intel_gpu_pvc
default_selector()      : gpu, Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
accelerator_selector()  : No device of requested type available.
cpu_selector()          : No device of requested type available.
gpu_selector()          : gpu, Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
custom_selector(gpu)    : gpu, Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
custom_selector(cpu)    : No device of requested type available.
custom_selector(acc)    : No device of requested type available.
<LOADER>[INFO]: unloaded adapter 0x0x608a6aae5210
<LOADER>[INFO]: unloaded adapter 0x0x608a6aae82f0
<LOADER>[INFO]: unloaded adapter 0x0x608a6aaeb6c0
<LOADER>[INFO]: ---> urLoaderTearDown() -> UR_RESULT_SUCCESS

Additional context

For the original reproducer based on pytorch CI test see intel/torch-xpu-ops#4842.

@sarnex, @KornevNikita