Skip to content

fix(npu): prevent ResourceCleaner timeout overflow on libstdc++ - #142

Open
cabelo wants to merge 2 commits into
intel:mainfrom
cabelo:ResourceCleaner-overflow
Open

fix(npu): prevent ResourceCleaner timeout overflow on libstdc++#142
cabelo wants to merge 2 commits into
intel:mainfrom
cabelo:ResourceCleaner-overflow

Conversation

@cabelo

@cabelo cabelo commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Issue for this PR

Fixes a crash in the Level Zero NPU driver during initialization on systems using libstdc++ implementations that convert steady_clock deadlines internally.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

This PR fixes a SIGABRT that occurs in the ResourceCleaner background thread during Level Zero initialization.

The cleaner currently uses:

std::chrono::steady_clock::time_point::max()

as the deadline passed to:

std::condition_variable::wait_until()

On some libstdc++ implementations, wait_until() converts the supplied steady_clock deadline to another clock by adding the clock offset to the requested time point.

Because time_point::max() is already close to the maximum value of the underlying signed integer representation, this conversion can overflow.

When compiled with signed-overflow trapping enabled, the overflow reaches the GCC runtime helper __addvdi3, which calls abort(). This terminates the entire process with SIGABRT.

The observed backtrace was:

__pthread_kill_implementation
raise
abort
__addvdi3
std::chrono::operator+
L0::ResourceCleaner

The failure occurred while running:

npu-umd-test --ze-init-test -c none

Five of the six initialization tests failed because the child process terminated with status 134, corresponding to SIGABRT.

This change avoids passing time_point::max() to wait_until().

When no cleanup timeout is active, the cleaner now uses:

cv.wait(lock);

When a cleanup timeout is requested, it continues using:

cv.wait_until(lock, timeout);

This preserves the existing behavior while avoiding the unsafe clock conversion and integer overflow.

The change also protects updates to idleTimeout with the cleaner mutex, preventing concurrent access between setIdleTimeout() and the background cleanup thread.

Root cause

The previous implementation used an artificial maximum deadline to represent an infinite wait:

auto timeout =
    std::chrono::steady_clock::time_point::max();

cv.wait_until(lock, timeout);

An infinite wait should not be represented by a maximum time point because the standard library may perform clock conversions internally.

Using condition_variable::wait() is the correct way to wait indefinitely.

How did you verify your code works?

The issue was reproduced on:

openSUSE Leap 15.6
Kernel 7.1.4-150600.4.gad68314-default
linux-npu-driver 1.35.0
NPU40xx
Firmware UD202628

Before the change:

[  PASSED  ] 1 test.
[  FAILED  ] 5 tests.

The failed child processes returned:

status: 134

GDB confirmed that the process aborted because of a signed integer overflow in the ResourceCleaner thread.

After applying the change:

npu-umd-test --ze-init-test -c none

completed successfully:

[  PASSED  ] 6 tests.

The NPU was subsequently detected correctly by OpenVINO:

import openvino as ov

core = ov.Core()
print(core.available_devices)

Result:

['CPU', 'NPU']

Checklist

  • I have tested my changes locally
  • I have reproduced the original failure
  • I have verified the fix with npu-umd-test
  • I have verified NPU detection in OpenVINO
  • I have not included unrelated changes in this PR

Changes

  • Replaces an indefinite wait_until(time_point::max()) call with condition_variable::wait()
  • Uses wait_until() only when an actual cleanup deadline exists
  • Prevents signed integer overflow during libstdc++ clock conversion
  • Prevents the ResourceCleaner thread from terminating the process with SIGABRT
  • Synchronizes updates to idleTimeout using the existing mutex

@cabelo

cabelo commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

@hmaarrfk , @vicamo , @sandraharon and @izelnakri

I am an Intel Innovator and the maintainer of the linux-npu-driver and the OpenVINO library for official openSUSE. Below are some news items as evidence.

https://news.opensuse.org/2025/12/02/NPU-arrives-in-os-distributions/
https://news.opensuse.org/2024/05/14/openvino-arrives-in-os-distributions/

  • level-zero-1.32.0
  • openvino-intel-npu-plugin-2026.2
  • linux-npu-driver-1.35.0

I created this PR because of the error below.

grep -E \
> 'Caught signal|SIGABRT|Program received|#0|#1|#2|#3|#4|#5|libze_|terminate|assert' \
> /tmp/npu-gdb.log
Catchpoint 1 (signal SIGABRT)
Thread 2.2 "npu-umd-test" hit Catchpoint 1 (signal SIGABRT), 0x00007ffff5ea94ac in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x00007ffff5ea94ac in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007ffff5e578c2 in raise () from /lib64/libc.so.6
#2  0x00007ffff5e3f64f in abort () from /lib64/libc.so.6
#3  0x00007ffff65d85a3 in __addvdi3 () from /lib64/libgcc_s.so.1
#4  0x00007ffff6169ce1 in std::chrono::operator+<long, std::ratio<1l, 1000000000l>, long, std::ratio<1l, 1000000000l> > (__rhs=..., __lhs=...) at /usr/include/c++/12/bits/chrono.h:693
#5  std::chrono::operator+<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, long, std::ratio<1l, 1000000000l> > (__rhs=..., __lhs=<synthetic pointer>std::chrono::sys_time = { 1785380739105344339ns [2026-07-30 03:05:39] }) at /usr/include/c++/12/bits/chrono.h:1099
#10 std::thread::_Invoker<std::tuple<L0::ResourceCleaner::ResourceCleaner(L0::Context*, std::chrono::milliseconds)::<lambda(L0::Context*)>, L0::Context*> >::_M_invoke<0, 1> (this=0x74e748) at /usr/include/c++/12/bits/std_thread.h:279
#11 std::thread::_Invoker<std::tuple<L0::ResourceCleaner::ResourceCleaner(L0::Context*, std::chrono::milliseconds)::<lambda(L0::Context*)>, L0::Context*> >::operator() (this=0x74e748) at /usr/include/c++/12/bits/std_thread.h:286
#12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<L0::ResourceCleaner::ResourceCleaner(L0::Context*, std::chrono::milliseconds)::<lambda(L0::Context*)>, L0::Context*> > >::_M_run(void) (this=0x74e740) at /usr/include/c++/12/bits/std_thread.h:231
#13 0x00007ffff62ef0e4 in ?? () from /usr/lib64/libstdc++.so.6
#14 0x00007ffff5ea761c in start_thread () from /lib64/libc.so.6
#15 0x00007ffff5f2eaa8 in clone3 () from /lib64/libc.so.6
#0  0x00007ffff5f2a5f2 in mmap64 () from /lib64/libc.so.6
#1  0x00007ffff619b631 in VPU::OsInterfaceImp::osiMmap (offset=<optimized out>, fd=<optimized out>, flags=1, prot=3, size=<optimized out>, addr=0x0, this=<optimized out>) at ../../vpu_driver/source/./umd/vpu_driver/source/os_interface/os_interface_imp.cpp:102
#2  VPU::VPUDriverApi::mmap(unsigned long, long) const [clone .isra.0] (size=<optimized out>, offset=<optimized out>, this=<optimized out>) at ../../vpu_driver/source/./umd/vpu_driver/source/os_interface/vpu_driver_api.cpp:330
#3  0x00007ffff6193ab7 in VPU::VPUBufferObject::create (drvApi=..., type=VPU::VPUBufferObject::Location::Shared, range=VPU::VPUBufferObject::Type::CachedDma, size=10) at ../../vpu_driver/source/./umd/vpu_driver/source/memory/vpu_buffer_object.cpp:90
#4  0x00007ffff6178e77 in VPU::VPUDeviceContext::createBufferObject (loc=<optimized out>, type=<optimized out>, size=<optimized out>, this=0x74e7c0) at ../../vpu_driver/source/./umd/vpu_driver/source/device/vpu_device_context.cpp:83
#5  VPU::VPUDeviceContext::createMemAlloc (loc=<optimized out>, type=<optimized out>, size=<optimized out>, this=0x74e7c0) at ../source/./umd/vpu_driver/source/device/vpu_device_context.hpp:70
#10 0x00000000005845f1 in executeCopyCommand (driver=<optimized out>) at ./validation/umd-test/test_init_drivers.cpp:87
        sContext = warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<_ze_context_handle_t*, zeScope::contextCreate(_ze_driver_handle_t*, _ze_context_desc_t&, _ze_result_t&)::{lambda(auto:1)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<_ze_context_handle_t*, zeScope::contextCreate(_ze_driver_handle_t*, _ze_context_desc_t&, _ze_result_t&)::{lambda(auto:1)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
        sCommandQueue = warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<_ze_command_queue_handle_t*, zeScope::commandQueueCreate(_ze_context_handle_t*, _ze_device_handle_t*, _ze_command_queue_desc_t const&, _ze_result_t&)::{lambda(auto:1)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<_ze_command_queue_handle_t*, zeScope::commandQueueCreate(_ze_context_handle_t*, _ze_device_handle_t*, _ze_command_queue_desc_t const&, _ze_result_t&)::{lambda(auto:1)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
        sCommandList = warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<_ze_command_list_handle_t*, zeScope::commandListCreate(_ze_context_handle_t*, _ze_device_handle_t*, _ze_command_list_desc_t const&, _ze_result_t&)::{lambda(auto:1)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<_ze_command_list_handle_t*, zeScope::commandListCreate(_ze_context_handle_t*, _ze_device_handle_t*, _ze_command_list_desc_t const&, _ze_result_t&)::{lambda(auto:1)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
#11 0x00000000005870ce in operator() (__closure=<optimized out>) at ./validation/umd-test/test_init_drivers.cpp:123
#12 std::__invoke_impl<void, ZeInitDriversTest_InitializeAndExecuteCopyCommand_Test::TestBody()::<lambda()>&> (__f=...) at /usr/include/c++/12/bits/invoke.h:61
#13 std::__invoke_r<void, ZeInitDriversTest_InitializeAndExecuteCopyCommand_Test::TestBody()::<lambda()>&> (__fn=...) at /usr/include/c++/12/bits/invoke.h:111
#14 std::_Function_handler<void(), ZeInitDriversTest_InitializeAndExecuteCopyCommand_Test::TestBody()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/12/bits/std_function.h:290
#15 0x0000000000583ff9 in std::function<void ()>::operator()() const (this=0x7fffffffe370) at /usr/include/c++/12/bits/std_function.h:591
#16 RunInFork(const std::function<void()> &) (testFunction=...) at ./validation/umd-test/test_init_drivers.cpp:26
#17 0x00000000005840db in ZeInitDriversTest_InitializeAndExecuteCopyCommand_Test::TestBody (this=<optimized out>) at ./validation/umd-test/test_init_drivers.cpp:108
#18 0x00007ffff6901c37 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) () from /usr/lib64/libgtest.so.1.17.0
#19 0x00007ffff68f16fe in testing::Test::Run() () from /usr/lib64/libgtest.so.1.17.0
#20 0x00007ffff68f1885 in testing::TestInfo::Run() () from /usr/lib64/libgtest.so.1.17.0
#21 0x00007ffff68f3947 in testing::TestSuite::Run() () from /usr/lib64/libgtest.so.1.17.0
#22 0x00007ffff68f7a4f in testing::internal::UnitTestImpl::RunAllTests() () from /usr/lib64/libgtest.so.1.17.0
#23 0x00007ffff69021a7 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) () from /usr/lib64/libgtest.so.1.17.0
#24 0x00007ffff68f1921 in testing::UnitTest::Run() () from /usr/lib64/libgtest.so.1.17.0
#25 0x000000000061c0eb in RUN_ALL_TESTS () at /usr/include/gtest/gtest.h:2334
#26 0x00000000004320cc in main (argc=5, argv=0x7fffffffed18) at ./validation/umd-test/m/home/cabelo/rpmbuild/BUILD/linux-npu-driver-1.35.0/validation/umd-test/test_init_drivers.cpp:32: Failure
0x00007ffff693e2d0  0x00007ffff69ad8a2  Yes (*)     /usr/lib64/libze_loader.so.1
0x00007ffff60f8ee0  0x00007ffff61b84ca  Yes         /usr/lib64/libze_intel_npu.so.1
0x00007ffff6044d80  0x00007ffff60d2cbc  Yes (*)     /usr/lib64/libze_tracing_layer.so.1

Comment thread umd/level_zero_driver/source/context.cpp Outdated
@cabelo

cabelo commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

@mateusztabaka
Everything working perfectly on all openSUSE distributions.

Evidence bellow

https://build.opensuse.org/package/show/home:cabelo:intel/linux-npu-driver

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants