After changing from @testsets to @testitems Genie tests produce a segfault for julia 1.10.
I am not sure that this change produced the error, it might also be a change of HTTP since the last testing.
The stacktrace is pointing to HTTP, but I am not deep enough involved in HTTP to see the root cause. I ask Gemini for a possible root cause and that's what I got after some iteration:
Bug Description
When running heavy concurrent/parallel integration tests (specifically using TestItemRunner.jl which spins up multiple tasks/workers simultaneously), HTTP.jl causes a hard crash via a Segmentation fault (Signal 11) or a SIGABRT (Signal 22) inside the Garbage Collector during process exit (atexit).
This issue happens deterministically on Julia 1.10 (LTS), but seems to be masked or avoided on Julia 1.11+ due to newer memory/GC models and scheduling behaviors.
Stacktraces
1. Segmentation Fault during runtime:
signal (11.1): Segmentation fault
in expression starting at /home/runner/work/Genie.jl/Genie.jl/test/tests_responses.jl:30
ijl_array_copy at /cache/build/builder-amdci4-7/julialang/julia-ci/src/array.c:1179
copy at ./array.jl:411 [inlined]
Dict at ./dict.jl:73 [inlined]
Set at ./set.jl:46 [inlined]
copymutable at ./set.jl:116 [inlined]
copy at ./set.jl:114 [inlined]
_retry_partition_set_capacity! at /home/runner/.julia/packages/HTTP/RuJUS/src/http_retry.jl:162 [inlined]
#96 at /home/runner/.julia/packages/HTTP/RuJUS/src/http_retry.jl:193
lock at ./lock.jl:229
2. SIGABRT in GC during atexit cleanup:
signal (22): SIGABRT
in expression starting at C:\Users\helmu\.julia\dev\Genie\test\tests_content_negotiation.jl:13
crt_sig_handler at C:/workdir/src\signals-win.c:95
raise at C:\WINDOWS\System32\msvcrt.dll (unknown line)
abort at C:\WINDOWS\System32\msvcrt.dll (unknown line)
gc_dump_queue_and_abort at C:/workdir/src\gc.c:1815
gc_mark_outrefs at C:/workdir/src\gc.c:2521 [inlined]
gc_mark_loop_serial_ at C:/workdir/src\gc.c:2690
...
ijl_array_copy at C:/workdir/src\array.c:1179
copy at .\array.jl:411 [inlined]
Dict at .\dict.jl:73 [inlined]
Set at .\set.jl:46 [inlined]
copymutable at .\set.jl:116 [inlined]
copy at .\set.jl:114 [inlined]
_retry_partition_set_capacity! at C:\Users\helmu\.julia\packages\HTTP\RuJUS/src\http_retry.jl:162 [inlined]
#96 at C:\Users\helmu\.julia\packages\HTTP\RuJUS/src\http_retry.jl:193
lock at .\lock.jl:229
acquire at C:\Users\helmu\.julia\packages\HTTP\RuJUS/src\http_retry.jl:188 [inlined]
_arm_request_retry! at C:\Users\helmu\.julia\packages\HTTP\RuJUS/src\http_client_retry.jl:242
_do_incoming! at C:\Users\helmu\.julia\packages\HTTP\RuJUS/src\http_client.jl:1019
Cause Analysis
The crash occurs in src/http_retry.jl inside _retry_partition_set_capacity!. Under heavy multithreaded test loads (where multiple test blocks call HTTP.request concurrently using the default global client state), a race condition triggers during the copy(set) / copymutable operations.
Because Set and Dict structures are not thread-safe in Julia, simultaneous mutations or reads from different tasks during _arm_request_retry! cause data corruption in the underlying C-array (ijl_array_copy), immediately leading to a memory fault or an invalidated GC state.
Environment
- HTTP.jl Version: v2.6.6 (latest stable)
- Julia Version: 1.10.x (LTS)
- OS: Occurs on both Linux CI (GitHub Actions) and local Windows machines.
Workaround
Explicitly passing an isolated client = HTTP.Client() to each concurrent test task avoids hitting the global shared state in http_retry.jl and mitigates the crash. However, concurrent calls via the default global client shouldn't cause hard segfaults.
An internal ReentrantLock might be missing around the capacity/retry layer management inside http_retry.jl.
After changing from
@testsets to@testitems Genie tests produce a segfault for julia 1.10.I am not sure that this change produced the error, it might also be a change of HTTP since the last testing.
The stacktrace is pointing to HTTP, but I am not deep enough involved in HTTP to see the root cause. I ask Gemini for a possible root cause and that's what I got after some iteration:
Bug Description
When running heavy concurrent/parallel integration tests (specifically using
TestItemRunner.jlwhich spins up multiple tasks/workers simultaneously),HTTP.jlcauses a hard crash via a Segmentation fault (Signal 11) or a SIGABRT (Signal 22) inside the Garbage Collector during process exit (atexit).This issue happens deterministically on Julia 1.10 (LTS), but seems to be masked or avoided on Julia 1.11+ due to newer memory/GC models and scheduling behaviors.
Stacktraces
1. Segmentation Fault during runtime:
2. SIGABRT in GC during
atexitcleanup:Cause Analysis
The crash occurs in
src/http_retry.jlinside_retry_partition_set_capacity!. Under heavy multithreaded test loads (where multiple test blocks callHTTP.requestconcurrently using the default global client state), a race condition triggers during thecopy(set)/copymutableoperations.Because
SetandDictstructures are not thread-safe in Julia, simultaneous mutations or reads from different tasks during_arm_request_retry!cause data corruption in the underlying C-array (ijl_array_copy), immediately leading to a memory fault or an invalidated GC state.Environment
Workaround
Explicitly passing an isolated
client = HTTP.Client()to each concurrent test task avoids hitting the global shared state inhttp_retry.jland mitigates the crash. However, concurrent calls via the default global client shouldn't cause hard segfaults.An internal
ReentrantLockmight be missing around the capacity/retry layer management insidehttp_retry.jl.