[CI investigation] Ruby 4.0.6 hang: A/B test 4.0.5 vs 4.0.6 — do not merge - #726
Closed
soutaro wants to merge 9 commits into
Closed
[CI investigation] Ruby 4.0.6 hang: A/B test 4.0.5 vs 4.0.6 — do not merge#726soutaro wants to merge 9 commits into
soutaro wants to merge 9 commits into
Conversation
The 'Ruby 4.0 with Gemfile' jobs on ubuntu and macos started hanging until the job timeout when ruby/setup-ruby was bumped past 1.316.0 (PR #724, #725). Newer setup-ruby resolves ruby-version '4.0' to 4.0.6 instead of 4.0.5, and PR #723 (same day as #724, still on 1.316.0/Ruby 4.0.5) passed, so Ruby 4.0.6 is the suspected culprit. This narrows the matrix to Gemfile x {4.0.5, 4.0.6} x {ubuntu, macos}, runs Minitest verbosely with synced stdout, and interrupts the suite with SIGINT after 15 minutes so the hanging test and its backtrace appear in the log. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
macos-latest runners have neither timeout nor gtimeout, so the previous step failed immediately under bash -e. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…achinery The hung job's orphan-process list showed the spawned bundle/sh/ruby chain still alive at cancellation, and the interrupted test was stuck polling for the RunnerClient to finish booting. This job boots the exact same subprocess in a loop to get a minimal repro and an in-process backtrace via SIGINT/SIGABRT. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… on hang server.rb start requires a capabilities JSON as ARGV[1]; without it every iteration died immediately in JSON.parse. Also attach gdb/lldb before the SIGINT escalation so a hang yields all-thread native backtraces. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Boot-loop result: 100 standalone boots of the rails runner server pass on Ruby 4.0.6 on both ubuntu and macos, so the deadlock needs the threaded test-process context. Drop the boot-loop job, focus the matrix on 4.0.6, and on hang attach a debugger to parent and children to see which side is stuck and where. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Boots the runner server through with_server/addon activation exactly like the feature tests, 120 times, with an in-process watchdog that dumps Ruby-level backtraces of all threads after 120s of no progress. The shell watchdog adds gdb/lldb native backtraces of parent and children. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
If the rails runner server fails to boot (or hangs during boot), kill it, surface the failure reason (including the server's stderr) and retry up to MAX_BOOT_ATTEMPTS times before falling back to NullClient. Also warn() the failure reason in create_client so CI logs show why boots fail, and apt-get update before installing gdb in the CI watchdogs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The previous run showed the boot thread vanishing without any warn and without Ruby's report_on_exception output — meaning it likely finished normally, i.e. took a NullClient path without raising. Trace every path through create_client (entry with cwd, success, bin/rails-missing, giving up, non-StandardError) and have the stress watchdog inspect the addon's boot thread (status + stored exception via #value). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI tracing showed the intermittent hang happens when the boot thread checks
File.exist?("bin/rails") while another thread has temporarily changed the
process working directory — RuboCop's config loader does
Dir.chdir(config dir) { ERB.new(...).result } while resolving inherit_gem,
so the check sees the rubocop-shopify gem directory and create_client
silently returns a NullClient, which the tests busy-wait on forever.
Ruby 4.0.6's scheduler changes (Bug #21685 backport) made this pre-existing
race fire frequently on CI.
Use global_state.workspace_path for the bin/rails check and pass
chdir: workspace_path when spawning the server so neither depends on the
racy process-global working directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Do not merge — CI experiment only
The
Ruby 4.0 on {ubuntu,macos}-latest with Gemfilejobs hang until the 25-minute job timeout on #724 and #725 (theruby/setup-rubybumps).Evidence so far
ruby-version: "4.0"to Ruby 4.0.5; 1.319.0/1.321.0 resolve it to Ruby 4.0.6 (released 2026-07-14).Gemfile(Rails ~> 8.1.0) hangs;Gemfile-rails-8-0andGemfile-rails-mainpass on 4.0.6. Windows also passes.What this PR does
Gemfile× Ruby {4.0.5, 4.0.6} × {ubuntu, macos} for a direct A/B comparison.--verboseand synced stdout so the log shows which test is executing.SIGINTafter 15 minutes so Minitest reports the interrupted test with a backtrace of where it was stuck.Expected outcome
🤖 Generated with Claude Code