Add random-netsplits tc, trace improvements - #42
Merged
Conversation
Serialize become_leader_ with an async all_alive sync lock and abort_on_deadlock so competing leaders abort instead of deadlocking on state merge. Harden CT netsplit setup for peer/OTP 25+ (prevent_overlapping_partitions, allow/1, mesh). Document findings and next steps in LOCKS_LEADER_IDEAS.md. Structured netsplit tests pass; random_netsplits still fails under extreme churn.
Replace gen_server+safe_loop with gen_statem states candidate/following/ syncing/leader. Add locks_pg society discovery, async sync-lock merge, postpone for early leader_calls, and netsplit-aware transitions. Structured CT suite is green; random_netsplits remains quarantined. Document design in LOCKS_LEADER_IDEAS.md.
Require same-pid leader consensus after each start_incremental join, and add a scripted late_join case (stabilize 3, add 4th, state catch-up). Also calibrate random_netsplits (100 ops, 1s rejoin settle) for local runs.
Drop the intermediate 1000-event ETS window (silent truncation on long runs) in favour of a single-pass stream. Add format_events/1,2 for a one-line-per-event greppable timeline (no state dumps); CT with_trace always writes that next to the raw ttb dir. Full format/1,2 still dumps pretty state when needed.
Matrix build on ubuntu-24.04 with rebar3: compile, eunit, and the structured CT suite (random_netsplits stays quarantined). Upload CT logs on failure for multi-node post-mortems.
Suites only use locks, gdict/test_cb, and locks_leader; the leader app from locks-test was never referenced. Remove the test-profile git dep (local _checkouts link was untracked and already removed).
OTP 29 CI failed g_3.gdict_netsplit when post-heal consensus arrived at ~6s while wait helpers gave up at 5s; exhausted retry returned a non-list and lists:usort raised opaque function_clause. Raise wait budget to 10s, error on retry_exhausted, and re-raise with original stack from with_trace.
maybe_connect called net_kernel:connect/1, which is gone on modern OTP. The proxy swallowed undef, so joining peers never connected, elected alone with empty state, and find(a) timed out. Mesh via connect_nodes, assert visibility, and wait for leader consensus before state checks.
Fix real issues (opaque gen_server reply tags, dead down/2 clause, cb_return missing noreply) and extend PLTs so ttb/ct/eunit/dbg are in scope under the test profile. Add both dialyzer runs to CI.
rebar3 aliases cannot span profiles, so orchestrate compile, default and test dialyzer, eunit, and ct from make test (alias: make ci).
ttb leaves a binary config in the working directory after CT tracing; delete any local copy and keep the tree clean after test runs.
OTP 28 CI failed g_5 when Rest (3 nodes) was still mid-election: usort(info leaders) was [undefined, Pid]. Elect each island fully before stores/lookups; use wait_find with the full wait budget for state checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP. Added a random_netsplits test case to the
locks_leader_SUITE. It currently fails. I've been working on improving thelocks_ttbtracing support, and it now works much better for long-running test cases (dropping the initial trace data and keeping the last 1000 events). A pretty-printed trace with an emacs mode header is generated for each failing test case.The random_netsplits test case is greatly inspired by the locks-test project by @ten0s but I decided to write my own to achieve better integration into the existing test suite.