From 69971b272d4da7a11197c1e9276df95989cb6cfc Mon Sep 17 00:00:00 2001 From: Matee ullah Malik Date: Thu, 30 Jul 2026 20:41:21 +0000 Subject: [PATCH 1/2] fix(devnet): resolve upgrade script paths relative to the script, not the caller Three defects that together made `devnet/scripts/upgrade.sh` unusable for a rehearsal driven from the repository root, which is how the upgrade flow is documented and invoked. 1. CWD-RELATIVE COMPOSE FILE (root cause) ----------------------------------------- submit-upgrade-proposal.sh and vote-all.sh both hardcoded COMPOSE_FILE="../docker-compose.yml" That only resolves when the script is invoked from devnet/scripts/. Run from the repository root, every `docker compose -f "$COMPOSE_FILE" exec ...` call failed, so key_address() and account_exists() returned empty. The scripts then misreported perfectly healthy accounts: Governance helper account lumera1kd5... is not present on-chain Unable to resolve fallback proposer key supernova_validator_1_key Both claims were false. The account held 1,000,000 LUME and the key was in the container keyring; only the compose lookup was broken. The failure mode is especially costly because the message points at governance/keyring state and sends you debugging the chain instead of the path. Fixed by resolving from BASH_SOURCE, with an env override retained: SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" COMPOSE_FILE="${COMPOSE_FILE:-${SCRIPT_DIR}/../docker-compose.yml}" 2. LOCALLY-BUILT BINARIES CANNOT PASS THE VERSION PROBE ------------------------------------------------------- upgrade-binaries.sh refuses to proceed unless ` version` prints a non-empty string. Binaries built from a branch with plain `go build` (no -ldflags) print nothing, so rehearsing an unreleased branch was impossible: Failed to determine version for binary: .../devnet/bin-v1.20.2/lumerad Release artifacts self-report and are unaffected. Added a DEVNET_BINARY_VERSION override plus an actionable error that names it. Confirmed empirically: release v1.20.1 prints "1.20.1"; a local branch build prints "". 3. VOTE SKIPPED ON FEE TOP-UP SEQUENCE MISMATCH ----------------------------------------------- Observed during the end-to-end run: supernova_validator_2: fee top-up failed: account sequence mismatch, expected 2, got 1: incorrect account sequence Skipping vote for supernova_validator_2 because fee top-up failed Back-to-back top-ups from the same funding account race the sequence number, and the script silently drops that validator's vote (4 of 5 voted). The proposal still passed here, but on a network where one validator carries decisive weight this would silently change the outcome. NOT fixed in this PR - it needs a sequence-aware retry and deserves its own change. Documented so the next person does not rediscover it from scratch. EVIDENCE -------- Same script, same devnet, same arguments, differing only in working directory: from devnet/scripts/ ... proposer resolved, proceeded to vote (worked) from repository root ... "not present on-chain", exit 1 (failed) That isolates CWD as the variable. After the fix, from the repository root, on a fresh 5-validator devnet at v1.20.1 upgrading to a branch-built v1.20.2: Governance proposer: governance_key (lumera1rtk...) <- previously failed Total Votes Cast: 4000000000000 Binaries upgrade complete Upgrade to v1.20.2 initiated successfully exit code 0 q upgrade applied v1.20.2 -> height 121 audit module version ..... 3 all 5 validators ......... height 175, in lockstep CORRECTION ---------- An earlier report of mine claimed upgrade.sh "exits 0 on failure". That was my measurement error: I read the exit code after piping stdout to `tail`, so I was reading tail's status. upgrade.sh sets `set -euo pipefail` and correctly returns 1. No change was needed and none was made. RISKS ----- Shell-only, confined to devnet tooling. No chain code, no state machine, no consensus path. The env-var indirection preserves any existing COMPOSE_FILE override. ROLLBACK -------- Revert. The scripts return to working only when invoked from devnet/scripts/. --- devnet/scripts/submit-upgrade-proposal.sh | 9 ++++++++- devnet/scripts/upgrade-binaries.sh | 13 +++++++++++++ devnet/scripts/vote-all.sh | 8 +++++++- 3 files changed, 28 insertions(+), 2 deletions(-) diff --git a/devnet/scripts/submit-upgrade-proposal.sh b/devnet/scripts/submit-upgrade-proposal.sh index 71b7819d..fdc74f43 100755 --- a/devnet/scripts/submit-upgrade-proposal.sh +++ b/devnet/scripts/submit-upgrade-proposal.sh @@ -19,7 +19,14 @@ LUMERA_SHARED="/tmp/${CHAIN_ID}/shared" KEYRING="test" HOST_PROPOSAL_FILE="${LUMERA_SHARED}/upgrade_${VERSION}.json" CONTAINER_PROPOSAL_FILE="/shared/upgrade_${VERSION}.json" -COMPOSE_FILE="../docker-compose.yml" +# Resolve paths relative to THIS script, not the caller's working directory. +# Previously this was "../docker-compose.yml", which only resolved when the +# script happened to be invoked from devnet/scripts/. Run from the repository +# root (as documented for the rehearsal flow), every `docker compose -f` call +# silently failed, so key/account lookups returned empty and the script +# misreported working accounts as "not present on-chain". +SUBMIT_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +COMPOSE_FILE="${COMPOSE_FILE:-${SUBMIT_SCRIPT_DIR}/../docker-compose.yml}" STATUS_DIR="${LUMERA_SHARED}/status" ACCOUNT_REGISTRY_FILE="${STATUS_DIR}/${SERVICE}/accounts.json" diff --git a/devnet/scripts/upgrade-binaries.sh b/devnet/scripts/upgrade-binaries.sh index da64fd86..69c257ff 100755 --- a/devnet/scripts/upgrade-binaries.sh +++ b/devnet/scripts/upgrade-binaries.sh @@ -42,10 +42,23 @@ binary_version() { return 1 fi + # Allow an explicit override. Binaries built locally from a branch (plain + # `go build`, no -ldflags) print an EMPTY version string, so probing them + # can never succeed. That is the normal case when rehearsing an unreleased + # branch, so refusing to proceed would make branch rehearsals impossible. + # Release artifacts still self-report and need no override. + if [[ -n "${DEVNET_BINARY_VERSION:-}" ]]; then + printf '%s\n' "$(normalize_version "${DEVNET_BINARY_VERSION}")" + return 0 + fi + version="$("${binary}" version 2>/dev/null | head -n 1 | tr -d '\r')" version="$(normalize_version "${version}")" if [[ -z "${version}" ]]; then echo "Failed to determine version for binary: ${binary}" >&2 + echo " Binaries built with plain 'go build' report no version." >&2 + echo " Set DEVNET_BINARY_VERSION= to supply it explicitly, e.g." >&2 + echo " DEVNET_BINARY_VERSION=v1.20.2 ./devnet/scripts/upgrade.sh v1.20.2 auto-height devnet/bin-v1.20.2" >&2 return 1 fi printf '%s\n' "${version}" diff --git a/devnet/scripts/vote-all.sh b/devnet/scripts/vote-all.sh index d325aa14..2930843c 100755 --- a/devnet/scripts/vote-all.sh +++ b/devnet/scripts/vote-all.sh @@ -11,7 +11,13 @@ CHAIN_ID="lumera-devnet-1" KEYRING_BACKEND="test" PROPOSAL_ID="$1" SERVICE_NAME="supernova_validator_1" -COMPOSE_FILE="../docker-compose.yml" +# Resolve relative to THIS script, not the caller's working directory. +# See submit-upgrade-proposal.sh for the full rationale: a caller-relative +# path made every `docker compose -f` call fail when the script was invoked +# from anywhere other than devnet/scripts/, which surfaced as bogus +# "no votes available" / "unable to resolve key" errors. +VOTE_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +COMPOSE_FILE="${COMPOSE_FILE:-${VOTE_SCRIPT_DIR}/../docker-compose.yml}" FEES="5000ulume" # Gas configuration — use a fixed gas amount by default. `--gas auto` simulates # a gov vote at ~57.9k and even with a 1.3x bump lands right at the real usage From 21b8e9d41c42c502fe46ba48014a8487fcedf6f9 Mon Sep 17 00:00:00 2001 From: Matee ullah Malik Date: Fri, 31 Jul 2026 00:08:16 +0000 Subject: [PATCH 2/2] fix(devnet): two more upgrade-script defects found during the Phase 2 rehearsals MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both were found by running the scripts for real against mainnet-shaped devnets, not by reading them. Neither is theoretical. 1. HALT DETECTION ABORTED A CORRECT UPGRADE -------------------------------------------- Observed on the two-hop rehearsal (v1.12.0 -> v1.20.1). The chain had stopped correctly at the plan height and the logs said so: UPGRADE "v1.20.1" NEEDED at height: 207 but the script refused to continue: ERROR: chain is still producing blocks (height 207) and has NOT halted The upgrade did not execute — the proposal likely failed to pass (quorum/threshold). Both statements were false. The proposal had PASSED (4000000000000 yes, 0 no) and `q upgrade plan` showed the plan registered at 207. Root cause: a node stopped for an upgrade KEEPS SERVING RPC at the halt height. The guard treated any numeric height as proof of liveness, so height == plan height read as "still producing blocks". It is a liveness check written as a value check. Fixed by sampling the height twice and comparing: only an ADVANCING height means blocks are still being produced. A static height across two samples is the stopped state. The error message now points at the two queries that actually distinguish the cases instead of asserting a quorum failure that did not happen. This failed SAFE — it refused to swap binaries rather than swapping wrongly — but it turned a green rehearsal into a manual recovery. 2. SEQUENCE RACE SILENTLY DROPPED A VALIDATOR'S VOTE ----------------------------------------------------- Observed on BOTH Phase 2 rehearsals: ❌ supernova_validator_2: fee top-up failed (txhash: 8D157071...): account sequence mismatch, expected 2, got 1: incorrect account sequence ❌ Skipping vote for supernova_validator_2 because fee top-up failed 📈 Total Votes Cast: 4000000000000 (4 of 5) Every voter is topped up from the same funding account in a tight loop, so back-to-back sends race the account sequence: the second is broadcast before the first is committed. The caller then skips that validator's vote entirely. The proposal still passed in both runs, so this is easy to miss. On a network where one validator carries decisive weight, a silently dropped vote changes a governance outcome. Fixed with a bounded retry that matches the sequence-mismatch error specifically and waits for the prior send to commit (TOPUP_MAX_ATTEMPTS, TOPUP_RETRY_SECS). VERIFICATION ------------ bash -n ................. clean on both files shellcheck -S error ..... clean on both files Both defects were reproduced on real devnet runs before being changed, and the halt-detection fix is what allowed the two-hop rehearsal (v1.12.0 -> v1.20.1 -> v1.20.2) to be completed and compared against the single-hop path. RISK ---- Shell-only, devnet tooling. No chain code, no state machine, no consensus path. Both new behaviours are env-tunable and default to conservative values. ROLLBACK -------- Revert. The scripts return to aborting on a correct halt and silently dropping racing votes. --- devnet/scripts/upgrade.sh | 22 +++++++++++++++++----- devnet/scripts/vote-all.sh | 28 +++++++++++++++++++++++----- 2 files changed, 40 insertions(+), 10 deletions(-) diff --git a/devnet/scripts/upgrade.sh b/devnet/scripts/upgrade.sh index 3034a421..d790bd10 100755 --- a/devnet/scripts/upgrade.sh +++ b/devnet/scripts/upgrade.sh @@ -216,11 +216,23 @@ if ! detect_upgrade_halt; then LIVE_HEIGHT="$(docker compose -f "${COMPOSE_FILE}" exec -T "${SERVICE}" \ lumerad status 2>/dev/null | jq -r '.sync_info.latest_block_height // empty' 2>/dev/null || true)" if [[ "${LIVE_HEIGHT}" =~ ^[0-9]+$ ]]; then - echo "ERROR: chain is still producing blocks (height ${LIVE_HEIGHT}) and has NOT halted for the ${RELEASE_NAME} upgrade." >&2 - echo " The upgrade did not execute — the proposal likely failed to pass (quorum/threshold)." >&2 - echo " Inspect: lumerad query gov proposal ${PROPOSAL_ID}" >&2 - echo " Refusing to swap binaries; running ${RELEASE_NAME} on un-upgraded state would crash all nodes." >&2 - exit 1 + # A node that has stopped for an upgrade KEEPS SERVING RPC at the halt + # height, so a numeric height is not proof the chain is live. Sample twice + # and compare: only an ADVANCING height means blocks are still being made. + # Without this the guard aborted correct upgrades whose halt height had + # just been reached (observed on the two-hop rehearsal at height 207). + sleep "${HALT_CONFIRM_SECS:-12}" + LIVE_HEIGHT_2="$(docker compose -f "${COMPOSE_FILE}" exec -T "${SERVICE}" \ + lumerad status 2>/dev/null | jq -r '.sync_info.latest_block_height // empty' 2>/dev/null || true)" + if [[ "${LIVE_HEIGHT_2}" =~ ^[0-9]+$ ]] && ((LIVE_HEIGHT_2 > LIVE_HEIGHT)); then + echo "ERROR: chain is still producing blocks (${LIVE_HEIGHT} -> ${LIVE_HEIGHT_2}) and has NOT stopped for the ${RELEASE_NAME} upgrade." >&2 + echo " Check the proposal actually PASSED and that its plan height is in the future:" >&2 + echo " lumerad query gov proposal ${PROPOSAL_ID}" >&2 + echo " lumerad query upgrade plan" >&2 + echo " Refusing to swap binaries; running ${RELEASE_NAME} on un-upgraded state would crash all nodes." >&2 + exit 1 + fi + echo "Height static at ${LIVE_HEIGHT} across two samples — chain is stopped at the upgrade boundary; proceeding." fi echo "⚠️ No upgrade-halt marker found and RPC is unreachable; assuming a genuine halt and proceeding." >&2 fi diff --git a/devnet/scripts/vote-all.sh b/devnet/scripts/vote-all.sh index 2930843c..778c2b85 100755 --- a/devnet/scripts/vote-all.sh +++ b/devnet/scripts/vote-all.sh @@ -119,11 +119,29 @@ ensure_fee_funds() { return 0 fi echo " 💧 $svc voter has ${spendable}ulume spendable (< ${FEE_MIN}); topping up ${FEE_TOPUP} from ${SERVICE_NAME}/${PRIMARY_FUNDING_KEY}" - topup_json=$(docker compose -f "$COMPOSE_FILE" exec -T "$SERVICE_NAME" \ - lumerad tx bank send "$PRIMARY_FUNDING_KEY" "$addr" "$FEE_TOPUP" \ - --keyring-backend "$KEYRING_BACKEND" --chain-id "$CHAIN_ID" \ - --gas "$GAS_AMOUNT" --gas-prices 0.025ulume -y -o json 2>&1) - topup_rc=$? + # Every voter is topped up from the SAME funding account in a tight loop, so + # back-to-back sends race the account sequence: the second is broadcast before + # the first is committed and fails with "account sequence mismatch, expected N, + # got N-1". The caller then SKIPS that validator's vote entirely — observed + # twice during the Phase 2 rehearsals, each time dropping one of five votes + # (4000000000000 instead of 5000000000000). The proposal still passed there, + # but on a network where one validator carries decisive weight this silently + # changes a governance outcome. Retry on sequence mismatch specifically. + local attempt=1 max_attempts="${TOPUP_MAX_ATTEMPTS:-4}" + while :; do + topup_json=$(docker compose -f "$COMPOSE_FILE" exec -T "$SERVICE_NAME" \ + lumerad tx bank send "$PRIMARY_FUNDING_KEY" "$addr" "$FEE_TOPUP" \ + --keyring-backend "$KEYRING_BACKEND" --chain-id "$CHAIN_ID" \ + --gas "$GAS_AMOUNT" --gas-prices 0.025ulume -y -o json 2>&1) + topup_rc=$? + if echo "$topup_json" | grep -q "account sequence mismatch" && [ "$attempt" -lt "$max_attempts" ]; then + echo " ⏳ $svc: sequence mismatch on top-up (attempt ${attempt}/${max_attempts}); waiting for the previous send to commit" + sleep "${TOPUP_RETRY_SECS:-6}" + attempt=$((attempt + 1)) + continue + fi + break + done if [ "$topup_rc" -ne 0 ]; then echo " ❌ $svc: fee top-up command failed with exit code $topup_rc" >&2 echo "$topup_json" >&2