CircleCI build on Darwin (arm64) frequently segfaults and sometimes deadlocks

[toastwaffle]

There seems to be a data race which only triggers on darwin_arm64. This is particularly bad, because sometimes Please doesn't detect it and instead gets deadlocked, and then spends 3 hours burning CircleCI credits to do nothing

WARNING: DATA RACE
Write at 0x00c000229b30 by goroutine 55:
  runtime.recvDirect()
      /Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/runtime/chan.go:405 +0x7c
  github.com/thought-machine/please/src/output.(*buildingTargets).handleOutput.(*BuildState).Stop.func1()
      /Users/distiller/project/src/core/state.go:419 +0x40
  sync.(*Once).doSlow()
      /Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/sync/once.go:78 +0x94
  sync.(*Once).Do()
      /Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/sync/once.go:69 +0x40
  github.com/thought-machine/please/src/core.(*BuildState).Stop()
      /Users/distiller/project/src/core/state.go:418 +0x2e4
  github.com/thought-machine/please/src/output.(*buildingTargets).handleOutput()
      /Users/distiller/project/src/output/targets.go:107 +0x288
  github.com/thought-machine/please/src/output.(*buildingTargets).ProcessResult.deferwrap1()
      /Users/distiller/project/src/output/targets.go:84 +0x34
  runtime.deferreturn()
      /Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/runtime/panic.go:668 +0x5c
  github.com/thought-machine/please/src/output.MonitorState()
      /Users/distiller/project/src/output/shell_output.go:57 +0x7c0
  main.runPlease.func1()
      /Users/distiller/project/src/please.go:1230 +0x94
Previous read at 0x00c000229b30 by goroutine 17861:
  runtime.chansend1()
      /Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/runtime/chan.go:162 +0x2c
  github.com/thought-machine/please/src/core.(*BuildState).addPendingParse.func1()
      /Users/distiller/project/src/core/state.go:355 +0xec
Goroutine 55 (running) created at:
  main.runPlease()
      /Users/distiller/project/src/please.go:1229 +0x78c
  main.Please()
      /Users/distiller/project/src/please.go:1197 +0xe68
  main.runBuild()
      /Users/distiller/project/src/please.go:1341 +0x200
  main.init.func1()
      /Users/distiller/project/src/please.go:478 +0x54
  main.execute()
      /Users/distiller/project/src/please.go:1623 +0x6a0
  main.main()
      /Users/distiller/project/src/please.go:1627 +0x44
Goroutine 17861 (finished) created at:
  github.com/thought-machine/please/src/core.(*BuildState).addPendingParse()
      /Users/distiller/project/src/core/state.go:351 +0x190
  github.com/thought-machine/please/src/core.(*BuildState).queueTarget()
      /Users/distiller/project/src/core/state.go:1100 +0x4bc
  github.com/thought-machine/please/src/core.(*BuildState).queueTargetAsync()
      /Users/distiller/project/src/core/state.go:1181 +0x154
  github.com/thought-machine/please/src/core.(*BuildState).queueResolvedTarget.func1.gowrap1()
      /Users/distiller/project/src/core/state.go:1161 +0x58
==================
==================
WARNING: DATA RACE
Write at 0x00c000229c10 by goroutine 55:
  runtime.recvDirect()
      /Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/runtime/chan.go:405 +0x7c
  github.com/thought-machine/please/src/output.(*buildingTargets).handleOutput.(*BuildState).Stop.func1()
      /Users/distiller/project/src/core/state.go:420 +0x58
  sync.(*Once).doSlow()
      /Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/sync/once.go:78 +0x94
  sync.(*Once).Do()
      /Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/sync/once.go:69 +0x40
  github.com/thought-machine/please/src/core.(*BuildState).Stop()
      /Users/distiller/project/src/core/state.go:418 +0x2e4
  github.com/thought-machine/please/src/output.(*buildingTargets).handleOutput()
      /Users/distiller/project/src/output/targets.go:107 +0x288
  github.com/thought-machine/please/src/output.(*buildingTargets).ProcessResult.deferwrap1()
      /Users/distiller/project/src/output/targets.go:84 +0x34
  runtime.deferreturn()
      /Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/runtime/panic.go:668 +0x5c
  github.com/thought-machine/please/src/output.MonitorState()
      /Users/distiller/project/src/output/shell_output.go:57 +0x7c0
  main.runPlease.func1()
      /Users/distiller/project/src/please.go:1230 +0x94
Previous read at 0x00c000229c10 by goroutine 17897:
  runtime.chansend1()
      /Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/runtime/chan.go:162 +0x2c
  github.com/thought-machine/please/src/core.(*BuildState).addPendingBuild.func1()
      /Users/distiller/project/src/core/state.go:366 +0x6c
Goroutine 55 (running) created at:
  main.runPlease()
      /Users/distiller/project/src/please.go:1229 +0x78c
  main.Please()
      /Users/distiller/project/src/please.go:1197 +0xe68
  main.runBuild()
      /Users/distiller/project/src/please.go:1341 +0x200
  main.init.func1()
      /Users/distiller/project/src/please.go:478 +0x54
  main.execute()
      /Users/distiller/project/src/please.go:1623 +0x6a0
  main.main()
      /Users/distiller/project/src/please.go:1627 +0x44
Goroutine 17897 (finished) created at:
  github.com/thought-machine/please/src/core.(*BuildState).addPendingBuild()
      /Users/distiller/project/src/core/state.go:362 +0xe4
  github.com/thought-machine/please/src/core.(*BuildState).queueTargetAsync()
      /Users/distiller/project/src/core/state.go:1216 +0x50c
  github.com/thought-machine/please/src/core.(*BuildState).queueResolvedTarget.func1.gowrap1()
      /Users/distiller/project/src/core/state.go:1161 +0x58

[peterebden]

Strange that this triggers only on darwin_arm64. Also strange how it reports the data race - the 'write' line is closing a channel, the 'read' side is writing an item to that channel? Maybe it is trying to warn us about racing the two (which is legit, although we somewhat messily handle the potential panic).
It might be quite hard to fix that issue though without some major surgery on those core channels. I had some ideas about this some time back but never finished it (not sure if it is on a branch on my fork or lost to the mists of history now).

[sean-]

Maybe related? #3512 I chased a few of these a while back but all of my PRs are still sitting there so I haven't made any noise about them recently.