builder: declare the BSD socket API in the darwin libSystem stub - #5636
Open
yohimik wants to merge 5 commits into
Open
builder: declare the BSD socket API in the darwin libSystem stub#5636yohimik wants to merge 5 commits into
yohimik wants to merge 5 commits into
Conversation
This was referenced Aug 30, 2026
The third parameter of fcntl is variadic, and on darwin/arm64 a variadic argument goes on the stack and not in a register. A call to libc fcntl through a plain three-argument function pointer thus makes the callee read that argument from an unrelated stack slot. open() already has a C wrapper for the same reason. The symptom is quiet. fcntl(fd, F_SETFD, FD_CLOEXEC) sets the flag or does not, which depends on the stack contents, so the result is the same for one binary and different between binaries. syscall.CloseOnExec is the main caller, so when it fails, every descriptor of the program goes into every process that it starts. A child that holds a copy of the write end of a pipe keeps that pipe from a report of EOF, which is how os/exec collects the output of a command. Measured on macOS 26.6 arm64 before this change, fcntl(fd, F_DUPFD, 100) returns EINVAL, and three F_SETFL calls with 0x4, 0x0 and 0x8 all leave F_GETFL with 0x48. The wrapper takes the argument as a uintptr_t so that the pointer commands reached through syscall.fcntlPtr use it too. Both spellings go through libc_fcntl_trampoline, and on a little-endian target the int commands read the low half of the same stack slot. The new tests in src/os cover both shapes. TestFcntlSetNonblock fails on darwin before this change and passes after it.
The process layer was a stub. StartProcess refused every ProcAttr that carried Dir, Sys or Files, and os/exec always passes three Files, so no command could run. Wait, Kill and Signal returned ErrNotImplemented, and ProcessState was an empty struct whose methods all reported a failure. The code that did exist was a fork() and an execve() with no branch on the result of the fork, so the parent fell into the exec as well. Use posix_spawn(3) on hosted Linux and macOS, which are the two targets where the standard library os/exec and syscall packages compile against this override. Those targets run the threads scheduler and collect with Boehm, so a fork from Go gives the child one thread that holds the locks of the other threads, malloc among them, and the stop-the-world signal of the collector can arrive between the fork and the exec. posix_spawn does the clone and the exec inside libc, where no Go code runs, and it reports a failed exec as its return value, so the usual status pipe is not necessary. The descriptors of the child come from a file-actions list. There is a dup2 for each entry of ProcAttr.Files, a close for a missing one, and an addchdir_np for Dir. A nil Env means the environment of the parent, as Go documents. The attribute block installs an empty signal mask, because a blocked mask survives an exec and the spawning thread can carry the signal of the collector blocked. Setpgid and Pgid are honoured through posix_spawnattr_setpgroup, which is the one SysProcAttr request that posix_spawn can express. Every other field is refused by name, and the error unwraps to ErrNotImplementedSys. Wait reaps with wait4 and retries on EINTR, which a thread in wait4 gets as a matter of course, because the collector interrupts it. ProcessState now carries the pid and the real syscall.WaitStatus, so exec.ExitError reports "exit status N", and ExitCode, Exited, Success and Sys work. A killed child is reported as signalled. Signal refuses a pid that Wait reaped and maps ESRCH to ErrProcessDone, which is what exec.CommandContext expects when its context fires as the command finishes. macOS has no pipe2, so os.Pipe there marks both descriptors close-on-exec afterwards, under ForkLock. Without the flag every pipe goes into every child, and a child that holds a copy of a write end keeps that pipe from a report of EOF. Linux asks for O_CLOEXEC in pipe2 and gets it atomically. The minimal macOS SDK in lib/macos-minimal-sdk does not declare <spawn.h>, so the generated libSystem stub has none of the posix_spawn symbols and a darwin program that starts a process does not link. The builder now assembles the missing names into a second stub object. posix_spawn_file_actions_addchdir_np came with macOS 10.15, so a binary from this toolchain needs at least that release. Targets without a process model keep the previous stubs. Only the build tag on exec_other.go changes, to let macOS through to the new implementation.
The runtime had weak.runtime_registerWeakPointer but not its counterpart, so a program that reads a weak pointer back did not link. crypto/tls does this in the certificate cache that it keeps behind a weak.Pointer. Weak pointers are not weak here. registerWeakPointer returns the pointer that it got, so the value it refers to stays and the way back to a strong pointer is the identity too. weak.Pointer.Value thus never reports a collected value, which the documented contract permits. testdata/weak.go does not link on the current dev branch and prints the expected value with this change.
TinyGo replaces crypto/tls with a stub whose handshake does nothing, so a program that dials https gets a plaintext connection behind the TLS API. That stub is correct for a target with no OS below it, which has neither the code size for a full TLS implementation nor usually a socket to speak it over. Hosted linux and macOS have both, and there the crypto/tls of the Go standard library compiles and runs. Make the override conditional. Without an entry in the map, crypto/tls falls under the "crypto/" merge, which links the package of the standard library into the synthetic GOROOT. GOOS alone cannot decide this, because a baremetal target reports GOOS=linux, so the build tags decide as well. The goroot cache key is a hash of the merge links, so the two variants get separate cache entries. testdata/hostcryptotls.go does a TLS handshake over an in-memory pipe with a certificate that it makes at run time. On the current dev branch it prints "negotiated an unexpected version: 0", because the stub does no handshake. With this change the handshake completes, the data goes through, and a client that does not trust the certificate refuses it. loader/goroot_test.go covers the targets that keep the stub, the baremetal one that reports GOOS=linux included.
The stub libSystem.dylib that TinyGo links a darwin binary against comes from lib/macos-minimal-sdk, whose generator reads a fixed list of headers that does not have <sys/socket.h>. socket, bind, connect, setsockopt and the rest are thus absent, and a darwin program that opens a socket through the standard library syscall package does not link. libSystem exports them. Add the names to the same extra-symbol list that the posix_spawn family uses.
yohimik
force-pushed
the
upstream-pr/darwin-socket-symbols
branch
from
September 2, 2026 08:50
702b4e8 to
8153e9a
Compare
Author
|
Rebased on dev after the 0.42.0 release. The change applies on top of v0.42.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
builder: declare the BSD socket API in the darwin libSystem stub
Last PR of the series. See "Dependencies".
What this does
The stub
libSystem.dylibthat TinyGo links a darwin binary against comes fromlib/macos-minimal-sdk, whose generator reads a fixed list of headers that doesnot have
<sys/socket.h>.socket,bind,connect,setsockoptand therest are thus absent, and a darwin program that opens a socket through the
standard library
syscallpackage does not link.libSystem exports them. This adds the names to the same extra-symbol list that
the
posix_spawnfamily uses, which theos: start processes with posix_spawnPR introduces.
Why it is last
This is the piece that makes host networking work on darwin, and it only has an
effect once three other things are in place.
upstream-pr/os-exec-posix-spawnintroducesdarwinExtraLibSystemSymbolsand the second stub object file. If that PR is not taken first, this PR must
carry that infrastructure instead. It is about 30 lines.
upstream-pr/hosted-crypto-tlsgives hosted targets the realcrypto/tls,which is what an HTTPS client needs.
src/netsubmodule must point at a tinygo-org/net commit that has thedarwin host netdev and the darwin trust roots. See the four tinygo-org/net
PRs in this series.
TODO before this can merge
The
src/netsubmodule pin is not updated in this branch. It still pointsat the current commit. Once the tinygo-org/net PRs land, this PR needs a second
commit that moves
src/netto the merged tinygo-org/net commit. Do not mergethis PR until that commit is present, or darwin still has no netdev and the new
symbols are unused.
Evidence
There is no test in this PR on its own, because the symbols are only reachable
once the submodule carries the darwin netdev. Once the pin moves,
TEST_PACKAGES_DARWINalready includesnet, somake tinygo-testin themacOS CI job exercises it with no makefile change.
With the fork that carries the full series, on macOS 26.6 arm64, a program that
calls
net.LookupHostandhttp.Get("https://example.com/")builds andcompletes.
A downstream product ships binaries built with these changes in a production
release. dispat v1.4.0 is published and is not a prerelease. It carries
dispat-tiny-linux-amd64anddispat-tiny-linux-arm64, built by the forkrelease v0.42.0-net.4 from sha256-pinned tarballs and smoke-executed under
binfmt before upload, beside six binaries from the gc toolchain.
https://github.com/yohimik/dispat/releases/tag/services%2Fdispat%2Fv1.4.0
The acceptance record of that repository is committed at
packages/docs/docs/internals/tinygo.md. It reports the net.2 to net.4acceptance history, an integration suite of 694 rows that passes with 0 failures
and 1 documented skip on darwin, and a size table of 0.58x to 0.63x against the
gc equivalents with TinyGo
-opt=z -no-debugagainstgo build -trimpath -ldflags "-s -w". Those figures come from that document. They are not ameasurement of this branch.
The suite exercises file I/O, environment variables, process spawning with
pipes, goroutine concurrency under the threads scheduler, net, TLS and x509, and
time and context handling. The self-update path runs over real TLS against a
live host, with negative rows. An unknown CA is refused, and a plaintext server
on a TLS port is refused. The shipped binaries in that release are for linux.
The darwin evidence is the acceptance record and the checks above.
Known gaps
SO_RCVTIMEOandSO_SNDTIMEO, and a call thatEINTR restarts begins its timeout again, so a deadline can be later than it
should be under a heavy GC load.
http.Client.Timeoutis inert in the tinygo-org/net client.behaviour is unchanged.
Related
netpackage with the Go one, which needs a netpoller. This series keeps thenetdev abstraction of tinygo-org/net. Under the threads scheduler a blocking
syscall blocks one thread and the others keep running, so no netpoller is
needed.
Enables
A command line program that needs an HTTPS client path on macOS. Self-update, an
API client, a webhook sender. Client side only. There is no server claim here
beyond what the tests show.
Related pull requests
This change is part of one body of work. Together the changes make programs that use the network and child processes work on hosted linux and macOS. A full CLI was tested end to end with all of them and ships binaries built this way, see dispat v1.4.0 in the evidence section.
In this repository
In tinygo-org/net
A merge order that works. The three bug fixes are independent. #5633 goes before #5635. HTTPS on linux needs only #5633 and #5635. Full darwin support also needs #5636, the net changes and a new src/net submodule pin.