Alias all LFS64 symbols to their non-LFS64 counterparts on musl by bossmc · Pull Request #2935 · rust-lang/libc

bossmc · 2022-09-30T23:05:23Z

As per #2934 the LFS64 symbols on musl-libc are simply aliases to the non-LFS64 symbols. Currently this is done both in the header files (as #define entries) and in the library (as aliasing symbols). There is a desire in musl to drop the ABI compatibility shims (the symbol aliases) - currently the libc crate exports the LFS64 symbols by extern-ing the compatibility shims which will fail if musl removes them.

This changes the musl build of libc to replicate the aliasing that's in the C header files (with pub use xxx as xxx64) so the API from the libc crate is unchanged, but the crate is now compatible with the upcoming musl release.

I've also checked and all the LFS64 types (e.g. off64_t) are already the same as their non-LFS64 equivalents.

This is an annoying change to test, libc-test seems expect to build against musl 1.1.24 (in fact, does Rust even support musl 1.2? It's not obvious that it does... e.g. see

libc/src/unix/linux_like/linux/musl/mod.rs

Lines 288 to 292 in d99c37d

    
           // FIXME: musl added paddings and adjusted 
        
           // layout in 1.2.0 but our CI is still 1.1.24. 
        
           // So, I'm leaving some fields as comments for now. 
        
           // ref. https://github.com/bminor/musl/commit/ 
        
           // 1e7f0fcd7ff2096904fd93a2ee6d12a2392be392

)

rust-highfive · 2022-09-30T23:05:26Z

r? @Amanieu

(rust-highfive has picked a reviewer for you, use r? to override)

JohnTitor · 2022-10-02T22:52:19Z

This is a breaking change, we have to deprecate them on musl first otherwise user code depending on them would be broken.

bossmc · 2022-10-02T23:10:12Z

Unless I've missed something, this isn't a breaking change (at a crate API level at least) - all the types and functions that used to exist still do and still have the same name/arguments/structure/alignment etc. The only difference is that when rust code calls xxx64(*const xxx64_t) it will actually call the xxx function in the musl library (which it was already doing since the xxx64 symbol in the library was a link-time alias to the xxx symbol).

I've done a bit more work on this (ensuring that the complex types are rust-level aliases to each other on musl so they can be passed to either function without failing type checks), though I'm not sure it's complete (I've not done this for the structs in musl's b64 or b32 modules yet - and I'm unsure what needs to happen in the b32 case).

JohnTitor · 2022-10-03T22:40:06Z

+// Musl's standard entrypoints are already LFS64 compatible, historically the library aliased
+// these together in header files (as `#define`s) _and_ in the library with weak symbol aliases.
+//
+// Since <version> these aliases were removed from the library (both in the API and the ABI) so we
+// alias them here to keep the crate API stable.
+#[allow(dead_code)]
+fn check_type_aliases(
+    dirent: ::dirent,
+    ino: ::ino_t,
+    flock: ::flock,
+    off: ::off_t,
+    pos: ::fpos_t,
+    rlimit: ::rlimit,
+    stat: ::stat,
+    statfs: ::statfs,
+    statvfs: ::statvfs,
+) {
+    let _dirent: ::dirent64 = dirent;
+    let _ino: ::ino64_t = ino;
+    let _flock: ::flock64 = flock;
+    let _off: ::off64_t = off;
+    let _pos: ::fpos64_t = pos;
+    let _rlimit: ::rlimit64 = rlimit;
+    let _stat: ::stat64 = stat;
+    let _statfs: ::statfs64 = statfs;
+    let _statvfs: ::statvfs64 = statvfs;
+}


This could be moved to test, I guess?

Could be, though this is merely a compile-time check that the interchangeable types are aliases (rather than identically defined separate types) it's not a test in the classical sense.

An alternative might be to #[cfg(test)] this function?

JohnTitor · 2022-10-03T22:44:39Z

Ah, I see! Thanks for clarifying. Could you fix the style failure and squash commits?

This is an annoying change to test, libc-test seems expect to build against musl 1.1.24 (in fact, does Rust even support musl 1.2

No, we don't support musl 1.2.0 or higher as it contains the time64 change. We've tried to update and the discussion is ongoing: #2088
I'm not sure how it's related to this change though, does it matter?

bossmc · 2022-10-03T23:07:24Z

No, we don't support musl 1.2.0 or higher as it contains the time64 change. We've tried to update, and the discussion is ongoing: #2088 I'm not sure how it's related to this change though, does it matter?

There's a proposed patch available today to remove the symbols from the musl library on the 1.2.3 branch, to test this change has sufficiently removed the references to the LFS64 symbols, I plan to use that patched library. Unfortunately, this also entails making a few changes to the libc crate (e.g. fixing up that linked struct definition) to get the tests passing which are changes that don't want to go into the crate while it's targeting 1.1.

Maybe this PR should be deferred until @wesleywiser's 1.2 support lands - since these symbols won't be disappearing in the 1.1.x releases.

bors · 2023-02-12T04:34:43Z

☔ The latest upstream changes (presumably #3095) made this pull request unmergeable. Please resolve the merge conflicts.

colincross · 2023-03-30T18:17:59Z

I tested this against ToT musl that includes the LFS64 symbol removal (https://git.musl-libc.org/cgit/musl/commit/?id=246f1c811448f37a44b41cd8df8d0ef9736d95f4), and it required the attached addition to the second patch:
dirent64.patch.txt.

bossmc · 2023-04-13T18:10:20Z

Thanks @colincross I'd not noticed the extra-traits feature. Apparently there's a catchup merge/rebase needed too now 💯

Amanieu · 2023-05-20T13:25:16Z

Style check is failing, but otherwise this looks good to merge. It is not a breaking change.

bossmc · 2023-05-22T21:06:05Z

From the discussion in #3248 this probably needs some markups. In that discussion @joshtriplett would prefer the libc crate to expose LFS64 symbols even on Musl 1.2.4 so cross-libc-targeting crates don't need to use #[cfg] to find the 64-bit compatible file operations. This seems reasonable (if a little unusual, not following the model used elsewhere in the libc crate) but it exposes another few questions about how the libc crate exposes the aliased symbols.

Currently the implementation exposes aliased types with type <foo>64 = <foo> and aliased functions with use <foo> as <foo>64 this has a few impacts:

The two types are not distinct, so (e.g.) trait impls on them overlap where they wouldn't for e.g. glibc
The function type signatures are "wrong", e.g. the <foo>64 function takes *mut <bar> not *mut <bar>64 arguments
The aliased functions are "the same" function (e.g. casting them to raw pointers will give the same value)

The use of use is also fairly messy, use pulls the named symbol into the current module in every namespace (value, function, type) unless there's already a symbol of that name defined directly in the current module. This, combined with the name clash in stat()/stat64() vs struct stat/struct stat64 means that it's not immediately obvious where symbols are coming from and if they're correct.

An alternative approach would be for the musl backend to define each LFS64-affected type twice and to define shim functions of the form:

fn stat64(path: *const c_char, buf: *mut stat64) -> c_int {
  return stat(path, buf.cast());
}

Which assumes:

The inliner will do the right thing in release builds
The two structures are .cast()-able to each other which would want a regression test (or deterministic codegen?)

Thoughts?

Amanieu · 2023-05-23T09:16:30Z

I believe there was some confusion in the discussion in #3248. @joshtriplett was talking about time64, not LFS64.

You're right that changing the *64 types would be a breaking change due to possibe impls, and the alternative solution would be better.

bossmc · 2023-05-26T21:45:13Z

Ok, I've added the set of shim functions (in .../musl/lfs64.rs) and restored the duplicate definitions of the structs. I think this is now the correct shape for review!

bors · 2023-05-31T18:07:09Z

⌛ Testing commit 75ac488 with merge 1e8c55c...

bors · 2023-05-31T19:12:25Z

☀️ Test successful - checks-actions, checks-cirrus-freebsd-12, checks-cirrus-freebsd-13, checks-cirrus-freebsd-14
Approved by: Amanieu
Pushing 1e8c55c to main...

This fixes the build with musl-1.2.4, for those that are still experiencing issues these steps should resolve them: 1. Downgrade to musl-1.2.3 2. Rebuild dev-lang/rust with these patches 3. Upgrade to musl-1.2.4 again 4. Rebuild rust with USE=system-bootstrap At this point USE=system-boostrap will be required with >= musl-1.2.4 until uptream merges these patches and updates their boostrap. This was tested with musl-1.2.3, musl-1.2.4 and glibc-2.37-r3. Closes: https://bugs.gentoo.org/903607 Upstream-PR: rust-lang/rust#106246 Upstream-Issue: rust-lang/libc#2934 Upstream-PR: rust-lang/libc#2935 Upstream-PR: rust-random/getrandom#326 Upstream-Commit: rust-random/getrandom@7f73e3c Signed-off-by: orbea <orbea@riseup.net>

Unify definitions of `siginfo_t`, `statvfs` and `statfs` in `musl` targets During #2935 I noticed there were multiple identical definitions of some of the `bits/***.h` types in the musl target, as well as a few places where a type was defined twice in the module tree (leading to the "upper-most" definition being the one exported by the library, in contradiction to the expectation that the "most-specific" definition would be used. This change moves the definitions of `struct statvfs(64)` and `struct siginfo_t` to be global for all `musl` targets (see https://git.musl-libc.org/cgit/musl/tree/include/sys/statvfs.h and https://git.musl-libc.org/cgit/musl/tree/include/signal.h which are architecture-agnostic headers) and `struct statfs(64)` to be global for all 64-bit `musl` targets (see https://git.musl-libc.org/cgit/musl/tree/arch/generic/bits/statfs.h). This also required moving `fsblkcnt64_t` and `fsfilcnt64_t` to be musl-wide too (for use in `struct statvfs64`). It also removes a bunch of redundant (and unreachable) definitions in the `riscv64` and `riscv32` targets (there seems to be a `riscv32` target in the crate, but not in `musl` itself or at least there's no `arch/riscv32` folder in tree). Upshot of the above is that this change has no externally visible effect, if the more specific types were intended to be used they weren't being so removing them is a no-op. To actually use more specific type definitions one would need to `cfg` out the general definition as well as providing the specific one. <details> <summary>To find most of these issues I used this process</summary> ``` $ for target in $(rustc --print target-list | grep musl) do echo $target RUSTDOCFLAGS="--output-format json --document-private-items" cargo +nightly doc -Z build-std=core --target $target done $ for json in target/**/doc/libc.json do echo $json jq '.index[] | select(.inner | keys[0] == "struct") | .name' $json | sort | uniq -d done ``` The first command uses rustdoc to create a JSON representation of the API of the crate for each (`musl`) target and the second searches that output for two exported structs of the same name within a single target. Where there's a duplicate, only one of the two symbols is actually usable (and due to import rules, symbols defined locally take precedence over symbols imported from submodules so the less specific symbol is the one that wins). You can do similar tests for `enum`, `typedef`, `union`, constant` by changing the second command in the obvious way, you can also do the same for `function` though you need to additionally filter on `extern "C"` (since e.g. there's many many `clone` functions defined in the crate): ``` $ jq '.index[] | select(.inner | keys[0] == "function") | select(.inner.function.header.abi | (type == "object" and keys[0] == "C")) | .name' $json | sort | uniq -d ``` </details> It feels like adding the checks in that methodology to CI for each target would be a good way to catch issues where a more specific definition is masked by a less-specific one.

This fixes the build with musl-1.2.4, for those that are still experiencing issues these steps should resolve them: 1. Downgrade to musl-1.2.3 2. Rebuild dev-lang/rust with these patches 3. Upgrade to musl-1.2.4 again 4. Rebuild rust with USE=system-bootstrap At this point USE=system-boostrap will be required with >= musl-1.2.4 until uptream merges these patches and updates their boostrap. This was tested with musl-1.2.3, musl-1.2.4 and glibc-2.37-r3. Closes: https://bugs.gentoo.org/903607 Upstream-PR: rust-lang/rust#106246 Upstream-Issue: rust-lang/libc#2934 Upstream-PR: rust-lang/libc#2935 Upstream-PR: rust-random/getrandom#326 Upstream-Commit: rust-random/getrandom@7f73e3c Signed-off-by: orbea <orbea@riseup.net>

rust-highfive assigned Amanieu Sep 30, 2022

rust-highfive added the S-waiting-on-review label Sep 30, 2022

bossmc mentioned this pull request Sep 30, 2022

Rust binding to LFS64 ABI-compat symbols rather than public names on musl libc #2934

Closed

JohnTitor reviewed Oct 3, 2022

View reviewed changes

bossmc marked this pull request as draft October 3, 2022 22:43

JohnTitor added S-blocked and removed S-waiting-on-review labels Oct 6, 2022

This was referenced Feb 24, 2023

Use mmap64 on Linux. rust-lang/backtrace-rs#501

Merged

Non-LFS functions on 32-bit platforms rust-lang/rust#94173

Closed

bossmc force-pushed the alias-lfs64-symbols-on-musl branch from 328ec95 to f2ec9e0 Compare April 13, 2023 18:02

bossmc and others added 3 commits April 13, 2023 19:19

Alias all LFS64 symbols to their non-LFS64 counterparts on musl

b2d1051

Move musl-exclusions to cfg_if blocks and alias types on Musl too

b95fbee

Fix up duplicate derives with extra_traits

cd78c71

bossmc force-pushed the alias-lfs64-symbols-on-musl branch from f2ec9e0 to cd78c71 Compare April 13, 2023 18:29

12101111 mentioned this pull request May 6, 2023

undefined LFS64 symbol when link to musl 1.2.4 rust-lang/rust#111285

Closed

Tidy up build.rs now that all xxx64 are absent for the musl target

5719564

joshtriplett mentioned this pull request May 15, 2023

Tracking issue for libc 1.0 #3248

Open

19 tasks

Provide shim functions to backfill LFS64 ABI

eea022d

bors merged commit 1e8c55c into rust-lang:main May 31, 2023

bossmc deleted the alias-lfs64-symbols-on-musl branch May 31, 2023 19:16

bossmc mentioned this pull request Jun 1, 2023

Unify definitions of siginfo_t, statvfs and statfs in musl targets #3261

Merged

nikarh mentioned this pull request Jun 6, 2023

Improved std support for ps vita target rust-lang/rust#111819

Merged

kleisauke mentioned this pull request Jul 26, 2023

Fix compatibility with Emscripten >= 3.1.44 #3308

Merged

LinuxUserGD mentioned this pull request Sep 18, 2023

Use non-lfs64 api for getrlimit()/setrlimit() wrapper functions mozilla/audio_thread_priority#18

Closed

alyssais mentioned this pull request Oct 9, 2023

musl: 1.2.3 -> 1.2.5 NixOS/nixpkgs#229439

Merged

47 tasks

plarsson mentioned this pull request Jan 16, 2024

linker error when building on alpine linux 3.19 osohq/oso#1735

Open

170210 mentioned this pull request Jan 25, 2024

fix: change docker image version to fix build error Finschia/finschia#316

Merged

4 tasks

wesleywiser mentioned this pull request Jul 12, 2024

Update bundled musl to 1.2.5 rust-lang/rust#125692

Closed

mattip mentioned this pull request Oct 9, 2024

Don't use *64 interfaces on aplin/musl linux (like posix_fallocate64) pypy/pypy#5071

Closed

Gankra mentioned this pull request Apr 11, 2025

uv pip install private pypi repo doesn't work when used in docker by default astral-sh/uv#8450

Closed

Gankra mentioned this pull request Jun 18, 2025

Upgrade *-linux-musl targets to musl 1.2.5 rust-lang/compiler-team#887

Closed

3 tasks

	// FIXME: musl added paddings and adjusted
	// layout in 1.2.0 but our CI is still 1.1.24.
	// So, I'm leaving some fields as comments for now.
	// ref. https://github.com/bminor/musl/commit/
	// 1e7f0fcd7ff2096904fd93a2ee6d12a2392be392

Uh oh!

Conversation

bossmc commented Sep 30, 2022

Uh oh!

rust-highfive commented Sep 30, 2022

Uh oh!

JohnTitor commented Oct 2, 2022

Uh oh!

bossmc commented Oct 2, 2022

Uh oh!

JohnTitor Oct 3, 2022

Choose a reason for hiding this comment

Uh oh!

bossmc Oct 3, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JohnTitor commented Oct 3, 2022

Uh oh!

bossmc commented Oct 3, 2022

Uh oh!

bors commented Feb 12, 2023

Uh oh!

colincross commented Mar 30, 2023

Uh oh!

bossmc commented Apr 13, 2023

Uh oh!

Amanieu commented May 20, 2023

Uh oh!

bossmc commented May 22, 2023

Uh oh!

Amanieu commented May 23, 2023

Uh oh!

bossmc commented May 26, 2023

Uh oh!

bors commented May 31, 2023

Uh oh!

bors commented May 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants