Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update bundled musl to 1.2.5 #125692

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Conversation

est31
Copy link
Member

@est31 est31 commented May 29, 2024

Update the bundled musl library from 1.2.3 to the 1.2.5 release from February 29, 2024.

Releases page

Prior update: #107129

r? wesleywiser

try-job: dist-x86_64-musl
try-job: dist-i586-gnu-i586-i686-musl
try-job: dist-x86_64-linux

@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels May 29, 2024
@est31 est31 force-pushed the update_musl_1.2.5 branch from c4a778e to 2173f48 Compare May 29, 2024 02:54
@est31 est31 added relnotes Marks issues that should be documented in the release notes of the next release. O-musl Target: The musl libc labels May 29, 2024
@Kobzol
Copy link
Contributor

Kobzol commented May 29, 2024

Let's try on CI.

@bors try

@bors
Copy link
Contributor

bors commented May 29, 2024

⌛ Trying commit 2173f48 with merge 14bf8cb...

bors added a commit to rust-lang-ci/rust that referenced this pull request May 29, 2024
Update bundled musl to 1.2.5

[Releases page](https://musl.libc.org/releases.html)

Prior update: rust-lang#107129

r? `@wesleywiser`

try-job: dist-x86_64-musl
try-job: dist-i586-gnu-i586-i686-musl
@bors
Copy link
Contributor

bors commented May 29, 2024

☀️ Try build successful - checks-actions
Build commit: 14bf8cb (14bf8cb2ba148468f901fb4ff5126efd3bec469c)

@est31 est31 marked this pull request as ready for review May 29, 2024 20:48
kaffarell added a commit to kaffarell/tracing that referenced this pull request Jun 7, 2024
When using the linux-musl target for rust, the file creation time
cannot be retrieved, as the current version does not support it yet (
This will be fixed with [0]). In the meantime, we parse the datetime
from the filename and use that as a fallback.

Fixes: tokio-rs#2999

[0]: rust-lang/rust#125692
kaffarell added a commit to kaffarell/tracing that referenced this pull request Jun 11, 2024
When using the linux-musl target for rust, the file creation time
cannot be retrieved, as the current version does not support it yet (
This will be fixed with [0]). In the meantime, we parse the datetime
from the filename and use that as a fallback.

Fixes: tokio-rs#2999

[0]: rust-lang/rust#125692
kaffarell added a commit to kaffarell/tracing that referenced this pull request Jun 12, 2024
When using the linux-musl target for rust, the file creation time
cannot be retrieved, as the current version does not support it yet (
This will be fixed with [0]). In the meantime, we parse the datetime
from the filename and use that as a fallback.

Fixes: tokio-rs#2999

[0]: rust-lang/rust#125692
kaffarell added a commit to kaffarell/tracing that referenced this pull request Jun 13, 2024
When using the linux-musl target for rust, the file creation time
cannot be retrieved, as the current version does not support it yet (
This will be fixed with [0]). In the meantime, we parse the datetime
from the filename and use that as a fallback.

Fixes: tokio-rs#2999

[0]: rust-lang/rust#125692
kaffarell added a commit to kaffarell/tracing that referenced this pull request Jun 17, 2024
When using the linux-musl target for rust, the file creation time
cannot be retrieved, as the current version does not support it yet (
This will be fixed with [0]). In the meantime, we parse the datetime
from the filename and use that as a fallback.

Fixes: tokio-rs#2999

[0]: rust-lang/rust#125692
@szokeasaurusrex
Copy link

Hello, are there any plans to merge this PR anytime soon? Doing so might unblock rust-cross/rust-musl-cross#152, which could resolve getsentry/sentry-cli#1929.

@Kobzol
Copy link
Contributor

Kobzol commented Jul 3, 2024

The release notes look relatively harmless. I'll nominate this for t-compiler discussion.

@Kobzol
Copy link
Contributor

Kobzol commented Jul 3, 2024

This looks like a rather small update, that could nevertheless unblock some projects.

@rustbot label +I-compiler-nominated

@rustbot rustbot added the I-compiler-nominated Nominated for discussion during a compiler team meeting. label Jul 3, 2024
@apiraino
Copy link
Contributor

Discussed in t-compiler triage meeting on Zulip.

cc @wesleywiser

@rustbot label -I-compiler-nominated

@rustbot rustbot removed the I-compiler-nominated Nominated for discussion during a compiler team meeting. label Jul 11, 2024
@wesleywiser
Copy link
Member

wesleywiser commented Jul 12, 2024

Looking at the symbol tables for libc.a in musl 1.2.3 vs 1.2.5, I see the following missing symbols:

Symbol Provided in libc crate Notes
aio_cancel64 Not widely used
aio_error64 Not widely used
aio_fsync64 Not widely used
aio_read64 Not widely used
aio_return64 Not widely used
aio_write64 Not widely used
aio_suspend64 Not widely used
lio_listio64 Not widely used
alphasort64 Not widely used
readdir64
readdir64_r
scandir64 Not widely used
versionsort64 Not widely used
creat64
open64
openat64
posix_fadvise64
posix_fallocate64
ftw64 Not widely used
fallocate64
getdents64 ⚠️ Some use in ecosystem
prlimit64
sendfile64
getrlimit64
lockf64 Not widely used
ntw64 Not widely used
setrlimit64
mmap64
glob64 ⚠️ Some use in ecosystem
globfree64 ⚠️ Some use in ecosystem
__fxstat64 Not widely used
__fxstatat64 Not widely used
__lxstat64 Not widely used
__xstat64 Not widely used
fstat64
fstatat64
lstat64
stat64
fstatfs64
fstatvfs64
statfs64
statvfs64
fgetpos64
fopen64
freopen64
fseeko64
fsetpos64
ftello64
tmpfile64
mkostemp64 Not widely used
mkostemps64 Not widely used
mkstemp64 Not widely used
mkstemps64 Not widely used
ftruncate64
lseek64
pread64
preadv64
pwrite64
pwritev64
truncate64

libc began providing these symbols in 0.2.145 (rust-lang/libc#2935).

I think we should definitely do a crater run to help gauge how much impact losing the LFS64 symbols will be.

@wesleywiser
Copy link
Member

The try build artifacts seem to have timed out. Let's go again:

@bors try

@bors
Copy link
Contributor

bors commented Jul 12, 2024

🔒 Merge conflict

This pull request and the master branch diverged in a way that cannot be automatically merged. Please rebase on top of the latest master branch, and let the reviewer approve again.

How do I rebase?

Assuming self is your fork and upstream is this repository, you can resolve the conflict following these steps:

  1. git checkout update_musl_1.2.5 (switch to your branch)
  2. git fetch upstream master (retrieve the latest master)
  3. git rebase upstream/master -p (rebase on top of it)
  4. Follow the on-screen instruction to resolve conflicts (check git status if you got lost).
  5. git push self update_musl_1.2.5 --force-with-lease (update this PR)

You may also read Git Rebasing to Resolve Conflicts by Drew Blessing for a short tutorial.

Please avoid the "Resolve conflicts" button on GitHub. It uses git merge instead of git rebase which makes the PR commit history more difficult to read.

Sometimes step 4 will complete without asking for resolution. This is usually due to difference between how Cargo.lock conflict is handled during merge and rebase. This is normal, and you should still perform step 5 to update this PR.

Error message
Auto-merging src/doc/rustc/src/platform-support.md
CONFLICT (content): Merge conflict in src/doc/rustc/src/platform-support.md
Automatic merge failed; fix conflicts and then commit the result.

@wesleywiser wesleywiser removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 12, 2024
@bors
Copy link
Contributor

bors commented Jul 15, 2024

⌛ Trying commit 1b6a1f8 with merge 115cea8...

bors added a commit to rust-lang-ci/rust that referenced this pull request Jul 15, 2024
Update bundled musl to 1.2.5

Update the bundled musl library from 1.2.3 to the 1.2.5 release from February 29, 2024.

[Releases page](https://musl.libc.org/releases.html)

Prior update: rust-lang#107129

`r? wesleywiser`

try-job: dist-x86_64-musl
try-job: dist-i586-gnu-i586-i686-musl
try-job: dist-x86_64-linux
bors added a commit to rust-lang-ci/rust that referenced this pull request Jul 15, 2024
[DO NOT MERGE] Build musl targets for crater run

Using to test rust-lang#125692

try-job: dist-x86_64-musl
try-job: dist-i586-gnu-i586-i686-musl
try-job: dist-x86_64-linux

r? `@ghost`
@bors
Copy link
Contributor

bors commented Jul 15, 2024

☀️ Try build successful - checks-actions
Build commit: 115cea8 (115cea8e821b9c40aef33b4644037c0963561120)

@wesleywiser
Copy link
Member

@craterbot run mode=build-and-test name=musl_upgrade_1_2_5 start=try#52500f34dbfa5ef09f94c5cb770491fbd834bcb6+target=x86_64-unknown-linux-musl end=try#115cea8e821b9c40aef33b4644037c0963561120+target=x86_64-unknown-linux-musl

@craterbot
Copy link
Collaborator

👌 Experiment musl_upgrade_1_2_5 created and queued.
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

@craterbot craterbot added S-waiting-on-crater Status: Waiting on a crater run to be completed. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 15, 2024
@craterbot
Copy link
Collaborator

🚧 Experiment musl_upgrade_1_2_5 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

@craterbot
Copy link
Collaborator

🎉 Experiment musl_upgrade_1_2_5 is completed!
📊 11788 regressed and 91 fixed (482394 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the blacklist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

@craterbot craterbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-crater Status: Waiting on a crater run to be completed. labels Jul 23, 2024
@wesleywiser
Copy link
Member

Looked at ~20 random regressions and they were all due to getrandom using an older version of libc which calls open64.

I think we need to consider building musl with -D_LARGEFILE64_SOURCE to get the LFS64 symbols back. That will give time for newer versions of the libc crate to propagate through the ecosystem.

@Skgland
Copy link
Contributor

Skgland commented Jul 23, 2024

Is this already fixed in a released libc version? I noticed one of my repos with getrandom v0.2.6 and libc v0.2.126 in there.
Those are only some patch versions behind latest getrandom v0.2.15 and libc v0.2.155, so I would expect running cargo update to be able to get the lockfile to latest. Is there someway I could test whether updating the lock file would resolve the problem?

@est31
Copy link
Member Author

est31 commented Jul 24, 2024

Is this already fixed in a released libc version?

Yes, it's fixed in 0.2.145, also mentioned in the post above. 0.2.145 was released on June 4, 2023, so more than one year ago, but I suppose not everyone runs cargo update regularly.

I think we need to consider building musl with -D_LARGEFILE64_SOURCE to get the LFS64 symbols back.

Yeah, agreed. To verify beyond your sample of 20 crates, I downloaded the regressed crates logs from the crater report and checked the regressions. There is only 146 regressions which do not contain "undefined reference". I checked 3-4 of them and they seemed all noise like out of space errors.

$ rg -g "*115cea8e821b9c40aef33b4644037c0963561120*" -l 'undefined reference' | wc -l
11642
$ rg -g "*115cea8e821b9c40aef33b4644037c0963561120*" --files-without-match 'undefined reference' | wc -l
146

Statistics of which functions were noted as missing:

$ rg -g "*115cea8e821b9c40aef33b4644037c0963561120*" 'undefined reference to' | sed 's/^.* undefined reference to//;s/ symbol //' | sort | uniq -c | sort -rn
  13041  `open64'
    176 'open64@@GLIBC_2.2.5'
     57  `fopen64'
     41  `mmap64'
     38  `fstat64'
     22  `fseeko64'
     15 'readdir64_r@@GLIBC_2.2.5'
     11  `tmpfile64'
     11  `posix_fadvise64'
     11  `mkstemp64'
     11  `ftello64'
      5  `lseek64'
      3  `setrlimit64'
      3  `readdir64_r'
      3  `getrlimit64'
      3  `fallocate64'
      2  `statvfs64'
      2  `readdir64'
      2  `openat64'
      1  `statfs64'
      1  `sendfile64'
      1  `pread64'
      1 'mkstemp64@@GLIBC_2.2.5'
      1  `lstat64'
      1 'lseek64@@GLIBC_2.2.5'
      1  `__fxstat64'
      1  `ftruncate64'
      1 'fstat64@@GLIBC_2.33'

not an expert but given they all contain 64 in their names, they seem to be from LFS64.

So next step is probably me figuring out how to pass -D_LARGEFILE64_SOURCE to the musl build...

@est31 est31 force-pushed the update_musl_1.2.5 branch from 1b6a1f8 to 974ed62 Compare July 24, 2024 00:51
@est31
Copy link
Member Author

est31 commented Jul 24, 2024

@bors try

bors added a commit to rust-lang-ci/rust that referenced this pull request Jul 24, 2024
Update bundled musl to 1.2.5

Update the bundled musl library from 1.2.3 to the 1.2.5 release from February 29, 2024.

[Releases page](https://musl.libc.org/releases.html)

Prior update: rust-lang#107129

`r? wesleywiser`

try-job: dist-x86_64-musl
try-job: dist-i586-gnu-i586-i686-musl
try-job: dist-x86_64-linux
@bors
Copy link
Contributor

bors commented Jul 24, 2024

⌛ Trying commit 974ed62 with merge 497c108...

@bors
Copy link
Contributor

bors commented Jul 24, 2024

☀️ Try build successful - checks-actions
Build commit: 497c108 (497c108e484bd2f2c090cc4ab84ab4c59703cecd)

@est31
Copy link
Member Author

est31 commented Jul 24, 2024

For trying out the try build, I'm using https://github.com/kennytm/rustup-toolchain-install-master . command:

rustup-toolchain-install-master -t x86_64-unknown-linux-musl -- 115cea8e821b9c40aef33b4644037c0963561120
rustup-toolchain-install-master -t x86_64-unknown-linux-musl -- 497c108e484bd2f2c090cc4ab84ab4c59703cecd

with that, one can check the custom toolchains via:

cargo +115cea8e821b9c40aef33b4644037c0963561120 build --target x86_64-unknown-linux-musl
cargo +497c108e484bd2f2c090cc4ab84ab4c59703cecd build --target x86_64-unknown-linux-musl

apparently though, neither works :(.

I see the flag added to CFLAGS though here...

@est31
Copy link
Member Author

est31 commented Jul 24, 2024

Ahhh apparently the -D_LARGEFILE64_SOURCE flag needs to be set by the application that wants to use these functions, not when compiling musl itself, see link here.

The actual linkable symbols were removed by this commit, first included in 1.2.4. The commit also adjusted the dynamic linker to account for that removal, but as we don't use it in this instance, that adjustment can't be leveraged.

I wonder if we should patch musl to make the symbols linkable again?

@JustSoup312
Copy link

I would love to see this get merged since it is a blocker for various downstream applications. Can I help in any way?

JustSoup312 added a commit to JustSoup312/bootupd that referenced this pull request Oct 29, 2024
Creation time will only be supported on the musl target once rust-lang/rust#125692 lands.
[`powerpc64-unknown-openbsd`](platform-support/openbsd.md) | ✓ | ✓ | OpenBSD/powerpc64
[`powerpc64-ibm-aix`](platform-support/aix.md) | ? | | 64-bit AIX (7.2 and newer)
`riscv32gc-unknown-linux-gnu` | | | RISC-V Linux (kernel 5.4, glibc 2.33)
`riscv32gc-unknown-linux-musl` | | | RISC-V Linux (kernel 5.4, musl 1.2.3 + RISCV32 support patches)
`riscv32gc-unknown-linux-musl` | | | RISC-V Linux (kernel 5.4, musl 1.2.5 + RISCV32 support patches)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`riscv32gc-unknown-linux-musl` | | | RISC-V Linux (kernel 5.4, musl 1.2.5 + RISCV32 support patches)
`riscv32gc-unknown-linux-musl` | | | RISC-V Linux (kernel 5.4, musl 1.2.5)

musl 1.2.5 officially supports riscv32. https://github.com/bminor/musl/blob/v1.2.5/WHATSNEW#L2413

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc O-musl Target: The musl libc relnotes Marks issues that should be documented in the release notes of the next release. S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.