Support for LLVM 18 fuzzing

[Darksonn]

The Tokio build on oss-fuzz has started failing with the following output:

Running fuzz_linked_list
Running fuzz_stream_map
Traceback (most recent call last):
  File "/usr/local/bin/profraw_update.py", line 129, in <module>
    sys.exit(main())
  File "/usr/local/bin/profraw_update.py", line 120, in main
    profraw_latest = upgrade(profraw_base, sect_prf_cnts, sect_prf_data)
  File "/usr/local/bin/profraw_update.py", line 87, in upgrade
    relativize_address(data, offset + 16, dataref, sect_prf_cnts, sect_prf_data)
  File "/usr/local/bin/profraw_update.py", line 35, in relativize_address
    value = struct.unpack('Q', data[offset:offset + 8])[0]
struct.error: unpack requires a buffer of 8 bytes
warning: /workspace/out/libfuzzer-coverage-x86_64/dumps/fuzz_stream_map.16753151517152022731_0.profraw: unsupported instrumentation profile format version
error: no profile can be merged
[2024-02-21 06:24:21,463 INFO] Finding shared libraries for targets (if any).
[2024-02-21 06:24:21,473 INFO] Finished finding shared libraries for targets.
error: fuzz_stream_map: Failed to load coverage: No such file or directory
error: Could not load coverage information
error: No such file or directory: Could not read profile data!
Traceback (most recent call last):
  File "/usr/local/bin/profraw_update.py", line 129, in <module>
    sys.exit(main())
  File "/usr/local/bin/profraw_update.py", line 120, in main
    profraw_latest = upgrade(profraw_base, sect_prf_cnts, sect_prf_data)
  File "/usr/local/bin/profraw_update.py", line 87, in upgrade
    relativize_address(data, offset + 16, dataref, sect_prf_cnts, sect_prf_data)
  File "/usr/local/bin/profraw_update.py", line 35, in relativize_address
    value = struct.unpack('Q', data[offset:offset + 8])[0]
struct.error: unpack requires a buffer of 8 bytes
warning: /workspace/out/libfuzzer-coverage-x86_64/dumps/fuzz_linked_list.14910788477734546202_0.profraw: unsupported instrumentation profile format version
error: no profile can be merged

This error seems very similar to #6268, and the latest Rust release has the following in its changelog:

Update the minimum external LLVM to 16.

Let me know if the problem is on our end, or if there's anything I can do to help.

For more information, see the chromium bug for the build failure.

[djc]

Seeing this across all of the Rust projects I'm involved with:

• askama
• redis, cc @jaymell
• rustls, cc @ctz, @cpu
• trust-dns (should rename this to hickory-dns), cc @bluejekyll

[jrmuizel]

Also happening with qcms.

[seanmonstar]

Also with httparse.

[maflcko]

I presume one way to fix this would be with #8108 in the infra

[manunio]

Coverage builds are failing for the following Rust projects as well:

• toml_edit
• naga
• miniz_oxide
• ron
• bson-rust
• gimli

cc @oliverchang @jonathanmetzman @DonggeLiu

[jonathanmetzman]

OK, I will do a roll this week or next depending on when my latest clusterfuzz work ends

[Xuanwo]

There is another failure from apache/opendal#4242

[jonathanmetzman]

Sorry, I'm putting out some fires and will need to get back to this, later this week.

[oliverchang]

ryu also seems to be failing for a similar reason: https://oss-fuzz-build-logs.storage.googleapis.com/log-acc3b39e-2aa0-4769-a499-4c224201e35e.txt

[oliverchang]

@DavidKorczynski @AdamKorcz do you think this is something you'd be able to help with?

I believe @jonathanmetzman made a start on this, but was blocked on various projects being broken with the upgrade.

Alternatively, could we also pin Rust to an older version for now (https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/install_rust.sh)?

[maflcko]

I believe @jonathanmetzman made a start on this, but was blocked on various projects being broken with the upgrade.

If there are not too many projects affected, an alternative could be to temporarily pin them to the old docker image by hash.

[DavidKorczynski]

Proposed a fix here #11681 downgrading Rust which can be used until LLVM 18 upgrade lands.

[jonathanmetzman]

We've upgraded to a recent clang. @DavidKorczynski Should we undo your change.

[DavidKorczynski]

I'm not sure, from the perspective one of the things I did was pin Rust to a specific version:

oss-fuzz/infra/base-images/base-builder/install_rust.sh

Line 18 in 6b05655

curl https://sh.rustup.rs | sh -s -- -y --default-toolchain=nightly-2023-12-28 --profile=minimal

Rust was the only language that wasn't pinned to a specific build, so I thought maybe this is something we'd like to keep moving forward?

[maflcko]

There is still the issue that a raw coverage profile generated from a current rust nightly is incompatible with the llvm version inside the oss-fuzz coverage image. Currently the rust pin is still needed for this reason. However, the pin can be moved, or re-moved, the next time llvm will be bumped again. However, that also requires un-pinning all projects, first. I can look into this, later this year.

edit: Possibly related issue #11938 (I'll look into this as well at some point)

[kaczmarczyck]

OpenSK has the same problem, but recently, also our fuzzing workflow broke.
I tried different Rust compiler versions between November 2023 and today, and all fail the same way.
Is that related to this issue, or should I open a new one?

Example failing run:
https://github.com/google/OpenSK/actions/runs/9254285343/job/25455744646

[maflcko]

According to https://oss-fuzz-build-logs.storage.googleapis.com/index.html#opensk the opensk build passed on May 17th. I don't see any changes on May 17th or 18th, that would cause the failure to happen, so this seems odd.

[kaczmarczyck]

OpenSK has an open coverage failure bug report since Feb 21st:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=66886

I saw this issue and figured that the problem is likely central, and not related to OpenSK.

Then on May 17th, the build failures started. There was no pull request on OpenSK around that time, so I assumed that the error might also be more general and not specific to OpenSK.
And I pinged this issue to see if people here are aware, or if they recognize my problem.

[kaczmarczyck]

Confirming this fixed the build problem:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=69138#c4

[alexcrichton]

I've attempted to review the current status of this and other various related updates and I've submitted #12075 to update the pinned Rust nightly to today's nightly. That being said if there are still remaining blockers I'm not aware of I can try to help hunt those down or otherwise fix our build issues via other means.

[maflcko]

Did you see #11626 (comment)?

I'd be surprised if the raw coverage profile version fixed itself in the meantime.

I think the required steps are:

• Unpin all projects to use the latest base-runner(s). See git grep '@sha256:' ./projects/ output and example commit d05f35b
• Bump clang/llvm. Similar to a2c60af
• Bump rust. Similar to 54cf7a9

If your project requires current rust-nightly, I think the only workaround for now is to disable/break the coverage build for this project.

[alexcrichton]

I saw yeah but I haven't tried to reproduce and I wasn't sure how applicable that was. Given that both seem to be using LLVM 18 I was assuming it might be stale by this point. Is there a way to see the failure that happens locally?

As for updating projects. I can grep for "2023-12-28" which has a number of hits and I can try to update those projects, but when you mentionto bump clang/llvm where is that? I thought it was

oss-fuzz/infra/base-images/base-clang/checkout_build_install_llvm.sh

Lines 52 to 53 in 7f91500

	# For manual bumping.
	OUR_LLVM_REVISION=llvmorg-18-init-4631-gd50b56d1

which already looked like llvm 18

[maflcko]

Is there a way to see the failure that happens locally?

Should be possible to reproduce in any rust project by bumping the nightly compiler for it and running the coverage build. For example:

git show 2149bb8eb67f7824414f67998ab286df083595fb | git apply --reverse
python infra/helper.py build_fuzzers --sanitizer coverage opensk
python infra/helper.py coverage opensk  # Maybe requires --no-corpus-download or --public

As for the other questions, I've updated my previous comment.

[alexcrichton]

Running that coverage command I'm getting warnings that look like:

warning: /out/...: raw profile version mismatch: Profile uses raw profile format version = 9; expected version = 8

In the image it looks like llvm-cov is 18.0.0 and digging around it looks like the profile format was bumped to 9 in LLVM 18.1.0. Rust is using 18.1.7 so if the base builder is using 18.0.0 that explains the mismatch.

I updated #12075 to remove all references to 2023-12-28 and have double-checked that all affected projects build with the new nightly version. There's only one project which uses base-builder-rust at a pinned sha256 which is cryptofuzz and I wasn't able to update that. The error didn't look related to Rust stuff, though, but I could very well be wrong.

I'm currently testing out updating to LLVM 18.1.7 and ensuring the coverage bits still work

[maflcko]

I'm currently testing out updating to LLVM 18.1.7 and ensuring the coverage bits still work

Coverage will likely break for the pinned projects (git grep '@sha256:' ./projects/), because they still use clang-15, last time I checked.

[catenacyber]

Late to the party, but what is the status here ? cc @maflcko cf #12077 (comment)
(coming from https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=70505 )

In case it helps : I took a look into https://reviews.llvm.org/D138846 and made https://github.com/catenacyber/oss-fuzz/tree/profraw9 :
It translates/downgrades profraw version 9 to version 8 so that current clang llvm-profdata merge can read nightly-rust profraw files.
But then, llvm-cov fails because there was also a bump for the elf section with profile stuff (INSTR_PROF_COVMAP_VERSION).
I can do it the other way (and update profraw8 to 9)

[catenacyber]

So, the alternative to llvm update to match rust, would be to patch profraw_update.py but also create a new program that updates the compiled fuzz targets covmap section to remove this new bitmap information...

TL;DR Updating llvm seems the right way to go

By the way, I am not sure profraw_update.py is needed anymore for profraw version 5 (swift ? )

[maflcko]

cc @maflcko cf #12077 (comment)

I haven't looked into profraw_update.py (or downgrade) at all.

TL;DR Updating llvm seems the right way to go

Agree. However, some legacy projects will have their coverage build broken by that. I am trying to fix them to work with llvm 18, but for some it is quite an effort, so help is appreciated. I fear for some projects it will be too hard and they will remain broken, but I am not sure what number of projects with broken coverage is acceptable.

[catenacyber]

Agree. However, some legacy projects will have their coverage build broken by that.

So updating profraw_update.py may help there

[catenacyber]

See #12365

[maflcko]

I think this is fixed now and can be closed. At least it seems to be working for #12556