[Minify] Initial integration #3848
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hey @AdamKorcz, TravisBuddy Request Identifier: 2745e630-99c9-11ea-8b80-f318e961985e |
|
Thank you @AdamKorcz for the effort! This will be of great help for the library as I've used fuzzing in the past succesfully to fix numerous bugs. I assume that OSS-Fuzz pulls in changes (from the library and the fuzz repository) automatically every now and then? Does it help to move the fuzz repository inside |
|
@tdewolff Moving the fuzzing repository into the main one won't make a difference. We can consider it in case any problems arise, but the fuzzers all run fine as of now. And yes, updates to the branches in the build files should be considered in the fuzz-jobs. Let's see if your good library is accepted, and we can then definitely consider fuzzing the underlying |
|
I've added the fuzz tests to
Agreed! |
We accept the project, @AdamKorcz , can you please update build.sh to use the single repo and then we can merge. |
|
Thanks for the work @AdamKorcz Should we incorporate https://github.com/tdewolff/parse in this PR, as it is quite tightly coupled with |
Can just do this in another PR. Committing this for now. |
Travis tests have failedHey @AdamKorcz, 1st Build./infra/travis/travis_build.pyTravisBuddy Request Identifier: 18e1a5d0-9b58-11ea-acf2-2540bfc6cd37 |
|
Hey @AdamKorcz, TravisBuddy Request Identifier: 779c7650-9b6b-11ea-acf2-2540bfc6cd37 |
Travis tests have failedHey @AdamKorcz, 1st Build./infra/travis/travis_build.pyTravisBuddy Request Identifier: 7795bf90-9b6b-11ea-acf2-2540bfc6cd37 |
tdewolff
reviewed
May 21, 2020
| $CXX $CXXFLAGS $LIB_FUZZING_ENGINE $fuzzer.a -o $OUT/$fuzzer | ||
| } | ||
|
|
||
| find $GOPATH/src/github.com/tdewolff/minify/tests/*/ -maxdepth 1 -type d | while read target |
There was a problem hiding this comment.
Shouldn't this be -maxdepth 0? It's showing me all subdirectories of each test case (i.e. suppressions, corpus, crashers, ...).
|
|
||
| find $GOPATH/src/github.com/tdewolff/minify/tests/*/ -maxdepth 1 -type d | while read target | ||
| do | ||
| fuzz_target=`echo $target | cut -d'/' -f 9` |
There was a problem hiding this comment.
Does the -f 9 always work? This depends on the length of $GOPATH, right? We could rewrite this as:
find $GOPATH/src/github.com/tdewolff/minify/tests/* -maxdepth 0 -type d | rev | cut -d'/' -f 1 | revMind the lack of a / at the end of the find path. The rev allows to select the last item.
Sorry @tdewolff , should have waited on your review, can you please submit a PR for these fixes + those other 5 fuzzers enabling. |
|
May I suggest moving the build script upstream? That way it will be easier for @tdewolff to work on updates? |
Yes totally, go ahead, we actually recommend that in ideal integration - https://google.github.io/oss-fuzz/advanced-topics/ideal-integration/ |
|
Cool. @tdewolf FYI: The plan is to add the build script to the upstream repository and run that one when fuzzing through oss-fuzz. I will get the build script added asap. |
zchcai
added a commit
to zchcai/oss-fuzz
that referenced
this pull request
Jun 3, 2020
* [cryptofuzz] Add Microsoft SymCrypt (google#3826) * [gRPC-gateway] Initial integration (google#3807) * [doc] Update new_project_guide.md (google#3828) * [knot-dns] link against liblmdb statically (google#3797) Relates to https://gitlab.labs.nic.cz/knot/knot-dns/-/commit/d4ec3a3aa8b7e2ba4d196e2f7984173069e3d91b Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22191 * Simplify rust project setup. (google#3830) * Simplify rust project setup. - Add rust and cargo-fuzz in base builder. - Set RUSTC_BOOSTRAP to make ASan available. - Set RUSTFLAGS and C,CXXFLAGS properly. * Quick fix for helper.py build_fuzzers. (google#3832) * Parse project language and use in infra/helper.py (google#3834) * Parse project language and use in infra/helper.py * Fix exception message. * Add FUZZING_LANGUAGE to build step. (google#3835) * Add fuzzing language correctly during compile step. (google#3837) * Remove unneeded rust hacks after google#3830. (google#3840) * Fix missing FUZZING_LANGUAGE in coverage build. (google#3843) * Use hardcoded FUZZING_LANGUAGE to unbreak CIFuzz. (google#3844) * solidity: Disable Z3 build in dockerfile (google#3831) * cifuzz: use pull_request.number instead of GITHUB_REF (google#3845) Closes google#3732 * wasmtime: build fuzz targets with --all-features (google#3850) This enables not only the binaryen-using fuzz targets, but also the peepmatic fuzz targets (which is necessary after bytecodealliance/wasmtime#1727). * Relax project language requirement. (google#3846) This is necessary for the bisector to be able to build older revisions. Print a warning instead. * libzmq: add maintainer, add UBSAN (google#3829) * Add another maintainer to libzmq's CC list * Enable UBSAN for libzmq * libzmq: disable afl fuzzer, CI fails * libsodium: do not let libsodium's autogen.sh download files from gnu.org There is no sanity check and if the download fails because gnu.org is down the build fails with unhelpful errors * Fixed changes added in clang-11.0. (google#3852) * Bump httplib2 from 0.11.3 to 0.18.0 in /infra/gcb (google#3853) Bumps [httplib2](https://github.com/httplib2/httplib2) from 0.11.3 to 0.18.0. - [Release notes](https://github.com/httplib2/httplib2/releases) - [Changelog](https://github.com/httplib2/httplib2/blob/master/CHANGELOG) - [Commits](httplib2/httplib2@v0.11.3...v0.18.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [TiDB] initial integration (google#3849) * [Minify] Initial integration (google#3848) * Set rust flags always in compile step. (google#3859) This is needed for projects that mix both rust and c/c++. * Remove unneeded RUSTC_BOOTSTRAP since nightly is used * [fasthttp] Initial integration (google#3858) * [libfido2] bump libcbor version from 0.5.0 to 0.7.0 (google#3861) * Fix cras build failure, use default rust nightly. * Don't add Rust sanitizer flags when using ubsan or i386. (google#3862) Rust sanitizer flags are not supported for these configs. Fixes Cras and ecc-diff-fuzzer builds. * [hermes] Initial setup for Hermes (google#3698) * [poppler] Exercise a few more methods (google#3857) * [minify] Moved build script upstream (google#3867) * Use available nightly rust for spidermonkey build * Use available rust nightly in spidermonkey-ufi * Use available rust nightly in firefox * [python3-libraries] Fix build (google#3869) * Use latest git version. (google#3870) Adds features needed by firefox and spidermonkey-ufi builds. * libzmq: fix MSAN (google#3868) * libzmq: build libsodium with --disable-asm to avoid false positives in MSAN checks * libzmq: set autoconf install prefix to avoid /usr/local being added to the linker library paths A non-working version of the standard library is installed in oss-fuzz's images under /usr/local/lib. Using it breaks MSAN's tests with false positives. Set the prefix in autoconf to something different from the default /usr/local, as /lib is added by autoconf automatically to the linker flags, which means this broken standard library is used instead of the instrumented one in /usr/lib. * build_specified_commit: revert to using cp. (google#3871) rsync seems to have disappeared in the latest images somehow, and we can't rely on it always existing. * Don't add sanitizer=coverage in RUSTFLAGS * Remove libpcap patch merged upstream (google#3865) * Install Python 3.8.3 on builder image. (google#3874) Part of fixing google#3756. * [qt] Get a shallow clone of qtbase (google#3872) Reduces the size by nearly 200 MB * [qt] Add fuzzer for QImage::loadFromData() (google#3873) * Fix jsoncpp build * build_specified_commit: Don't replace gitdir if already relative. (google#3875) Upgraded git broke this. * [skia] Speculative build fix (google#3847) * [stb] extend code coverage for stb_image (google#3784) * [stb] extend code coverage for stb_image * remove wget's from build.sh * add wget's to Dockerfile * fix Dockerfile * [hermes] Fix coverage build (google#3877) * Update email address to non-alias for clusterfuzz login (google#3878) * add hosts and interns (google#3879) Signed-off-by: Asra Ali <[email protected]> * [qt] Don't install build-essential (google#3880) It's in base-builder now. * [hermes] Add CC for issues (google#3882) * [systemd] turn off hongfuzz (google#3889) See google#3887 * Remove libbz2-dev dependency installed in base-builder for python3 (google#3890) * Remove libbz2-dev dependency installed in base-builder for python3 Fixes google#3888 * Revert "[systemd] turn off hongfuzz (google#3889)" (google#3891) This reverts commit d638fac. * Set up python in one docker layer, remove python deps. (google#3893) * Set up python in one docker layer. Address review comment in google#3890 (review) * Remove all python dependencies, keep build-essential. * Makes suricata work with rust sanitizers (google#3897) * [firefox] Fix libFuzzer cloning process (google#3896) * Fix spidermonkey-ufi build. * [nanopb] Use $(which python3) instead of hardcoding /usr/bin/python3 (google#3895) After the addition of Python 3.8 in google#3874, the default python3 is now /usr/local/bin/python3. The nanopb Dockerfile hardcoded a path of /usr/bin/python3, causing again problems of Python packages being installed for different version than attempting to run with. * Fix CIFuzz issue where targets assumed in OSS-Fuzz build if exists (google#3817) Make some other changes: 1. Refactor fuzz_target.py and fuzz_target_test.py 2. Introduce pyfakefs and parameterized as dependencies and use them in tests. 3. Fix infra-tests in CI so that they use installed dependencies * Fix spidermonkey-ufi build * Extend Zeek project auto_ccs (google#3898) * [libgd] Add another fuzzer to libgd (google#3892) * [qt] Add qtdeclarative (google#3901) * [qt] Add qtdeclarative * [qt] Ignore stdout from make "The job exceeded the maximum log length, and has been terminated." * Updating jsoncpp primary contact (google#3900) * [php] Remove --enable-json (google#3899) JSON is now always compiled in, and cannot be explicitly enabled. * Don't create empty corpus archive as it wont unpack. (google#3903) * Update email addresses for Prometheus (google#3905) Signed-off-by: Julius Volz <[email protected]> * [cryptofuzz] wolfCrypt: Build with support for AES-CFB, AES-OFB (google#3904) * libzmq: enable honggfuzz (google#3909) * Update Prometheus maintainer in Dockerfile (google#3906) ...to myself (Prometheus co-creator). Signed-off-by: Julius Volz <[email protected]> * Revert "[systemd] remove my non-gmail address (google#3606)" (google#3910) This reverts commit 65dcabc. * [GDAL] Move clone of dependencies to upstream repository (google#3911) Cloning of build dependencies and patching is now done by https://github.com/OSGeo/gdal/blob/master/gdal/fuzzers/build.sh * bump libtorrent to fuzz the 2.0 release branch (google#3913) * [FFmpeg] install rsync as it is required for the seed corpus (google#3912) Fixes: Issue 22500 (build failure) * Fix builds_status cron exception. (google#3914) When build log is not found, skip it and avoid 404. Fixes exception ``` Traceback (most recent call last): File "oss-fuzz/infra/gcb/builds_status.py", line 243, in <module> main() File "oss-fuzz/infra/gcb/builds_status.py", line 228, in main status_filename='status.json') File "oss-fuzz/infra/gcb/builds_status.py", line 159, in update_build_status last_build = find_last_build(builds, project, build_tag_suffix) File "oss-fuzz/infra/gcb/builds_status.py", line 102, in find_last_build log.download_to_filename(f.name) File "/var/jenkins_home/workspace/infra/builds_status/ENV/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 565, in download_to_filename file_obj, client=client, start=start, end=end) File "/var/jenkins_home/workspace/infra/builds_status/ENV/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 537, in download_to_file _raise_from_invalid_response(exc) File "/var/jenkins_home/workspace/infra/builds_status/ENV/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 1873, in _raise_from_invalid_response raise exceptions.from_http_response(error.response) google.api_core.exceptions.NotFound: 404 GET https://www.googleapis.com/download/storage/v1/b/oss-fuzz-gcb-logs/o/log-9fca5dab-72bf-4970-9557-86b93ede51e0.txt?alt=media: No such object: oss-fuzz-gcb-logs/log-9fca5dab-72bf-4970-9557-86b93ede51e0.txt Build step 'Execute shell' marked build as failure Finished: FAILURE ``` * Retry building project images in build_specified_commit. (google#3915) To mitigate transient network issues. * Prometheus: Don't use WORKDIR $SRC (google#3908) This makes build_fuzzers work with local checkouts. Without this, it complains with 'Cannot use local checkout with "WORKDIR /src".' Signed-off-by: Julien Pivotto <[email protected]> * [systemd] add DaanDeMeyer's address (google#3916) * [muparser] initial integration (google#3814) * [haproxy] Fix build failures. (google#3884) * Updated the haproxy fuzzers to build again. * The frame decoder needs additional updates since we need to call init_h2 to initialise a memory pool. Disabling this for now as this is a larger change in the code base and will fix up in the coming week. * Fixed hpack decode. * Updated the yaml since we dont want memory sanitizer. * Update project.yaml (google#3917) * install rsync (google#3920) * [LibRaw] Initial integration (google#3918) * Init integration * Fix formatting * Add size check * Update maintainers * PR Feedback * Add newline * Disable UBSan vptr * Disable building examples * Remove disabling UBSan vptr Co-authored-by: Jamie Pinheiro <[email protected]> * ecc-diff-fuzzer buils nettle without openssl (google#3922) * Update Dockerfile (google#3923) * Added contact to zlib-ng project auto_ccs. (google#3924) * [qt] Add pdf to image corpus (google#3926) * [haproxy] Minor fix (google#3928) * Revert "[qt] Add pdf to image corpus (google#3926)" (google#3929) To handle PDFs, QImage requires qtwebengine which doesn't build statically. This reverts commit 934c770. * Fix TensorFlow build (google#3930) * Remove [email protected] as he left Google * Use python3 as python2 is deprecated * Need to manually install numpy now as it is used in the toolchain * Use bazelisk instead of the grep configure -> get Bazel version -> curl hack * Remove C++11 constraint as TF now builds and uses C++14 * Handle review * Remove source sed/replace as it is no longer needed Co-authored-by: Guido Vranken <[email protected]> Co-authored-by: AdamKorcz <[email protected]> Co-authored-by: Daniel Salzman <[email protected]> Co-authored-by: Abhishek Arya <[email protected]> Co-authored-by: Oliver Chang <[email protected]> Co-authored-by: Bhargava Shastry <[email protected]> Co-authored-by: Evgeny Vereshchagin <[email protected]> Co-authored-by: Nick Fitzgerald <[email protected]> Co-authored-by: Luca Boccassi <[email protected]> Co-authored-by: DavidKorczynski <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Gabriel Kihlman <[email protected]> Co-authored-by: neildhar <[email protected]> Co-authored-by: tsdgeos <[email protected]> Co-authored-by: Catena cyber <[email protected]> Co-authored-by: Robert Löhning <[email protected]> Co-authored-by: Kevin Lubick <[email protected]> Co-authored-by: Randy <[email protected]> Co-authored-by: Will <[email protected]> Co-authored-by: asraa <[email protected]> Co-authored-by: Christian Holler (:decoder) <[email protected]> Co-authored-by: Petteri Aimonen <[email protected]> Co-authored-by: jonathanmetzman <[email protected]> Co-authored-by: Jon Siwek <[email protected]> Co-authored-by: Google AutoFuzz Team <[email protected]> Co-authored-by: kabeer27 <[email protected]> Co-authored-by: Nikita Popov <[email protected]> Co-authored-by: Julius Volz <[email protected]> Co-authored-by: Even Rouault <[email protected]> Co-authored-by: Arvid Norberg <[email protected]> Co-authored-by: Michael Niedermayer <[email protected]> Co-authored-by: Julien Pivotto <[email protected]> Co-authored-by: Ingo Berg <[email protected]> Co-authored-by: Teju Nareddy <[email protected]> Co-authored-by: Jamie Pinheiro <[email protected]> Co-authored-by: Jamie Pinheiro <[email protected]> Co-authored-by: Nathan Moinvaziri <[email protected]> Co-authored-by: Mihai Maruseac <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds initial integration of Minify.
Minify is a Go package that implements minifiers for web formats (HTML, JS, CSS etc). It provides minifiers that are faster than existing solutions and it is widely used in the community.
According to the author, the list of users includes but is not limited to:
Tagging the author to keep him updated on the progress of the integration: @tdewolff