Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement mixed script confusable lint. #72770

Merged
merged 2 commits into from
Jun 26, 2020

Conversation

crlf0710
Copy link
Member

This implements the mixed script confusable lint defined in RFC 2457.
This is blocked on #72069 and unicode-rs/unicode-security#13, and will need a Cargo.toml version bump after those are resolved.

The lint message warning is sub-optimal for now. We'll need a mechanism to properly output AugmentScriptSet to screen, this is to be added in unicode-security crate.

r? @Manishearth

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 30, 2020
@rust-highfive
Copy link
Collaborator

The job mingw-check of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
##[section]Starting: Linux mingw-check
##[section]Starting: Initialize job
Agent name: 'Azure Pipelines 3'
Agent machine name: 'fv-az619'
Current agent version: '2.169.1'
##[group]Operating System
16.04.6
LTS
LTS
##[endgroup]
##[group]Virtual Environment
Environment: ubuntu-16.04
Version: 20200517.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu16/20200517.1/images/linux/Ubuntu1604-README.md
##[endgroup]
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Download all required tasks.
Downloading task: Bash (3.163.2)
Checking job knob settings.
   Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} 
   Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} 
Start tracking orphan processes.
##[section]Finishing: Initialize job
##[section]Starting: Configure Job Name
==============================================================================
---
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/3cba7038-2abb-4a15-8d4a-1b5f65fa3867.sh

##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/72770/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
---
##[command]git remote add origin https://github.com/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/72770/merge:refs/remotes/pull/72770/merge
---
 ---> 3adb0605cc65
Step 6/7 : ENV RUN_CHECK_WITH_PARALLEL_QUERIES 1
 ---> Using cache
 ---> 28dbc326cb7f
Step 7/7 : ENV SCRIPT python3 ../x.py test src/tools/expand-yaml-anchors &&            python3 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu &&            python3 ../x.py build --stage 0 src/tools/build-manifest &&            python3 ../x.py test --stage 0 src/tools/compiletest &&            python3 ../x.py test src/tools/tidy &&            python3 ../x.py doc --stage 0 src/libstd &&            /scripts/validate-toolstate.sh
 ---> 537a01811900
Successfully built 537a01811900
Successfully tagged rust-ci:latest
Built container sha256:537a018119009dc218456238dec90b5530050db1e2a1e166550c218003f6159d
---

###################################################                       71.4%
######################################################################## 100.0%
extracting /checkout/obj/build/cache/2020-04-22/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz
    Updating git repository `https://github.com/crlf0710/unicode-security`
---
   Compiling rustc_llvm v0.0.0 (/checkout/src/librustc_llvm)
    Checking petgraph v0.4.13
   Compiling rustc_version v0.2.3
    Checking crossbeam-queue v0.1.2
    Checking unicode-security v0.0.3 (https://github.com/crlf0710/unicode-security#075e93a8)
    Checking num_cpus v1.10.1
    Checking parking_lot_core v0.7.1
    Checking memmap v0.7.0
    Checking jobserver v0.1.21
---
    Checking rustc_feature v0.0.0 (/checkout/src/librustc_feature)
    Checking fmt_macros v0.0.0 (/checkout/src/libfmt_macros)
    Checking rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
    Checking rustc_hir v0.0.0 (/checkout/src/librustc_hir)
    Checking chalk-rust-ir v0.10.0
    Checking rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
    Checking chalk-solve v0.10.0
    Checking rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
    Checking rustc_parse v0.0.0 (/checkout/src/librustc_parse)
    Checking rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
configure: rust.channel         := nightly
configure: build.submodules     := False
configure: rust.dist-src        := False
configure: dist.missing-tools   := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: 
configure: run `python /checkout/x.py --help`
configure: 
---
Hugepagesize:       2048 kB
DirectMap4k:      112576 kB
DirectMap2M:     5130240 kB
DirectMap1G:     4194304 kB
+ python3 ../x.py test src/tools/expand-yaml-anchors
Ensuring the YAML anchors in the GitHub Actions config were expanded
Ensuring the YAML anchors in the GitHub Actions config were expanded
Building stage0 tool expand-yaml-anchors (x86_64-unknown-linux-gnu)
   Compiling unicode-xid v0.2.0
   Compiling syn v1.0.11
   Compiling linked-hash-map v0.5.2
   Compiling lazy_static v1.4.0
   Compiling lazy_static v1.4.0
   Compiling yaml-rust v0.4.3
   Compiling quote v1.0.2
   Compiling thiserror-impl v1.0.5
   Compiling thiserror v1.0.5
   Compiling yaml-merge-keys v0.4.0
   Compiling expand-yaml-anchors v0.1.0 (/checkout/src/tools/expand-yaml-anchors)
Build completed successfully in 0:00:30
+ python3 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu
    Finished dev [unoptimized] target(s) in 0.22s
Checking rustdoc artifacts (x86_64-unknown-linux-gnu -> i686-pc-windows-gnu)
---
   Compiling rustc_llvm v0.0.0 (/checkout/src/librustc_llvm)
    Checking petgraph v0.4.13
   Compiling rustc_version v0.2.3
    Checking crossbeam-queue v0.1.2
    Checking unicode-security v0.0.3 (https://github.com/crlf0710/unicode-security#075e93a8)
    Checking rustc-hash v1.0.1
    Checking num_cpus v1.10.1
   Compiling memoffset v0.5.1
   Compiling parking_lot_core v0.6.2
---
    Checking rustc_feature v0.0.0 (/checkout/src/librustc_feature)
    Checking fmt_macros v0.0.0 (/checkout/src/libfmt_macros)
    Checking rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
    Checking rustc_hir v0.0.0 (/checkout/src/librustc_hir)
    Checking chalk-rust-ir v0.10.0
    Checking rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
    Checking chalk-solve v0.10.0
    Checking rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
    Checking rustc_parse v0.0.0 (/checkout/src/librustc_parse)
    Checking rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
   Compiling cargo_metadata v0.9.1
   Compiling tidy v0.1.0 (/checkout/src/tools/tidy)
    Finished release [optimized] target(s) in 24.52s
tidy check
tidy error: /checkout/src/test/ui/lint/rfc-2457-non-ascii-idents/lint-mixed-script-confusables.rs:7: trailing whitespace
invalid source: "git+https://github.com/crlf0710/unicode-security#075e93a8b4b994ceb6fd8fa5da52d89e3677a27b"
Found 496 error codes
Found 0 error codes with no tests
Done!
some tidy checks failed
some tidy checks failed


command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo"


failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
Build completed unsuccessfully in 0:00:35
Build completed unsuccessfully in 0:00:35
== clock drift check ==
  local time: Sat May 30 05:41:37 UTC 2020
  network time: Sat, 30 May 2020 05:41:37 GMT
== end clock drift check ==

##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/72770/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/72770/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (4834) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

Copy link
Member

@Manishearth Manishearth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall design looks good

src/librustc_lint/non_ascii_idents.rs Outdated Show resolved Hide resolved
src/librustc_lint/non_ascii_idents.rs Show resolved Hide resolved
src/librustc_lint/non_ascii_idents.rs Outdated Show resolved Hide resolved

cx.struct_span_lint(MIXED_SCRIPT_CONFUSABLES, sp, |lint| {
let message = format!(
"Unicode augmented script group `{:?}` usage in this crate consists solely of mixed script confusables, including {:?} and maybe other characters.",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The usage of Script Group {:?} in this crate consists solely of mixed script confusables, including {:?}"

Copy link
Member Author

@crlf0710 crlf0710 Jun 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in principle. Didn't find out a proper way to generate a long error message including line-breaks, so i moved the second sub-sentence to a note. Please review again to see if this is ok.

src/librustc_lint/non_ascii_idents.rs Show resolved Hide resolved
@bors
Copy link
Contributor

bors commented Jun 20, 2020

☔ The latest upstream changes (presumably #73511) made this pull request unmergeable. Please resolve the merge conflicts.

@crlf0710 crlf0710 force-pushed the mixed_script_confusable branch from d9b51ed to 748fdfe Compare June 23, 2020 11:45
@rust-highfive

This comment has been minimized.

@crlf0710 crlf0710 force-pushed the mixed_script_confusable branch from 748fdfe to 4add62a Compare June 23, 2020 15:12
#![deny(mixed_script_confusables)]

struct ΑctuallyNotLatin;
//~^ ERROR The usage of Script Group `AugmentedScriptSet {Grek}` in this crate consists solely of
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug impls should not be used here, we should instead have something that will display Greek and Greek + Foo + Bar without the AugmentedScriptSet {} bit. ScriptExtension already has a Debug impl that could be used for this. I can instead give it a Display impl that's better here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@Manishearth Manishearth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would like to have more tests, but this seems okay

@crlf0710
Copy link
Member Author

Sure, i can add more tests after switching Debug to Display, and try to make it test out more logic branching. And do you have anything specially want to cover within the tests?

@Manishearth
Copy link
Member

Manishearth commented Jun 24, 2020

Mostly just make sure all cases are covered. E.g. having an ident роре (Cyrillic) in a crate with and without other Cyrillic in it

@crlf0710 crlf0710 force-pushed the mixed_script_confusable branch from 4add62a to bec58fb Compare June 24, 2020 14:11
@crlf0710 crlf0710 marked this pull request as ready for review June 25, 2020 12:11
Copy link
Member

@Manishearth Manishearth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there, two minor issues

@crlf0710 crlf0710 force-pushed the mixed_script_confusable branch from bec58fb to 25e864e Compare June 25, 2020 17:40
@Manishearth
Copy link
Member

@bors r+

@bors
Copy link
Contributor

bors commented Jun 25, 2020

📌 Commit 25e864e has been approved by Manishearth

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 25, 2020
Manishearth added a commit to Manishearth/rust that referenced this pull request Jun 26, 2020
…=Manishearth

Implement mixed script confusable lint.

This implements the mixed script confusable lint defined in RFC 2457.
This is blocked on rust-lang#72069 and unicode-rs/unicode-security#13, and will need a Cargo.toml version bump after those are resolved.

The lint message warning is sub-optimal for now. We'll need a mechanism to properly output  `AugmentScriptSet` to screen, this is to be added in `unicode-security` crate.

r? @Manishearth
Manishearth added a commit to Manishearth/rust that referenced this pull request Jun 26, 2020
…=Manishearth

Implement mixed script confusable lint.

This implements the mixed script confusable lint defined in RFC 2457.
This is blocked on rust-lang#72069 and unicode-rs/unicode-security#13, and will need a Cargo.toml version bump after those are resolved.

The lint message warning is sub-optimal for now. We'll need a mechanism to properly output  `AugmentScriptSet` to screen, this is to be added in `unicode-security` crate.

r? @Manishearth
bors added a commit to rust-lang-ci/rust that referenced this pull request Jun 26, 2020
…arth

Rollup of 14 pull requests

Successful merges:

 - rust-lang#72617 (Add a fast path for `std::thread::panicking`.)
 - rust-lang#72738 (Self contained linking option)
 - rust-lang#72770 (Implement mixed script confusable lint.)
 - rust-lang#73418 (Add unstable `core::mem::variant_count` intrinsic)
 - rust-lang#73460 (Emit line info for generator variants)
 - rust-lang#73534 (Provide suggestions for some moved value errors)
 - rust-lang#73538 (make commented examples use valid syntax, and be more consistent )
 - rust-lang#73581 (Create 0766 error code)
 - rust-lang#73619 (Document the mod keyword)
 - rust-lang#73621 (Document the mut keyword)
 - rust-lang#73648 (Document the return keyword)
 - rust-lang#73673 (Fix ptr doc warnings.)
 - rust-lang#73674 (Tweak binop errors)
 - rust-lang#73687 (Clean up E0701 explanation)

Failed merges:

 - rust-lang#73708 (Explain move errors that occur due to method calls involving `self` (take two))

r? @ghost
@bors bors merged commit 23c9ac6 into rust-lang:master Jun 26, 2020
@crlf0710 crlf0710 added the F-non_ascii_idents `#![feature(non_ascii_idents)]` label Jun 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
F-non_ascii_idents `#![feature(non_ascii_idents)]` S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants