Tokenizer: improve tokenization of T_NULLABLE vs T_INLINE_THEN #2794
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is an attempt to make the differentiation between T_NULLABLE and T_INLINE_THEN more stable. There is only a limited set of tokens which can be used for type declarations. If a `?` is not followed by one of these, we can be certain that it is a ternary operator `T_INLINE_THEN`. The more resource intensive "walking back" check will now only be done when it could be either. The "walking back" token logic could probably still do with an additional check for `T_INSTANCEOF`, but the changes now made, should prevent the majority of issues where `T_INLINE_THEN` is misidentified as `T_NULLABLE`. Includes unit tests via the PSR12.Functions.NullableTypeDeclaration sniff.
jrfnl
force-pushed
the
feature/2791-tokenizer-issue-static-vs-typed-properties
branch
from
d70407d to
603d9c6
Compare
|
Rebased to show the build passes. |
Thanks a lot. This is a much better idea than how I originally implemented it. |
jrfnl
deleted the
feature/2791-tokenizer-issue-static-vs-typed-properties
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an attempt to make the differentiation between
T_NULLABLEandT_INLINE_THENmore stable.There is only a limited set of tokens which can be used for type declarations. If a
?is not followed by one of these, we can be certain that it is a ternary operatorT_INLINE_THEN.The more resource intensive "walking back" check will now only be done when it could be either.
The "walking back" token logic could probably still do with an additional check for
T_INSTANCEOF, but the changes now made, should prevent the majority of issues whereT_INLINE_THENis misidentified asT_NULLABLE.Includes unit tests via the PSR12.Functions.NullableTypeDeclaration sniff.
Fixes #2791
Note: the build failure is unrelated to this PR and will be fixed via #2793