Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added extra newlines in pretty tokenstream #2070

Merged

Conversation

RunDevelopment
Copy link
Member

This changes the pretty simplified token stream to have an extra newline if the test code also has a double newline there.

Example
+ - * / %
> >= <= < == != =
!~ =~

! && ||

[**] [***] [****]

..
1..100

-------------------------------------------------------------

[
    ["operator", "+"],
    ["operator", "-"],
    ["operator", "*"],
    ["operator", "/"],
    ["operator", "%"],
    ["operator", ">"],
    ["operator", ">="],
    ["operator", "<="],
    ["operator", "<"],
    ["operator", "=="],
    ["operator", "!="],
    ["operator", "="],
    ["operator", "!~"],
    ["operator", "=~"],

    ["operator", "!"],
    ["operator", "&&"],
    ["operator", "||"],

    ["punctuation", "["],
    ["operator", "**"],
    ["punctuation", "]"],
    ["punctuation", "["],
    ["operator", "***"],
    ["punctuation", "]"],
    ["punctuation", "["],
    ["operator", "****"],
    ["punctuation", "]"],

    ["range", ".."],
    ["number", "1"],
    ["range", ".."],
    ["number", "100"]
]

------------------------------------------------------

This is the output pretty token stream as is.
The different operator groups are neatly separated by an extra newline.

This makes it easier to read long token streams without having to do any manual formatting.
I wanted this feature because this is how I usually format pretty token streams.

@RunDevelopment
Copy link
Member Author

I can't merge this without a definition of a token stream. (These tests don't fail for nothing.)

Sorry, if I haven't made this clear that #2071 blocks this PR.

@mAAdhaTTah
Copy link
Member

Oh, I see, my bad. So as I commented on #2071, I would be in favor of bringing the types into alignment with the actual runtime behavior. Whatever properties of TokenStream tokenize actually produces at runtime should be reflected in the TokenStream type. If our current types are suggesting it returns things it doesn't, then I'd go with the simpler types, especially if your code has made assumptions about what tokenize produces and functional correctly with those assumptions.

@RunDevelopment
Copy link
Member Author

Ok yeah. I just looked at Markup templating and it also assumes the narrower definition. The minor difference here: While this code might break for other definition, MT is going to break, no questions. (See walkTokens in MT).

I'm gonna change the definition in #1782.

@RunDevelopment RunDevelopment mentioned this pull request Jun 27, 2020
@RunDevelopment RunDevelopment merged commit 681adee into PrismJS:master Jun 28, 2020
@RunDevelopment RunDevelopment deleted the pretty-tokenstream-newlines branch June 28, 2020 01:16
quentinvernot pushed a commit to TankerHQ/prismjs that referenced this pull request Sep 11, 2020
The formatter for the simplified token streams of our test cases will now emit extra (empty) newlines if the tested code also has an empty line at those positions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants