Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use the roslyn tokenizer #10702

Merged
merged 17 commits into from
Aug 12, 2024
Merged

Use the roslyn tokenizer #10702

merged 17 commits into from
Aug 12, 2024

Commits on Aug 1, 2024

  1. Remove unused ITokenizer interface, make tokenizers disposable.

    @333fred
    333fred committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    894a9ba View commit details
    Browse the repository at this point in the history

  2. Remove unneeded common states

    @333fred
    333fred committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    a15d7cc View commit details
    Browse the repository at this point in the history

  3. Allow setting the next tokenizer state after a razor comment body is … 

    …tokenized.
    @333fred
    333fred committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    3a8fd07 View commit details
    Browse the repository at this point in the history

  4. Plumb the Roslyn tokenizer through more tests

    @333fred
    333fred committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    60ac8d1 View commit details
    Browse the repository at this point in the history

  5. Nullable enable the tokenizer

    @333fred
    333fred committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    73aa717 View commit details
    Browse the repository at this point in the history

  6. Add consolidated syntax kinds. Since there aren't consumers that care… 

    … about the individual differences between most C# operators, as well the difference between C# numeric types, we'll just use a single kind in the new parser to keep things simpler.
    @333fred
    333fred committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    dcef8ed View commit details
    Browse the repository at this point in the history

  7. Give better error messages when syntax tree verification fails.

    @333fred
    333fred committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    5749a8e View commit details
    Browse the repository at this point in the history

  8. The big change: use the roslyn tokenizer for tokenizing C#.

    @333fred
    333fred committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    2823f64 View commit details
    Browse the repository at this point in the history

  9. Update SyntaxKinds to the new consolidated kinds in tests.

    @333fred
    333fred committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    c25ac20 View commit details
    Browse the repository at this point in the history

  10. Increase test coverage of a few scenarios, and bulk up existing tests.

    @333fred
    333fred committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    34ab111 View commit details
    Browse the repository at this point in the history

  11. Skip test for now, give a better exception.

    @333fred
    333fred committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    abd6aba View commit details
    Browse the repository at this point in the history

  12. Name change

    @333fred
    333fred committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    acf7a60 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Better support giving errors on raw string literals, increase test co… 

    …verage.
    @333fred
    333fred committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    640d377 View commit details
    Browse the repository at this point in the history

  2. PR feedback.

    @333fred
    333fred committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    3155200 View commit details
    Browse the repository at this point in the history

  3. Convert to the correct type

    @333fred
    333fred committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    1a0d57d View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Additional commenting and PR feedback. I've simplified the reset loop… 

    … to just be from the back of the results, as that's the most common order for the parser to reset in. I've also refactored a common advance loop to reduce duplication.
    @333fred
    333fred committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    aa24f69 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2024

  1. More feedback

    @333fred
    333fred committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    5b7e927 View commit details
    Browse the repository at this point in the history