Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce a size limit in getSpreadType #40755

Merged
merged 2 commits into from
Sep 30, 2020
Merged

Conversation

amcasey
Copy link
Member

@amcasey amcasey commented Sep 24, 2020

When a union is spread into a union, the sizes are multiplied,
potentially resulting in an enormous union (especially if there are
repeated spreads). This check detects cases that used to run out of
memory.

Fixes #40754

When a union is spread into a union, the sizes are multiplied,
potentially resulting in an enormous union (especially if there are
repeated spreads).  This check detects cases that used to run out of
memory.

Fixes microsoft#40754
@typescript-bot typescript-bot added the For Uncommitted Bug PR for untriaged, rejected, closed or missing bug label Sep 24, 2020
@amcasey
Copy link
Member Author

amcasey commented Sep 24, 2020

@typescript-bot user test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Sep 24, 2020

Heya @amcasey, I've started to run the parallelized community code test suite on this PR at 6650496. You can monitor the build here.

const resultSize = (left as UnionType).types.length * (right as UnionType).types.length;
if (resultSize > 100000) {
tracing.instant(tracing.Phase.Check, "getSpreadType_DepthLimit", { leftId: left.id, rightId: right.id });
error(currentNode, Diagnostics.Spread_expression_produces_a_union_type_that_is_too_complex_to_represent);
Copy link
Member

@weswigham weswigham Sep 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this error is going to be frustrating to encounter with it also providing a suggestion for a workaround. What is the general-form workaround? Casting something somewhere to any? Should we add a related span/quickfix for that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's a fix I'm sufficiently confident in to make it a quick fix. I gave this a new error number so that it would be easier to document/google.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did wonder about looking inside the union to figure out whether there's a smaller span we could squiggle.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the real code where this arose, the mitigation I suggested was adding a cast to each spread with a pre-existing type that listed all possible properties (optionally). I imagine any would also work, possible at the expense of completions and such.

Copy link
Member

@weswigham weswigham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code is definitely fine, and, even if we broaden #34853 to fix the reported issue without an error (which I think we should consider), we'll still fundamentally need this as a fallback.

@amcasey
Copy link
Member Author

amcasey commented Sep 24, 2020

If we broaden #34853, I'd probably drop the limit substantially. I based it on what would still compile quickly in my toy example.

@amcasey
Copy link
Member Author

amcasey commented Sep 24, 2020

@typescript-bot test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Sep 24, 2020

Heya @amcasey, I've started to run the extended test suite on this PR at 6650496. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

The user suite test run you requested has finished and failed. I've opened a PR with the baseline diff from master.

@amcasey
Copy link
Member Author

amcasey commented Sep 24, 2020

Since there are no occurrences of my new error code in the user test baseline diff, I'm assuming I can ignore the failure.

@DanielRosenwasser
Copy link
Member

@typescript-bot pack this
@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Sep 25, 2020

Heya @DanielRosenwasser, I've started to run the tarball bundle task on this PR at 6650496. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Sep 25, 2020

Heya @DanielRosenwasser, I've started to run the perf test suite on this PR at 6650496. You can monitor the build here.

Update: The results are in!

@DanielRosenwasser
Copy link
Member

@typescript-bot test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Sep 25, 2020

Heya @DanielRosenwasser, I've started to run the extended test suite on this PR at 6650496. You can monitor the build here.

if (resultSize > 100000) {
tracing.instant(tracing.Phase.Check, "getSpreadType_DepthLimit", { leftId: left.id, rightId: right.id });
error(currentNode, Diagnostics.Spread_expression_produces_a_union_type_that_is_too_complex_to_represent);
return errorType;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@minestarks raised an interesting point: should this be anyType so that users have the option of ts-ignoring this error?

@typescript-bot
Copy link
Collaborator

typescript-bot commented Sep 25, 2020

Hey @DanielRosenwasser, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/86514/artifacts?artifactName=tgz&fileId=D9585AAA2ECA42B07D99CAE4C35EB7EBDFBD0966E447E25C919235B4C846210402&fileName=/typescript-4.1.0-insiders.20200925.tgz"
    }
}

and then running npm install.


There is also a playground for this build.

@typescript-bot
Copy link
Collaborator

@DanielRosenwasser
The results of the perf run you requested are in!

Here they are:

Comparison Report - master..40755

Metric master 40755 Delta Best Worst
Angular - node (v10.16.3, x64)
Memory used 349,631k (± 0.02%) 349,621k (± 0.02%) -9k (- 0.00%) 349,480k 349,713k
Parse Time 2.00s (± 0.50%) 2.01s (± 0.53%) +0.01s (+ 0.45%) 1.99s 2.03s
Bind Time 0.83s (± 0.80%) 0.83s (± 0.63%) +0.00s (+ 0.12%) 0.82s 0.84s
Check Time 4.91s (± 0.54%) 4.92s (± 0.60%) +0.01s (+ 0.18%) 4.86s 4.97s
Emit Time 5.23s (± 0.90%) 5.21s (± 0.51%) -0.02s (- 0.36%) 5.16s 5.27s
Total Time 12.97s (± 0.40%) 12.96s (± 0.31%) -0.00s (- 0.02%) 12.88s 13.06s
Monaco - node (v10.16.3, x64)
Memory used 354,336k (± 0.02%) 354,356k (± 0.02%) +20k (+ 0.01%) 354,171k 354,534k
Parse Time 1.56s (± 0.31%) 1.56s (± 0.38%) +0.01s (+ 0.45%) 1.55s 1.58s
Bind Time 0.71s (± 0.69%) 0.72s (± 1.06%) +0.00s (+ 0.28%) 0.70s 0.73s
Check Time 5.05s (± 0.47%) 5.07s (± 0.73%) +0.02s (+ 0.38%) 4.99s 5.16s
Emit Time 2.75s (± 0.55%) 2.79s (± 0.97%) +0.05s (+ 1.64%) 2.74s 2.85s
Total Time 10.07s (± 0.30%) 10.14s (± 0.61%) +0.07s (+ 0.67%) 9.99s 10.28s
TFS - node (v10.16.3, x64)
Memory used 307,564k (± 0.02%) 307,574k (± 0.02%) +10k (+ 0.00%) 307,433k 307,704k
Parse Time 1.22s (± 0.62%) 1.22s (± 0.39%) +0.00s (+ 0.33%) 1.21s 1.23s
Bind Time 0.67s (± 0.74%) 0.67s (± 1.08%) -0.00s (- 0.44%) 0.65s 0.68s
Check Time 4.53s (± 0.48%) 4.54s (± 0.51%) +0.00s (+ 0.11%) 4.48s 4.59s
Emit Time 2.89s (± 0.81%) 2.91s (± 1.17%) +0.02s (+ 0.62%) 2.84s 2.99s
Total Time 9.31s (± 0.45%) 9.33s (± 0.52%) +0.03s (+ 0.29%) 9.24s 9.43s
material-ui - node (v10.16.3, x64)
Memory used 488,962k (± 0.01%) 488,937k (± 0.02%) -25k (- 0.01%) 488,709k 489,106k
Parse Time 1.99s (± 0.53%) 2.00s (± 0.68%) +0.00s (+ 0.20%) 1.98s 2.04s
Bind Time 0.65s (± 0.91%) 0.65s (± 0.61%) -0.00s (- 0.46%) 0.64s 0.66s
Check Time 13.61s (± 0.91%) 13.53s (± 1.06%) -0.08s (- 0.57%) 13.29s 13.88s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 16.26s (± 0.81%) 16.18s (± 0.92%) -0.08s (- 0.47%) 15.92s 16.55s
Angular - node (v12.1.0, x64)
Memory used 326,702k (± 0.09%) 326,810k (± 0.02%) +107k (+ 0.03%) 326,682k 326,975k
Parse Time 1.99s (± 0.40%) 2.00s (± 0.62%) +0.01s (+ 0.40%) 1.97s 2.03s
Bind Time 0.81s (± 0.58%) 0.81s (± 0.84%) -0.00s (- 0.49%) 0.79s 0.82s
Check Time 4.82s (± 0.99%) 4.84s (± 0.96%) +0.02s (+ 0.44%) 4.76s 4.97s
Emit Time 5.41s (± 1.06%) 5.37s (± 0.60%) -0.04s (- 0.78%) 5.30s 5.43s
Total Time 13.03s (± 0.59%) 13.01s (± 0.57%) -0.02s (- 0.15%) 12.87s 13.15s
Monaco - node (v12.1.0, x64)
Memory used 336,564k (± 0.02%) 336,492k (± 0.02%) -72k (- 0.02%) 336,346k 336,718k
Parse Time 1.55s (± 0.81%) 1.54s (± 0.84%) -0.01s (- 0.45%) 1.51s 1.57s
Bind Time 0.69s (± 0.64%) 0.69s (± 0.72%) +0.00s (+ 0.43%) 0.69s 0.71s
Check Time 4.86s (± 0.53%) 4.87s (± 0.43%) +0.01s (+ 0.12%) 4.81s 4.90s
Emit Time 2.82s (± 0.70%) 2.82s (± 0.66%) -0.00s (- 0.04%) 2.78s 2.87s
Total Time 9.92s (± 0.46%) 9.92s (± 0.34%) +0.00s (+ 0.01%) 9.84s 10.00s
TFS - node (v12.1.0, x64)
Memory used 291,855k (± 0.03%) 291,835k (± 0.01%) -19k (- 0.01%) 291,754k 291,951k
Parse Time 1.23s (± 0.54%) 1.23s (± 0.72%) -0.00s (- 0.24%) 1.21s 1.24s
Bind Time 0.64s (± 0.92%) 0.64s (± 0.62%) -0.00s (- 0.47%) 0.63s 0.65s
Check Time 4.43s (± 0.35%) 4.44s (± 0.49%) +0.01s (+ 0.16%) 4.39s 4.50s
Emit Time 2.94s (± 0.92%) 2.94s (± 0.77%) -0.00s (- 0.03%) 2.90s 3.00s
Total Time 9.24s (± 0.31%) 9.24s (± 0.38%) -0.00s (- 0.02%) 9.20s 9.33s
material-ui - node (v12.1.0, x64)
Memory used 466,943k (± 0.06%) 467,006k (± 0.01%) +63k (+ 0.01%) 466,873k 467,147k
Parse Time 2.00s (± 0.41%) 2.01s (± 0.63%) +0.01s (+ 0.55%) 1.98s 2.04s
Bind Time 0.64s (± 0.63%) 0.64s (± 0.57%) -0.00s (- 0.47%) 0.63s 0.64s
Check Time 12.09s (± 0.87%) 12.07s (± 0.75%) -0.02s (- 0.17%) 11.90s 12.25s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 14.73s (± 0.76%) 14.72s (± 0.67%) -0.01s (- 0.10%) 14.53s 14.91s
Angular - node (v8.9.0, x64)
Memory used 346,320k (± 0.02%) 346,249k (± 0.02%) -70k (- 0.02%) 346,040k 346,350k
Parse Time 2.55s (± 0.26%) 2.55s (± 0.71%) -0.00s (- 0.04%) 2.53s 2.62s
Bind Time 0.87s (± 0.75%) 0.86s (± 0.60%) -0.01s (- 0.81%) 0.85s 0.88s
Check Time 5.55s (± 0.41%) 5.55s (± 0.80%) -0.00s (- 0.02%) 5.45s 5.64s
Emit Time 6.08s (± 2.63%) 6.19s (± 1.34%) +0.11s (+ 1.88%) 6.06s 6.39s
Total Time 15.05s (± 1.05%) 15.16s (± 0.79%) +0.11s (+ 0.71%) 14.99s 15.52s
Monaco - node (v8.9.0, x64)
Memory used 355,609k (± 0.02%) 355,605k (± 0.02%) -4k (- 0.00%) 355,429k 355,708k
Parse Time 1.88s (± 0.36%) 1.89s (± 0.51%) +0.00s (+ 0.05%) 1.87s 1.91s
Bind Time 0.89s (± 0.76%) 0.89s (± 0.53%) -0.00s (- 0.11%) 0.88s 0.90s
Check Time 5.62s (± 0.45%) 5.62s (± 0.40%) +0.00s (+ 0.09%) 5.57s 5.67s
Emit Time 3.29s (± 1.35%) 3.29s (± 1.27%) -0.01s (- 0.18%) 3.19s 3.35s
Total Time 11.68s (± 0.46%) 11.68s (± 0.42%) -0.00s (- 0.02%) 11.59s 11.81s
TFS - node (v8.9.0, x64)
Memory used 309,284k (± 0.02%) 309,317k (± 0.02%) +33k (+ 0.01%) 309,202k 309,485k
Parse Time 1.55s (± 0.37%) 1.55s (± 0.34%) +0.00s (+ 0.26%) 1.54s 1.56s
Bind Time 0.67s (± 0.66%) 0.68s (± 1.18%) +0.01s (+ 0.89%) 0.66s 0.69s
Check Time 5.31s (± 0.40%) 5.33s (± 0.54%) +0.02s (+ 0.38%) 5.29s 5.41s
Emit Time 2.93s (± 0.97%) 2.95s (± 0.59%) +0.02s (+ 0.51%) 2.92s 3.00s
Total Time 10.46s (± 0.37%) 10.51s (± 0.40%) +0.05s (+ 0.46%) 10.44s 10.63s
material-ui - node (v8.9.0, x64)
Memory used 493,374k (± 0.01%) 493,373k (± 0.01%) -1k (- 0.00%) 493,226k 493,631k
Parse Time 2.41s (± 0.25%) 2.41s (± 0.51%) +0.00s (+ 0.08%) 2.38s 2.44s
Bind Time 0.81s (± 1.00%) 0.81s (± 1.54%) +0.00s (+ 0.12%) 0.79s 0.84s
Check Time 18.03s (± 0.77%) 18.03s (± 0.97%) -0.00s (- 0.03%) 17.78s 18.56s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 21.26s (± 0.68%) 21.25s (± 0.88%) -0.01s (- 0.03%) 20.99s 21.80s
Angular - node (v8.9.0, x86)
Memory used 198,619k (± 0.03%) 198,563k (± 0.03%) -56k (- 0.03%) 198,411k 198,676k
Parse Time 2.47s (± 0.83%) 2.47s (± 0.67%) +0.00s (+ 0.12%) 2.45s 2.53s
Bind Time 1.01s (± 0.95%) 1.01s (± 1.20%) -0.00s (- 0.49%) 0.98s 1.03s
Check Time 5.00s (± 0.45%) 5.00s (± 0.58%) +0.00s (+ 0.06%) 4.94s 5.07s
Emit Time 5.92s (± 0.87%) 5.95s (± 0.76%) +0.03s (+ 0.49%) 5.84s 6.05s
Total Time 14.40s (± 0.52%) 14.43s (± 0.54%) +0.03s (+ 0.20%) 14.25s 14.59s
Monaco - node (v8.9.0, x86)
Memory used 201,423k (± 0.01%) 201,444k (± 0.02%) +21k (+ 0.01%) 201,364k 201,525k
Parse Time 1.93s (± 0.55%) 1.93s (± 1.19%) 0.00s ( 0.00%) 1.90s 2.01s
Bind Time 0.70s (± 0.53%) 0.71s (± 1.02%) +0.01s (+ 0.99%) 0.70s 0.73s
Check Time 5.49s (± 1.48%) 5.45s (± 0.60%) -0.04s (- 0.67%) 5.39s 5.52s
Emit Time 3.04s (± 2.74%) 3.06s (± 0.54%) +0.02s (+ 0.76%) 3.03s 3.10s
Total Time 11.15s (± 0.36%) 11.15s (± 0.45%) -0.00s (- 0.03%) 11.09s 11.29s
TFS - node (v8.9.0, x86)
Memory used 176,823k (± 0.03%) 176,823k (± 0.01%) -0k (- 0.00%) 176,773k 176,890k
Parse Time 1.60s (± 0.99%) 1.59s (± 0.86%) -0.00s (- 0.25%) 1.57s 1.62s
Bind Time 0.65s (± 1.23%) 0.64s (± 0.62%) -0.01s (- 1.08%) 0.63s 0.65s
Check Time 4.81s (± 0.49%) 4.80s (± 0.41%) -0.01s (- 0.31%) 4.75s 4.84s
Emit Time 2.79s (± 1.13%) 2.83s (± 1.30%) +0.04s (+ 1.47%) 2.75s 2.91s
Total Time 9.84s (± 0.49%) 9.86s (± 0.46%) +0.02s (+ 0.17%) 9.77s 9.96s
material-ui - node (v8.9.0, x86)
Memory used 277,785k (± 0.01%) 277,807k (± 0.02%) +22k (+ 0.01%) 277,669k 277,962k
Parse Time 2.47s (± 0.82%) 2.49s (± 0.96%) +0.02s (+ 0.61%) 2.46s 2.57s
Bind Time 0.72s (± 4.58%) 0.73s (± 5.84%) +0.01s (+ 1.67%) 0.68s 0.83s
Check Time 16.49s (± 1.36%) 16.46s (± 0.60%) -0.03s (- 0.16%) 16.28s 16.68s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 19.68s (± 1.24%) 19.69s (± 0.69%) +0.01s (+ 0.04%) 19.45s 20.09s
System
Machine Namets-ci-ubuntu
Platformlinux 4.4.0-166-generic
Architecturex64
Available Memory16 GB
Available Memory1 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v10.16.3, x64)
  • node (v12.1.0, x64)
  • node (v8.9.0, x64)
  • node (v8.9.0, x86)
Scenarios
  • Angular - node (v10.16.3, x64)
  • Angular - node (v12.1.0, x64)
  • Angular - node (v8.9.0, x64)
  • Angular - node (v8.9.0, x86)
  • Monaco - node (v10.16.3, x64)
  • Monaco - node (v12.1.0, x64)
  • Monaco - node (v8.9.0, x64)
  • Monaco - node (v8.9.0, x86)
  • TFS - node (v10.16.3, x64)
  • TFS - node (v12.1.0, x64)
  • TFS - node (v8.9.0, x64)
  • TFS - node (v8.9.0, x86)
  • material-ui - node (v10.16.3, x64)
  • material-ui - node (v12.1.0, x64)
  • material-ui - node (v8.9.0, x64)
  • material-ui - node (v8.9.0, x86)
Benchmark Name Iterations
Current 40755 10
Baseline master 10

@amcasey
Copy link
Member Author

amcasey commented Sep 25, 2020

This should probably use checkCrossProductUnion.

@amcasey amcasey merged commit 3511123 into microsoft:master Sep 30, 2020
@amcasey amcasey deleted the SpreadLimit branch September 30, 2020 20:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
For Uncommitted Bug PR for untriaged, rejected, closed or missing bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Repeated conditional spreading of object literals results in OOM
4 participants