Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[test] Improve visual regression tests #19175

Merged
merged 1 commit into from
Jan 13, 2020
Merged

[test] Improve visual regression tests #19175

merged 1 commit into from
Jan 13, 2020

Conversation

oliviertassinari
Copy link
Member

@oliviertassinari oliviertassinari commented Jan 11, 2020 •
edited
Loading

We have had a bunch of test fails related to uploading the screenshots to argos-ci. I believe it's because of the endpoint on Heroku timeout after 30s. In this effort, I plan to /2 the number of screenshots we do.
We will benefit from a faster CI.

Edit: the test fails seems to be related to something else, it could be postgresql/network/fs related. The reduction in the test reduction comes from fewer screenshots and ignoring the material icons search.

Before

  • screenshots: 427
  • duration: ~02:31

After

  • screenshots: 298
  • duration: ~01:44

@oliviertassinari oliviertassinari force-pushed the clean-up-visual-regression-tests branch from 75af979 to 7db3fed Compare January 11, 2020 13:02
@mui-pr-bot
Copy link

mui-pr-bot commented Jan 11, 2020 •
edited
Loading

Details of bundle changes.

Comparing: 91168ef...c7c05d8

bundle Size Change Size Gzip Change Gzip
docs.main ▲ +88 B (+0.01% ) 615 kB ▲ +10 B (+0.01% ) 196 kB
Dialog ▲ +88 B (+0.11% ) 83.1 kB ▲ +9 B (+0.03% ) 26 kB
SwipeableDrawer ▲ +88 B (+0.10% ) 92.2 kB ▲ +8 B (+0.03% ) 29 kB
Drawer ▲ +88 B (+0.10% ) 84.8 kB ▲ +6 B (+0.02% ) 25.9 kB
Menu ▲ +88 B (+0.10% ) 88.8 kB ▲ +4 B (+0.01% ) 27.5 kB
Popover ▲ +88 B (+0.11% ) 83.1 kB ▲ +3 B (+0.01% ) 25.8 kB
Modal ▲ +88 B (+0.62% ) 14.4 kB ▲ +2 B (+0.04% ) 5.03 kB
Select ▲ +88 B (+0.08% ) 116 kB ▼ -2 B (-0.01% ) 34.5 kB
TablePagination ▲ +88 B (+0.06% ) 143 kB ▼ -2 B (-0.00% ) 41.8 kB
TextField ▲ +88 B (+0.07% ) 125 kB ▼ -2 B (-0.01% ) 36.5 kB
@material-ui/core ▲ +88 B (+0.02% ) 359 kB ▲ +1 B (0.00% ) 98.6 kB
Tabs ▲ +59 B (+0.07% ) 85.7 kB ▲ +15 B (+0.06% ) 27.3 kB
MenuList ▲ +58 B (+0.09% ) 66.3 kB ▲ +3 B (+0.01% ) 20.8 kB
Slider ▲ +29 B (+0.04% ) 76.8 kB ▲ +4 B (+0.02% ) 24.4 kB
ClickAwayListener ▲ +29 B (+0.75% ) 3.88 kB ▼ -2 B (-0.13% ) 1.54 kB
Snackbar ▲ +29 B (+0.04% ) 75.4 kB ▲ +2 B (+0.01% ) 23.7 kB
@material-ui/core[umd] -- 314 kB -- 90.8 kB
@material-ui/lab -- 184 kB -- 55 kB
@material-ui/styles -- 51.3 kB -- 15.5 kB
@material-ui/system -- 14.5 kB -- 4.05 kB
Alert -- 84 kB -- 26.4 kB
AlertTitle -- 64.4 kB -- 20.4 kB
AppBar -- 64.2 kB -- 20.2 kB
Autocomplete -- 131 kB -- 41.2 kB
Avatar -- 65.5 kB -- 20.7 kB
AvatarGroup -- 62.5 kB -- 19.7 kB
Backdrop -- 68 kB -- 21.1 kB
Badge -- 65.6 kB -- 20.5 kB
BottomNavigation -- 62.7 kB -- 19.7 kB
BottomNavigationAction -- 75.7 kB -- 24 kB
Box -- 71 kB -- 21.7 kB
Breadcrumbs -- 68.2 kB -- 21.5 kB
Button -- 79.9 kB -- 24.6 kB
ButtonBase -- 74.2 kB -- 23.4 kB
ButtonGroup -- 83.4 kB -- 25.6 kB
Card -- 63.1 kB -- 19.8 kB
CardActionArea -- 75.3 kB -- 23.8 kB
CardActions -- 62.3 kB -- 19.6 kB
CardContent -- 62.2 kB -- 19.6 kB
CardHeader -- 65.3 kB -- 20.6 kB
CardMedia -- 62.6 kB -- 19.8 kB
Checkbox -- 83.2 kB -- 26.4 kB
Chip -- 82.8 kB -- 25.5 kB
CircularProgress -- 64.4 kB -- 20.4 kB
Collapse -- 68.1 kB -- 21.2 kB
colorManipulator -- 3.85 kB -- 1.52 kB
Container -- 63.5 kB -- 19.9 kB
CssBaseline -- 57.8 kB -- 18.2 kB
DialogActions -- 62.3 kB -- 19.6 kB
DialogContent -- 62.5 kB -- 19.7 kB
DialogContentText -- 64.3 kB -- 20.3 kB
DialogTitle -- 64.5 kB -- 20.3 kB
Divider -- 62.8 kB -- 19.8 kB
docs.landing -- 50.9 kB -- 13.5 kB
ExpansionPanel -- 72.4 kB -- 22.8 kB
ExpansionPanelActions -- 62.3 kB -- 19.6 kB
ExpansionPanelDetails -- 62.2 kB -- 19.6 kB
ExpansionPanelSummary -- 78.3 kB -- 24.8 kB
Fab -- 77 kB -- 24.1 kB
Fade -- 23.3 kB -- 7.99 kB
FilledInput -- 73.8 kB -- 23 kB
FormControl -- 64.6 kB -- 20.2 kB
FormControlLabel -- 65.8 kB -- 20.7 kB
FormGroup -- 62.3 kB -- 19.6 kB
FormHelperText -- 63.5 kB -- 20 kB
FormLabel -- 63.7 kB -- 19.8 kB
Grid -- 65.3 kB -- 20.6 kB
GridList -- 62.7 kB -- 19.8 kB
GridListTile -- 64 kB -- 20.1 kB
GridListTileBar -- 63.5 kB -- 20 kB
Grow -- 23.9 kB -- 8.21 kB
Hidden -- 66.2 kB -- 20.8 kB
Icon -- 63 kB -- 19.8 kB
IconButton -- 76.4 kB -- 23.9 kB
Input -- 72.7 kB -- 22.8 kB
InputAdornment -- 65.3 kB -- 20.6 kB
InputBase -- 70.8 kB -- 22.3 kB
InputLabel -- 65.6 kB -- 20.3 kB
LinearProgress -- 65.6 kB -- 20.5 kB
Link -- 66.8 kB -- 21.2 kB
List -- 62.6 kB -- 19.6 kB
ListItem -- 77.3 kB -- 24.3 kB
ListItemAvatar -- 62.4 kB -- 19.6 kB
ListItemIcon -- 62.4 kB -- 19.6 kB
ListItemSecondaryAction -- 62.3 kB -- 19.6 kB
ListItemText -- 65.2 kB -- 20.6 kB
ListSubheader -- 63 kB -- 19.9 kB
MenuItem -- 78.4 kB -- 24.6 kB
MobileStepper -- 68.1 kB -- 21.4 kB
NativeSelect -- 77 kB -- 24.4 kB
NoSsr -- 2.19 kB -- 1.04 kB
OutlinedInput -- 74.2 kB -- 23.2 kB
Paper -- 62.6 kB -- 19.6 kB
Popper -- 28.7 kB -- 10.3 kB
Portal -- 2.9 kB -- 1.3 kB
Radio -- 84.3 kB -- 26.7 kB
RadioGroup -- 64.7 kB -- 20.1 kB
Rating -- 70.7 kB -- 22.8 kB
RootRef -- 4.21 kB -- 1.64 kB
Skeleton -- 63.2 kB -- 20.1 kB
Slide -- 25.3 kB -- 8.72 kB
SnackbarContent -- 63.8 kB -- 20.2 kB
SpeedDial -- 86.3 kB -- 27.3 kB
SpeedDialAction -- 118 kB -- 37.5 kB
SpeedDialIcon -- 64.8 kB -- 20.4 kB
Step -- 62.9 kB -- 19.8 kB
StepButton -- 82.5 kB -- 26.2 kB
StepConnector -- 63 kB -- 19.9 kB
StepContent -- 69.2 kB -- 21.8 kB
StepIcon -- 64.9 kB -- 20.3 kB
StepLabel -- 68.8 kB -- 21.7 kB
Stepper -- 65.1 kB -- 20.6 kB
styles/createMuiTheme -- 16.5 kB -- 5.85 kB
SvgIcon -- 63.3 kB -- 19.8 kB
Switch -- 82.4 kB -- 26.1 kB
Tab -- 76.6 kB -- 24.3 kB
Table -- 62.8 kB -- 19.8 kB
TableBody -- 62.4 kB -- 19.6 kB
TableCell -- 64.3 kB -- 20.3 kB
TableContainer -- 62.2 kB -- 19.6 kB
TableFooter -- 62.4 kB -- 19.6 kB
TableHead -- 62.4 kB -- 19.6 kB
TableRow -- 62.8 kB -- 19.8 kB
TableSortLabel -- 77.6 kB -- 24.5 kB
TextareaAutosize -- 5.09 kB -- 2.14 kB
ToggleButton -- 76.4 kB -- 24.3 kB
ToggleButtonGroup -- 63.5 kB -- 20 kB
Toolbar -- 62.6 kB -- 19.7 kB
Tooltip -- 102 kB -- 32.4 kB
TreeItem -- 74 kB -- 23.5 kB
TreeView -- 66.9 kB -- 21.1 kB
Typography -- 63.9 kB -- 20.1 kB
useAutocomplete -- 14.3 kB -- 5.18 kB
useMediaQuery -- 2.5 kB -- 1.05 kB
Zoom -- 23.4 kB -- 8.11 kB

Generated by 🚫 dangerJS against c7c05d8

@oliviertassinari oliviertassinari force-pushed the clean-up-visual-regression-tests branch 9 times, most recently from abfc009 to 4707968 Compare January 12, 2020 01:31
@oliviertassinari oliviertassinari force-pushed the clean-up-visual-regression-tests branch from 4707968 to c7c05d8 Compare January 12, 2020 01:34
@oliviertassinari oliviertassinari marked this pull request as ready for review January 12, 2020 01:36
eps1lon
eps1lon reviewed Jan 12, 2020
View reviewed changes
Copy link
Member

@eps1lon eps1lon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm missing why some tests where deleted outright. Did you just merge some tests into a single screenshot or actually removed some tests? If so why did you choose certain tests over others?

@oliviertassinari
Copy link
Member Author

oliviertassinari commented Jan 12, 2020 •
edited
Loading

@eps1lon We used to write dedicated visual regression tests before including all the demos.
I have tried to remove as many tests that duplicate the coverage.

@eps1lon eps1lon merged commit e0bd732 into mui:master Jan 13, 2020
@oliviertassinari oliviertassinari deleted the clean-up-visual-regression-tests branch January 13, 2020 09:19
@oliviertassinari
Copy link
Member Author

oliviertassinari commented Jan 13, 2020 •
edited
Loading

The one change I was unsure of was regarding the removal of the system tests. I have assumed that it would be equivalent to testing the output of the style functions, as plain JavaScript objects.

@oliviertassinari
Copy link
Member Author

oliviertassinari commented Jan 13, 2020 •
edited
Loading

I had a chat with @gregberge, told me that the company we wrote Argos for runs it on a dedicated Heroku instance, with 1,000 screenshots/PR and 100 workers. So, It seems that it will be able to scale, it's more a matter of pouring a few $/month to support the server infrastructure. So, it seems that the main bottleneck for us will be how we generate these screenshots (serial generation) and these weird random timeouts (>30s).

@gregberge
Copy link
Contributor

You have to uploads screenshot in two different batches:

https://github.com/argos-ci/argos-cli#upload

@oliviertassinari
Copy link
Member Author

Thanks for the tip.

@oliviertassinari oliviertassinari mentioned this pull request Jan 14, 2020
Merged
@snyk-bot snyk-bot mentioned this pull request Feb 15, 2020
Open
This was referenced Feb 23, 2020
Open
Closed
Merged
@snyk-bot snyk-bot mentioned this pull request Mar 20, 2020
Merged
@snyk-bot snyk-bot mentioned this pull request Mar 21, 2020
Merged
This was referenced Apr 6, 2020
Open
Closed
This was referenced Apr 16, 2020
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eps1lon eps1lon eps1lon left review comments

Assignees
No one assigned
Labels
test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@oliviertassinari @mui-pr-bot @gregberge @eps1lon