Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance Benchmarks for WordPress 6.7 #1572

Closed
mukeshpanchal27 opened this issue Oct 1, 2024 · 31 comments
Closed

Performance Benchmarks for WordPress 6.7 #1572

mukeshpanchal27 opened this issue Oct 1, 2024 · 31 comments
Assignees
Labels
[Type] Epic A high-level project / epic that will encompass several sub-issues WP Core Work relates to inclusion in WP Core only

Comments

@mukeshpanchal27
Copy link
Member

As we enter the beta phase for WordPress 6.7, I am opening this issue to gather and track all performance benchmarks. This will help us collaborate more effectively, identify potential regressions, and ensure all performance-related data is centralized in one place.

Feel free to contribute with your benchmarks, results, or any findings related to performance in WordPress 6.7.

@mukeshpanchal27 mukeshpanchal27 added [Type] Epic A high-level project / epic that will encompass several sub-issues WP Core Work relates to inclusion in WP Core only labels Oct 1, 2024
@mukeshpanchal27 mukeshpanchal27 self-assigned this Oct 1, 2024
@github-project-automation github-project-automation bot moved this to Not Started/Backlog 📆 in WP Performance 2024 Oct 1, 2024
@mukeshpanchal27 mukeshpanchal27 moved this from Not Started/Backlog 📆 to In Progress 🚧 in WP Performance 2024 Oct 1, 2024
@mukeshpanchal27
Copy link
Member Author

mukeshpanchal27 commented Oct 1, 2024

WP 6.7 Beta 1 Performance Benchmark Report:

I ran benchmarks comparing WordPress 6.7 Beta 1 and the latest released 6.6 branch (6.6.2). I took measurements with the twentytwentyone (TT1) theme using en_US locale, both with object caching disabled and enabled, then again with twentytwentyfour (TT4), This version of WordPress shows a regression in server rendering time from the previous release for TT4.

With object caching disabled, Classic themes (TT1) show a ~3% regression (0.35ms slower) in wp-before-template measurements; LCP remains relatively stable. Block themes (TT4) show a ~6% regression in TTFB (6ms slower) and a ~8% regression (8 ms slower) for wp-total measurements. LCP shows a ~4.5% regression (7ms). Running the tests with object caching enabled shows similar regressions.

Here are the results of the test runs:

Caching Disabled:
Classic theme - https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11122906286
Block theme - https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11123131063

Caching Enabled:
Classic theme - https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11123295178
Block theme - https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11123530293

cc. @felixarntz @joemcgill @adamsilverstein @westonruter @swissspidy

@joemcgill
Copy link
Member

Thanks @mukeshpanchal27. Repeating what I said in our weekly performance team chat earlier today:

the performance metrics for classic themes looks pretty steady, which is great! For TT4, it seems like we need to investigate the cause of the additional performance regression during template rendering, because adding 10% is not great.

I started trying to investigate the likely cause by looking at the WordPress Code Vitals dashboard and realized that the last commit recorded to that dashboard seems to be this one from five months ago 😱. It's likely that the following commit, r58165, likely broke these.

Fortunately, the Gutenberg tests are still being recorded, and it looks like this commit is worth investigating further. It was synced to WordPress in r58936. Here's the performance run from that commit in WP-dev.

@joemcgill
Copy link
Member

I've opened https://core.trac.wordpress.org/ticket/62153 to fix the dashboard issue.

@mukeshpanchal27
Copy link
Member Author

I did the quick compare through xhprof. Here is some finding.

  • Use of strpos is significantly more then the previous version.

The compression between 6.6.2 and 6.7 beta1

Screenshot 2024-10-02 at 11 51 41 AM

6.7 beta1 strpos function details:

Screenshot 2024-10-02 at 11 53 11 AM

6.6.2 strpos function details:

Screenshot 2024-10-02 at 12 17 48 PM

@mukeshpanchal27
Copy link
Member Author

mukeshpanchal27 commented Oct 2, 2024

Fortunately, the Gutenberg tests are still being recorded, and it looks like this commit is worth investigating further. It was synced to WordPress in r58936. Here's the performance run from that commit in WP-dev.

@joemcgill The code https://github.com/WordPress/wordpress-develop/blob/trunk/src/wp-includes/class-wp-theme-json.php#L3239-L3251 is too expensive see the profiling result. I removed the code and did the comparison.

Screenshot 2024-10-02 at 1 23 30 PM

Server-timing

Mertic Beta 1 Beta 1 with remove code Diff % Diff abs.
Response Time 252.84 247.12 -2.26% -5.72
wp-before-template 76.26 76.28 0.03% 0.02
wp-before-template-db-queries 3.3 3.35 1.52% 0.05
wp-template 173.42 167.66 -3.32% -5.76
wp-total 249.87 244.23 -2.26% -5.64
wp-template-db-queries 2.6 2.59 -0.38% -0.01

🔴 It clear shows that the code introduce ~3% regression

@mukeshpanchal27
Copy link
Member Author

JFYI, I re-open https://core.trac.wordpress.org/ticket/61858 so editor team also can check it.

@mukeshpanchal27
Copy link
Member Author

Opened PR WordPress/wordpress-develop#7486 that solve the regression. Waiting for the feedback.

@felixarntz
Copy link
Member

Since we now reinstated the Code Vitals dashboard, I just took a look at it and noticed https://www.codevitals.run/project/wordpress/home-block-theme-lcpMinusTtfb shows a notable regression that must have happened at some point while it was not working.

It's interesting that our Web Vitals benchmarks for "LCP - TTFB" didn't show a similar regression. Since the Code Vitals dashboard is still using the TT3 theme (which we should probably update at some point), it would be helpful to run another Web Vitals benchmark comparison using TT3, to check whether we see the regression there too - in which case the problem might be something that mostly affects TT3 for some reason.

@mukeshpanchal27
Copy link
Member Author

mukeshpanchal27 commented Oct 8, 2024

I ran performance benchmarks for Beta 1 with TT3 theme: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11228728635

Web Vitals benchmarks

Mertic WP 6.6.2 Beta 1 Diff % Diff abs.
FCP 99.7 99.15 -0.55% -0.55
LCP 99.7 99.15 -0.55% -0.55
TTFB 54.8 56 2.19% 1.20
LCP-TTFB 44.6 42.8 -4.04% -1.80

The benchmarks didn't show the regression that we see in dashboard.

@mukeshpanchal27
Copy link
Member Author

WP 6.7 Beta 2 Performance Benchmark Report:

With object caching disabled,

  • Classic themes (TT1) web vitals and server-timing remains relatively stable.
  • Block themes (TT3) show slight regression but remains relatively stable.
  • Block themes (TT4) show a ~6% regression in LCP shows a ~4.5% regression (7ms), TTFB shows a ~8% regression (6ms), wp-template shows a ~10% regression (6ms) and wp-total shows a ~10% regression (6ms)

Here are the results of the test runs:

TT1: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11228940062
TT3: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11229064497
TT4: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11229259874

TT1 (twentytwentyone) benchmarks

Web Vitals

Metric WP 6.6.2 WP 6.7 Beta 2 Diff % Diff abs.
FCP 102 102.65 0.64% 0.65
LCP 102 102.65 0.64% 0.65
TTFB 47.4 47.25 -0.32% -0.15
LCP-TTFB 54.8 55.2 0.73% 0.40

Server-Timing

Metric WP 6.6.2 WP 6.7 Beta 2 Diff % Diff abs.
Response Time 39.8 39.71 -0.23% -0.09
wp-before-template 11.47 11.61 1.22% 0.14
wp-template 27.33 27.21 -0.44% -0.12
wp-total 38.93 38.86 -0.18% -0.07

TT3 (twentytwentythree) benchmarks

Web Vitals

Metric WP 6.6.2 WP 6.7 Beta 2 Diff % Diff abs.
FCP 99.35 99.3 -0.05% -0.05
LCP 99.35 99.3 -0.05% -0.05
TTFB 55 55.95 1.73% 0.95
LCP-TTFB 43.35 42.55 -1.85% -0.80

Server-Timing

Metric WP 6.6.2 WP 6.7 Beta 2 Diff % Diff abs.
Response Time 47.64 47.9 0.55% 0.26
wp-before-template 15.89 15.77 -0.76% -0.12
wp-template 30.76 31.15 1.27% 0.39
wp-total 46.63 46.89 0.56% 0.26

TT4 (twentytwentyfour) benchmarks

Web Vitals

Metric WP 6.6.2 WP 6.7 Beta 2 Diff % Diff abs.
FCP 155.55 162.75 4.63% 7.20
LCP 155.55 162.75 4.63% 7.20
TTFB 81.55 88.35 8.34% 6.80
LCP-TTFB 73.8 73.7 -0.14% -0.10

Server-Timing

Metric WP 6.6.2 WP 6.7 Beta 2 Diff % Diff abs.
Response Time 74.11 80.16 8.16% 6.05
wp-before-template 16.31 16.38 0.43% 0.07
wp-template 56.7 62.62 10.44% 5.92
wp-total 72.95 79.01 8.31% 6.06

🔴 alert for TT4, In WordPress/wordpress-develop#7486 we are checking one approach to solve the background image regression that account for 3-4% regression in block theme.

@joemcgill
Copy link
Member

Following up on #1572 (comment), I noticed while testing a fix to the Performance Workflow that the LCP-TTFB metric has returned to a more "normal" range but doesn't seem related to any specific code change, so I'm thinking there was a temporary problem with the workflow that has cleared itself up.

Image

@mukeshpanchal27
Copy link
Member Author

It means we should debug as the difference it ~6ms for normal code commit.

@mukeshpanchal27
Copy link
Member Author

WP 6.7 Beta 3 Performance Benchmark Report:

With object caching disabled,

  • Classic themes (TT1) web vitals and server-timing remains relatively stable.
  • Block themes (TT3) show slight regression but remains relatively stable.
  • Block themes (TT4)
    • LCP shows a ~3% regression (~5ms)
    • TTFB shows a ~4% regression (~4ms)
    • wp-template shows a ~9% regression (5ms)
    • wp-total shows a ~7% regression (5ms)

Here are the results of the test runs:

TT1: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11339147727
TT3: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11339238140
TT4: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11339330952

TT1 (twentytwentyone) benchmarks

Web Vitals

Metric WP 6.6.2 WP 6.7 Beta 3 Diff % Diff abs.
FCP 100.5 99.7 -0.80% -0.80
LCP 100.5 99.7 -0.80% -0.80
TTFB 45.7 45.7 0.00% 0.00
LCP-TTFB 54.4 53.65 -1.38% -0.75

Server-Timing

Metric WP 6.6.2 WP 6.7 Beta 3 Diff % Diff abs.
Response Time 37.87 38.07 0.53% 0.20
wp-before-template 10.43 10.74 2.97% 0.31
wp-template 26.66 26.52 -0.53% -0.14
wp-total 37.14 37.31 0.46% 0.17

TT3 (twentytwentythree) benchmarks

Web Vitals

Metric WP 6.6.2 WP 6.7 Beta 3 Diff % Diff abs.
FCP 97.4 97.45 0.05% 0.05
LCP 97.4 97.45 0.05% 0.05
TTFB 53.95 54.5 1.02% 0.55
LCP-TTFB 42.9 42.75 -0.35% -0.15

Server-Timing

Metric WP 6.6.2 WP 6.7 Beta 3 Diff % Diff abs.
Response Time 47.29 47.74 0.95% 0.45
wp-before-template 15.88 15.95 0.44% 0.07
wp-template 30.3 30.56 0.86% 0.26
wp-total 46.19 46.59 0.87% 0.40

TT4 (twentytwentyfour) benchmarks

Web Vitals

Metric WP 6.6.2 WP 6.7 Beta 3 Diff % Diff abs.
FCP 157.2 162.05 3.09% 4.85
LCP 157.2 162.05 3.09% 4.85
TTFB 81.3 85.6 5.29% 4.30
LCP-TTFB 75.45 75.3 -0.20% -0.15

Server-Timing

Metric WP 6.6.2 WP 6.7 Beta 3 Diff % Diff abs.
Response Time 71.41 76.4 6.99% 4.99
wp-before-template 14.99 15.02 0.20% 0.03
wp-template 55.5 60.38 8.79% 4.88
wp-total 70.5 75.44 7.01% 4.94

Beta 3 shows improvement compared to Beta 2, but there is still a regression in Web Vitals and Server-Timing.

@mukeshpanchal27
Copy link
Member Author

Just for visibility, I found another background image related PR WordPress/wordpress-develop#6836 (comment) that is responsible for regression.

@ramonjd
Copy link
Member

ramonjd commented Oct 15, 2024

Just for visibility, I found another background image related PR WordPress/wordpress-develop#6836 (comment) that is responsible for regression.

Following the set up in the docs, I'm having trouble replicating the figures you quoted here locally.

Comparing with and then without those changes, my benchmark stats are unmoved or are neglible in TT5, TT4 and TT3.

I know I'm probably doing something wrong though.

E.g., TT3,

Metric Percentile WP trunk WP trunk without the changes in #6836 Diff %
wp-before-template p10 16.58 17.21 3.80%
wp-before-template p25 16.78 17.38 3.58%
wp-before-template p50 (median) 17.28 17.79 2.95%
wp-before-template p75 18.04 18.21 0.94%
wp-before-template p90 18.47 18.78 1.68%
wp-template p10 12.71 12.79 0.63%
wp-template p25 12.8 12.84 0.31%
wp-template p50 (median) 12.88 12.95 0.54%
wp-template p75 13.01 13.14 1.00%
wp-template p90 13.41 13.7 2.16%
wp-total p10 29.32 30.03 2.42%
wp-total p25 29.66 30.32 2.23%
wp-total p50 (median) 30.2 30.87 2.22%
wp-total p75 31 31.22 0.71%
wp-total p90 31.81 32.6 2.48%

That the "without change" measurement is higher for me I cannot explain, but I ran it multiple times and it swings +- by 0.5 on each side.

@mukeshpanchal27
Copy link
Member Author

mukeshpanchal27 commented Oct 15, 2024

Again ran the benchmark for TT4

Server-Timing benchmarks

Mertic Beta 3 WP trunk without the changes in #6836 Diff % Diff abs.
Response Time (p10) 65.43 66.71 1.96% 1.28
Response Time (p25) 65.72 67.1 2.10% 1.38
Response Time (p50) 66.6 68.14 2.31% 1.54
Response Time (p75) 67.65 69.45 2.66% 1.80
Response Time (p90) 69.65 72.08 3.49% 2.43
wp-before-template (p10) 29.91 30.51 2.01% 0.60
wp-before-template (p25) 30.19 30.77 1.92% 0.58
wp-before-template (p50) 30.48 31.18 2.30% 0.70
wp-before-template (p75) 30.82 31.89 3.47% 1.07
wp-before-template (p90) 31.75 32.51 2.39% 0.76
wp-template (p10) 33.38 34.07 2.07% 0.69
wp-template (p25) 33.56 34.29 2.18% 0.73
wp-template (p50) 33.93 34.69 2.24% 0.76
wp-template (p75) 34.84 35.68 2.41% 0.84
wp-template (p90) 35.42 37.51 5.90% 2.09
wp-total (p10) 63.5 64.74 1.95% 1.24
wp-total (p25) 63.81 65.14 2.08% 1.33
wp-total (p50) 64.72 66.08 2.10% 1.36
wp-total (p75) 65.52 67.27 2.67% 1.75
wp-total (p90) 67.34 69.87 3.76% 2.53

@ramonjd
Copy link
Member

ramonjd commented Oct 16, 2024

Again ran the benchmark for TT4

Thanks for running it again. I think I still need some clarification on how to interpret these results.

Based on the tables, the server load time increases when the changeset in question is reverted.

Unless a higher number is better 😅 ? My assumption is that a lower server load time number is preferable.

Anyway, I'll keep testing and report back if I find something significant.

It's probably worth looking into caching the output of the method as well.

Thank you for your help!

@ramonjd
Copy link
Member

ramonjd commented Oct 16, 2024

I've run tests both locally and on CI and cannot find any conclusive evidence that WordPress/wordpress-develop#6836 causes a statistically significant regression in performance.

In fact, I'm more confounded than anything. Some reports suggest that trunk is faster than the reverted branch, other times the needle falls in the other direction by between 1-4ms.

Example from a complete revert on CI ("After" is the revert):

Image

Locally the benchmark tests reveal nothing significant for me as mentioned above. I've been running all variations for a couple of hours and am yet to find any reliable results.

Furthermore I added some relative paths to TT4 and tried caching the results of WP_Theme_JSON_Resolver::resolve_theme_file_uris() and WP_Theme_JSON_Resolver::get_resolved_theme_uris(), both of which yielded inconclusive results.

All that said, I do believe we need a caching strategy for these resolved URIs however.

I think perhaps caching them in WP_Theme_JSON_Resolver as is done with theme_json might work, using get_stylesheet() or get_stylesheet_directory() as a key. I'll add it to my list of things to look into.

Edit: This will improve things in general perhaps:

@aaronrobertshaw
Copy link

I've run tests both locally and on CI and cannot find any conclusive evidence that WordPress/wordpress-develop#6836 causes a statistically significant regression in performance.

I've also followed the steps outlined in the docs, using a completely fresh site locally for testing. Every test run appears to yield inconsistent results for me.

This was the latest set of results for me, running a batch of 10000 requests, with PR being a branch reverting the changes in question:

In fact, I'm more confounded than anything.

💯 in the same boat.

I've been running all variations for a couple of hours and am yet to find any reliable results.

+1

@mukeshpanchal27
Copy link
Member Author

mukeshpanchal27 commented Oct 16, 2024

Have you both checked for TT4?

@ramonjd
Copy link
Member

ramonjd commented Oct 16, 2024

Have you both checked for TT4?

Yes.

@joemcgill
Copy link
Member

Just committed r59256, which shows pretty good improvements for TT4

@ramonjd
Copy link
Member

ramonjd commented Oct 20, 2024

@joemcgill does r59256 require backporting to Gutenberg? Happy to take care of it if you don't have time.

@aaronrobertshaw
Copy link

does r59256 require backporting to Gutenberg?

I believe it does. I was also planning to help out there if needed 👍

Core's action calling wp_enqueue_global_styles is replaced in Gutenberg with its version. Which ultimately means gutenberg_add_global_styles_for_blocks is used instead of wp_add_global_styles_for_blocks.

@ramonjd
Copy link
Member

ramonjd commented Oct 21, 2024

I think perhaps caching them in WP_Theme_JSON_Resolver as is done with theme_json might work, using get_stylesheet() or get_stylesheet_directory() as a key.

Took a naive stab at this: WordPress/gutenberg#66155

@mukeshpanchal27
Copy link
Member Author

mukeshpanchal27 commented Oct 23, 2024

WP 6.7 RC 1 Performance Benchmark Report:

With object caching disabled,

  • Classic themes (TT1) web vitals and server-timing (~2% improvement) remains relatively stable.
  • Block themes (TT3)
    • LCP shows a ~3% improvement (~3ms)
    • TTFB shows a ~2% improvement (~1ms)
    • LCP-TTFB shows a ~5% improvement (~2ms)
    • wp-template shows a ~4% improvement (1ms)
    • wp-total shows a ~2% improvement (1ms)
  • Block themes (TT4)
    • LCP shows a ~5% regression (~8ms)
    • TTFB shows a ~8% regression (~6ms)
    • wp-template shows a ~10% regression (6ms)
    • wp-total shows a ~8% regression (6ms)

Here are the results of the test runs:

TT1: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11472667306
TT3: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11472757880
TT4: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11477476622

TT1 (twentytwentyone) benchmarks

Web Vitals

Metric WP 6.6.2 WP 6.7 RC 1 Diff % Diff abs.
FCP 100.1 99.6 -0.50% -0.50
LCP 100.1 99.6 -0.50% -0.50
TTFB 46.1 45.55 -1.19% -0.55
LCP-TTFB 53.8 53.95 0.28% 0.15

Server-Timing

Metric WP 6.6.2 WP 6.7 RC 1 Diff % Diff abs.
Response Time 39.54 38.7 -2.12% -0.84
wp-before-template 11.35 11.26 -0.79% -0.09
wp-template 27.19 26.48 -2.61% -0.71
wp-total 38.61 37.77 -2.18% -0.84

TT3 (twentytwentythree) benchmarks

Web Vitals

Metric WP 6.6.2 WP 6.7 RC 1 Diff % Diff abs.
FCP 97 93.4 -3.71% -3.60
LCP 97 93.4 -3.71% -3.60
TTFB 53.85 52.65 -2.23% -1.20
LCP-TTFB 42.3 40.2 -4.96% -2.10

Server-Timing

Metric WP 6.6.2 WP 6.7 RC 1 Diff % Diff abs.
Response Time 46.53 45.26 -2.73% -1.27
wp-before-template 15.4 15.4 0.00% 0.00
wp-template 30.14 28.82 -4.38% -1.32
wp-total 45.59 44.32 -2.79% -1.27

TT4 (twentytwentyfour) benchmarks

Web Vitals

Metric WP 6.6.2 WP 6.7 RC 1 Diff % Diff abs.
FCP 155.9 164.15 5.29% 8.25
LCP 155.9 164.25 5.36% 8.35
TTFB 81.25 87.8 8.06% 6.55
LCP-TTFB 73.65 76.25 3.53% 2.60

Server-Timing

Metric WP 6.6.2 WP 6.7 RC 1 Diff % Diff abs.
Response Time 73.54 79.48 8.08% 5.94
wp-before-template 16.2 16.28 0.49% 0.08
wp-template 55.95 61.95 10.72% 6.00
wp-total 72.28 78.24 8.25% 5.96

@mukeshpanchal27
Copy link
Member Author

WP 6.7 RC 2 Performance Benchmark Report:

With object caching disabled,

  • Classic themes (TT1) web vitals and server-timing (~2% improvement) remains relatively stable.
  • Block themes (TT3)
    • LCP shows a ~1% improvement (~1ms)
    • TTFB shows a ~1% improvement (~1ms)
    • LCP-TTFB shows a ~1% improvement (~0.5ms)
    • wp-template shows a ~4% improvement (~1ms)
    • wp-total shows a ~2% improvement (~1ms)
  • Block themes (TT4)
    • LCP shows a ~1% regression (~1.5ms)
    • TTFB shows a ~1.5% regression (~1ms)
    • wp-template shows a ~1% regression (0.5ms)
    • wp-total shows a ~0.5% regression (~0.5ms)

Here are the results of the test runs:

TT1: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11579114106
TT3: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11579298956
TT4: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11579469822

TT1 (twentytwentyone) benchmarks

Web Vitals

Metric Before After Diff % Diff abs.
FCP 100.2 99.65 -0.55% -0.55
LCP 100.2 99.65 -0.55% -0.55
TTFB 46.35 45.8 -1.19% -0.55
LCP-TTFB 53.75 54 0.47% 0.25

Server-Timing

Metric Before After Diff % Diff abs.
Response Time 40.55 40.16 -0.96% -0.39
wp-before-template 11.86 12.04 1.52% 0.18
wp-template 27.63 27.07 -2.03% -0.56
wp-total 39.53 39.08 -1.14% -0.45

TT3 (twentytwentythree) benchmarks

Web Vitals

Metric Before After Diff % Diff abs.
FCP 98.6 96.9 -1.72% -1.70
LCP 98.6 96.9 -1.72% -1.70
TTFB 55 54.2 -1.45% -0.80
LCP-TTFB 42.85 42.4 -1.05% -0.45

Server-Timing

Metric Before After Diff % Diff abs.
Response Time 48.23 46.9 -2.76% -1.33
wp-before-template 16.25 16.22 -0.18% -0.03
wp-template 30.82 29.6 -3.96% -1.22
wp-total 47.1 45.82 -2.72% -1.28

TT4 (twentytwentyfour) benchmarks

Web Vitals

Metric Before After Diff % Diff abs.
FCP 155.95 157.5 0.99% 1.55
LCP 155.95 157.5 0.99% 1.55
TTFB 80.85 82.1 1.55% 1.25
LCP-TTFB 74.5 74.3 -0.27% -0.20

Server-Timing

Metric Before After Diff % Diff abs.
Response Time 72.95 73.39 0.60% 0.44
wp-before-template 15.79 15.78 -0.06% -0.01
wp-template 55.94 56.51 1.02% 0.57
wp-total 71.82 72.28 0.64% 0.46

RC 2 shows improvement compared to RC 1 in block theme.

@mukeshpanchal27
Copy link
Member Author

I ran local benchmarks for TT4 on versions 6.6.2 and 6.7 RC2 and saw similar results. Core didn’t introduce any performance-related changes between RC1 and RC2, though the Editor packages were updated in the meantime.

Interestingly, the Code Vitals analysis didn’t show any improvement 🤔

@mukeshpanchal27
Copy link
Member Author

mukeshpanchal27 commented Nov 6, 2024

WP 6.7 RC3 Performance Benchmark Report:

With object caching disabled,

  • Classic themes (TT1)
    • TTFB shows a ~2% improvement (~1ms)
    • wp-template shows a ~2% improvement (~0.5ms)
  • Block themes (TT3)
    • LCP shows a ~3% improvement (~3ms)
    • TTFB shows a ~2% improvement (~1ms)
    • LCP-TTFB shows a ~3.5% improvement (~1.5ms)
    • wp-before-template shows a ~2% regression (~0.3ms)
    • wp-template shows a ~3% improvement (~1ms)
    • wp-total shows a ~1.5% improvement (~0.7ms)
  • Block themes (TT4)
    • LCP shows a ~2.5% regression (~4ms)
    • TTFB shows a ~2.7% regression (~2ms)
    • LCP-TTFB shows a ~1.3% regression (~1ms)
    • wp-template shows a ~3% regression (1.8ms)
    • wp-total shows a ~2.7% regression (~2ms)

Here are the results of the test runs:

TT1: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11696995231
TT3: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11697098693
TT4: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11697201653

TT1 (twentytwentyone) benchmarks

Web Vitals

Metric Before After Diff % Diff abs.
FCP 100.6 100.55 -0.05% -0.05
LCP 100.6 100.55 -0.05% -0.05
TTFB 46.4 45.3 -2.37% -1.10
LCP-TTFB 53.9 54.75 1.58% 0.85

Server-Timing

Metric Before After Diff % Diff abs.
Response Time 38.17 37.85 -0.84% -0.32
wp-before-template 10.7 10.91 1.96% 0.21
wp-template 26.68 26.19 -1.84% -0.49
wp-total 37.42 37.1 -0.86% -0.32

TT3 (twentytwentythree) benchmarks

Web Vitals

Metric Before After Diff % Diff abs.
FCP 96.2 93.15 -3.17% -3.05
LCP 96.2 93.15 -3.17% -3.05
TTFB 53.5 52.5 -1.87% -1.00
LCP-TTFB 41.7 40.25 -3.48% -1.45

Server-Timing

Metric Before After Diff % Diff abs.
Response Time 45.83 45.04 -1.72% -0.79
wp-before-template 15.1 15.39 1.92% 0.29
wp-template 29.66 28.69 -3.27% -0.97
wp-total 44.86 44.11 -1.67% -0.75

TT4 (twentytwentyfour) benchmarks

Web Vitals

Metric Before After Diff % Diff abs.
FCP 155.75 159.9 2.66% 4.15
LCP 155.75 159.9 2.66% 4.15
TTFB 81.2 83.4 2.71% 2.20
LCP-TTFB 74.55 75.55 1.34% 1.00

Server-Timing

Metric Before After Diff % Diff abs.
Response Time 75 77.06 2.75% 2.06
wp-before-template 16.49 16.7 1.27% 0.21
wp-template 57.28 59.09 3.16% 1.81
wp-total 73.75 75.79 2.77% 2.04

RC3 shows slight regression compared to RC2 in block theme.

@mukeshpanchal27
Copy link
Member Author

WP 6.7 Performance Benchmark Report:

With object caching disabled,

  • Classic themes (TT1)
    • LCP and TTFB shows slight improvement
    • wp-template and wp-total shows a ~3% improvement (~1ms)
  • Block themes (TT3)
    • LCP shows a ~0.6% improvement (~0.6ms)
    • TTFB shows a ~1.5% improvement (~0.8ms)
    • LCP-TTFB shows slight regression
    • wp-before-template shows a ~0.3% improvement (~0.06ms)
    • wp-template shows a ~3.6% improvement (~1.1ms)
    • wp-total shows a ~1.5% improvement (~0.7ms)
  • Block themes (TT4)
    • LCP shows a ~3.6% regression (~5.7ms)
    • TTFB shows a ~2.8% regression (~2.3ms)
    • LCP-TTFB shows a ~3.7% regression (~2.7ms)
    • wp-template and wp-total shows slight improvement

Here are the results of the test runs:

TT1: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11931170428
TT3: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11931351109
TT4: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11931556138

TT1 (twentytwentyone) benchmarks

Web Vitals

Metric Before After Diff % Diff abs.
FCP 100.5 100.3 -0.20% -0.20
LCP 100.5 100.3 -0.20% -0.20
TTFB 46.7 46.5 -0.43% -0.20
LCP-TTFB 53.95 54.1 0.28% 0.15

Server-Timing

Metric Before After Diff % Diff abs.
Response Time 40.58 39.17 -3.47% -1.41
wp-before-template 11.75 11.34 -3.49% -0.41
wp-template 27.86 26.92 -3.37% -0.94
wp-total 39.56 38.32 -3.13% -1.24

TT3 (twentytwentythree) benchmarks

Web Vitals

Metric Before After Diff % Diff abs.
FCP 98.7 98.1 -0.61% -0.60
LCP 98.7 98.1 -0.61% -0.60
TTFB 55.5 54.65 -1.53% -0.85
LCP-TTFB 42.8 42.9 0.23% 0.10

Server-Timing

Metric Before After Diff % Diff abs.
Response Time 48.99 47.97 -2.08% -1.02
wp-before-template 16.58 16.52 -0.36% -0.06
wp-template 31.32 30.19 -3.61% -1.13
wp-total 47.8 46.8 -2.09% -1.00

TT4 (twentytwentyfour) benchmarks

Web Vitals

Metric Before After Diff % Diff abs.
FCP 154.25 159.9 3.66% 5.65
LCP 154.35 160.05 3.69% 5.70
TTFB 81.2 83.5 2.83% 2.30
LCP-TTFB 73 75.7 3.70% 2.70

Server-Timing

Metric Before After Diff % Diff abs.
Response Time 74 73.35 -0.88% -0.65
wp-before-template 16.22 15.89 -2.03% -0.33
wp-template 56.47 56.34 -0.23% -0.13
wp-total 72.68 72.22 -0.63% -0.46

@mukeshpanchal27
Copy link
Member Author

Closing this as no further benchmarking is needed for WP 6.7. In #1681, I’m working on the Core post for 6.7 Performance Improvements.

@github-project-automation github-project-automation bot moved this from In Progress 🚧 to Done 😃 in WP Performance 2024 Dec 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Type] Epic A high-level project / epic that will encompass several sub-issues WP Core Work relates to inclusion in WP Core only
Projects
Status: Done 😃
Development

No branches or pull requests

5 participants