chore(ci/regressions): test respec-w3c differences across branches

[sidvishnoi]

This workflow finds the impact of changes in ReSpec on real-world specs. It's run on manual requests, to avoid wasting resources. It works as follows:

1. Build the W3C profile, and cache it (actions/upload-artifact) .
2. For each matrix.source:
   1. Process ReSpec document using the ReSpec script in the document, and write before.html.
   2. Process ReSpec document using the ReSpec script generated in Step 1 (thanks to https://github.com/w3c/respec/pull/2957), and write after.html.
   3. Compare before.html and after.html. If there is a difference, exit with error code to draw manual attention.

A sample workflow run is available at https://github.com/w3c/respec/actions/runs/704622451 (corresponding to w3c@4aff5e7)

[marcoscaceres]

so olde?

[sidvishnoi]

Using v12 everywhere else and I think spec-generator is on v12 also. So, should move all of them together. Breaking change?
Node v12 goes into EOL 2022-04-30 FWIW.

[marcoscaceres]

We should be tied to LTS instead (everywhere)... 12 seems way too limiting.

[sidvishnoi]

Started https://github.com/w3c/respec/pull/3450

[marcoscaceres]

wonder if this versioned software will become a problem to maintain? Probably not... just a thought.

[sidvishnoi]

I was thinking of converting it into a separate GitHub action, which I can update as needed.
cargo install took ~4min, so had to use a pre-compiled binary.

[marcoscaceres]

Ok, but time taken on Ci shouldn't be an issue right? That's the "cost of doing business".

[sidvishnoi]

No issues now as I've moved this into a separate GitHub action. It takes 1s to install rather than 4min.

[marcoscaceres]

Your meme game is strong, Padawan.

[marcoscaceres]

Few little questions mostly... but looks good.

I'm a bit worried that we are following a moving target though... maybe we want a fake "kitchen sink" spec that tests a whole bunch of things... that might allow us to catch more errors in practice.

[sidvishnoi]

I'm a bit worried that we are following a moving target though

Do want to clarify that we'll run regression test on PR branches manually, when we feel like something in the spec might break.
Also, as tests will run on live spec using two different ReSpec versions, at almost same time, it's not much of a moving target.

that might allow us to catch more errors in practice

Might be better to test the actual practice instead of trying to copy practice in kitchen sink test?

[marcoscaceres]

Might be better to test the actual practice instead of trying to copy practice in kitchen sink test?

Tradeoffs... happy to try this out for now and see how we go.