Reduce time for workflow/Build

[NiKiZe]

• Use cache for .pio path, this keeps buildtime down when unchanged.
• Use dynamic matrix to run jobs in parallel
  • These jobs are generated from location of platformio.ini files
  • All jobs runs on their own, some duplication of setup
  • Status is reported for each job
  • One final job that requires all other jobs to succeed

[crankyoldgit]

Good luck. 🤞

[NiKiZe]

Normal runtime was around 30 minutes, we are down to 2-3 minutes as long as there is no code change.
Might be good to test this with code change to see how it responds, especially so that it don't fail. but also check that the cache is updated.

[NiKiZe]

Tested in #1711 Total runtime of Builds work was 15m 35s, the longest was IRrecvDumpV3
New build cache was not used.
And messages like Unable to reserve cache with key Linux-, another job may be creating this cache. are seen when they should be created.

I tried to look into having a common setup step and then run jobs, but did not find anything really promising.

[NiKiZe]

After modifying cache key for pio build data and create one per matrix item it now builds fine and also reuses cached for later started builds.

[crankyoldgit]

I'd be lying if I said I understood it fully. But the results are awesome.

Approved!

[NiKiZe]

Might need to revisit this to get new caches created correctly. Now Sony was changed and as other files change as well we might lose cache hits.

[crankyoldgit]

Just for the record, I completely have no clue what you're saying. :-P Well, I understand what you're saying, but I don't know what it technically means. i.e. I have no clue how any of your science caching magic works. I say you're a witch. Witch! Burn him! 🔥

[NiKiZe]

This part extracts the .pio directory from a tar archive before build so that cached builds can be used.
As times goes on more and more of the build cache will become out of date.
Since the key already matches perfectly new archives are not created.

Nothing will (probably) break, but builds will get slower again.
If something breaks just pull out that cache from the yml.

I do intend to keep an eye on this and create some kind of fix.

magic is just technology we don't understand

I don't know either, I just modified the existing yml that you created. - now which is the witch? :P

[crankyoldgit]

I do intend to keep an eye on this and create some kind of fix.

So if I understand you correctly, we are referencing a point-in-time object cache, which will grow outdated with every change.

Can we automate creating the cache at all? Say, a weekly or daily etc tasks with what ever the master branch is at?

[NiKiZe]

Yes, the key is only a string.
We could use a key with the commit hash, but that would generate new caches on each commit.
But adding year and week number is probably a really good balance between not creating it every time causing many files in cache and not having it to outdated.

Right now we don't have much data, my thinking is to do the revisit when we see this causing an impact.
In worst case we should see build times of 12-15 minutes instead of the 2-3 minutes we have now. (but that is still better than the 30 minutes we had before)

[NiKiZe]

There is key: and restore-keys
if key matches it is used and no new cache is created.
On non key match, restore-keys is searched for alternative, and a new cache is created.

by adding more conditions to the main key, new caches will be created.
Max 10GB of caches in total will be kept.

https://docs.github.com/en/actions/advanced-guides/caching-dependencies-to-speed-up-workflows#input-parameters-for-the-cache-action

Maybe better example: https://github.com/actions/cache#creating-a-cache-key

[crankyoldgit]

sounds good to me. I'll leave it with you. ;-)