Add more options and optimization for our tests #1

AfzalivE · 2021-02-24T19:10:29Z

Added options for disabling spellcheck, autofill, enabling hw keyboard (and disabling soft keyboard), changing longpress timeout
Added option to modify the number of cores used by the emulator (because GitHub macOS VM has 3 cores, the emulator defaults to 1 core, which is not stable)
Added option to enable reading and saving logcat output

Note: The lib folder is generated by npm. A release branch called release/v2-doist has been created from this branch to use these changes until a tag is ready.

Once this is merged to main, I will tag a version so we can use that in our repos. Since this is our fork, we can control the tags (correct me if I'm wrong). @goncalossilva and @Doist/android, would appreciate your thoughts on this part as well.

Given the amount of new options here, I wouldn't be surprised if some of these changes get rejected upstream.

Add input-validator and tests

goncalossilva

Once this is merged to main, I will tag a version so we can use that in our repos. Since this is our fork, we can control the tags (correct me if I'm wrong). @goncalossilva and @Doist/android, would appreciate your thoughts on this part as well.

Yes, we can use tags. 👍

goncalossilva · 2021-02-25T02:09:59Z

src/emulator-manager.ts

@@ -27,6 +27,9 @@ export async function launchEmulator(
    `sh -c \\"echo no | avdmanager create avd --force -n "${avdName}" --abi '${target}/${arch}' --package 'system-images;android-${apiLevel};${target};${arch}' ${profileOption} ${sdcardPathOrSizeOption}"`
  );

+  // set number of cores to 2 (instead of 1 because there are 3 cores on the Github-hoster macOS VM)
+  await exec.exec(`sh -c \\"printf 'hw.cpu.ncore=2\n' >> ~/.android/avd/"${avdName}".avd"/config.ini`);


Could this be configurable, too, with some kind of default (i.e., do nothing)? I feel this is the only change that doesn't generalize as is to submit upstream.

Had a discussion with the maintainer about this change once I figured this was the difference between our Azure pipeline and the Emulator on GitHub. I will generalize it though, to make it easier to modify this number.

ReactiveCircus#126

I was going to suggest exactly this. 👏

Done! Added a cores option now :)

Great investigation @AfzalivE. 🙌

pedrodanielcsantos

Only left one comment, related with what Gonçalo already mentioned above! 👏

Other than that, looks great! 🚀

pedrodanielcsantos · 2021-02-25T13:32:56Z

lib/emulator-manager.js

    return __awaiter(this, void 0, void 0, function* () {
        // create a new AVD
        const profileOption = profile.trim() !== '' ? `--device '${profile}'` : '';
        const sdcardPathOrSizeOption = sdcardPathOrSize.trim() !== '' ? `--sdcard '${sdcardPathOrSize}'` : '';
        console.log(`Creating AVD.`);
        yield exec.exec(`sh -c \\"echo no | avdmanager create avd --force -n "${avdName}" --abi '${target}/${arch}' --package 'system-images;android-${apiLevel};${target};${arch}' ${profileOption} ${sdcardPathOrSizeOption}"`);
+        // set number of cores to 2 (instead of 1 because there are 3 cores on the Github-hoster macOS VM)
+        yield exec.exec(`sh -c \\"printf 'hw.cpu.ncore=2\n' >> ~/.android/avd/"${avdName}".avd"/config.ini`);


Same as below: https://github.com/Doist/android-emulator-runner/pull/1/files#r582439112

Actually, I saw this from the PR description _ The lib folder is generated by npm._ after posting the comment so it's probably not applicable. 😅

It's in private beta and we don't have access to it

pedrodanielcsantos

Looks good @AfzalivE ! Great that everything's parameterizable now 🚀 LGTM!

AfzalivE · 2021-02-25T19:50:35Z

Thank you both for the review!!

goncalossilva · 2021-02-27T00:31:05Z

A question about the OS we're using: macOS is 10x more expensive to run than Linux. Rasto did comment that macOS images were faster, but is it significantly different?

AfzalivE · 2021-02-27T01:10:49Z

A question about the OS we're using: macOS is 10x more expensive to run than Linux. Rasto did comment that macOS images were faster, but is it significantly different?

Unfortunately, the Linux and Windows VMs that GitHub provides don't have hardware virtualization/acceleration support and they're not very reliable above API level 16. Last time I tried Ubuntu with API 29, it didn't boot in 2 hours. Trying again now considering all the other changes I've made since. Edit: Boot completed but then it got stuck.

Here's some more background into virtualization support for Linux and Windows:
actions/runner-images#114
actions/runner-images#183
ReactiveCircus#89
ReactiveCircus#46

However, since macOS is 10x more expensive. I ran the current Azure pipeline stats to see how much it would cost us, here's the spreadsheet. Spoiler: It's not cheap when the jobs start timing out a lot but this past week, average day had less than 3 hours of jobs. At 4 hours per day, 30 days would be 120 minutes. On a Mac VM: 1200 minutes just for Android runs. Plus if we use matrix to run tests faster (half the time compared to Azure): it's ~ 1.5 billable hours per build.

It is possible that this change on GitHub's roadmap for Q1 2021 will bring improvements to the Linux/Windows machines that would allow reliably running the emulator on those OS.

AfzalivE · 2021-02-27T08:10:47Z

I didn't look at this in detail earlier but Firebase Test Lab might actually be much cheaper for us. GitHub's macOS VM comes out to $4.8/hr ($0.08/min) and Firebase Test Lab is $1/hour ($0.0166/min) on a virtual device. Plus, they offer 60 minutes per day of virtual device for free.

Sharding tests on Firebase would also end up faster and might not incur much more cost than not sharding since compilation would happen on GitHub Actions (ubuntu) so only tests will run on Firebase.

Although, I'm not sure how the performance is. I see on Twist that Firebase Test Lab was considered multiple times before. Do you know why we didn't continue pursuing it?

goncalossilva · 2021-03-01T19:19:49Z

Great analysis, @AfzalivE! It's a shame that GitHub is approaching this so strangely....

My take is that unless we are talking about different magnitudes of cost, the most valuable aspect is developer time.

The more we can shard the tests, the better. The faster they run, the better. So if macOS on GitHub is 2x faster, that is the better choice (even if more expensive). Plus, we can reuse our know-how on GitHub Actions across the rest of Doist.

But if Firebase Test Lab is even faster (and allows us to shard more), then that's the better option. Let's use it. 😄

Personally, I would love to see feedback loops brought down to 15 minutes at most.

AfzalivE · 2021-03-01T20:09:00Z

My take is that unless we are talking about different magnitudes of cost, the most valuable aspect is developer time. The more we can shard the tests, the better. The faster they run, the better. So if macOS on GitHub is 2x faster, that is the better choice (even if more expensive). Plus, we can reuse our know-how on GitHub Actions across the rest of Doist.

In that case we're good! This is true and so far with 4 shards, 2x faster. 6 shards wasn't much of a difference. I didn't go so far as 8 shards though 😅 .

But if Firebase Test Lab is even faster (and allows us to shard more), then that's the better option. Let's use it. 😄

Not sure about this. I didn't try with a physical device on Firebase but I did execute 1 run and it failed in 15 minutes with this error:
The test timed out. The test ran longer than its maximum allowed duration, and was stopped.

I think it would be valuable explore this further in the future to speed this up even more. Especially if this would allow us to cache the UI tests as well in the gradle cache in the future. That's around 5 minutes of compile time.

There's also a project called Flank that Airbnb uses that would help in sharding on Firebase Test Lab.

AfzalivE added 4 commits February 24, 2021 12:51

Add option to disable spellcheck

90288b0

Add input-validator and tests

Add option to disable autofill service

640f06b

Add option to change long-press timeout

393c3c5

Add option to enable hw keyboard and disable soft keyboard

4b6ca99

AfzalivE requested a review from pedrodanielcsantos February 24, 2021 19:14

goncalossilva approved these changes Feb 25, 2021

View reviewed changes

pedrodanielcsantos approved these changes Feb 25, 2021

View reviewed changes

AfzalivE added 2 commits February 25, 2021 12:22

Add option to modify the number of cores

87a8f95

Add option to enable reading and saving logcat output

e980520

AfzalivE force-pushed the doist_updates branch from b0db911 to e980520 Compare February 25, 2021 17:24

Remove macos-11 job

4d9d722

It's in private beta and we don't have access to it

AfzalivE requested a review from pedrodanielcsantos February 25, 2021 19:20

pedrodanielcsantos approved these changes Feb 25, 2021

View reviewed changes

AfzalivE merged commit 1160537 into main Feb 25, 2021

AfzalivE deleted the doist_updates branch February 25, 2021 19:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more options and optimization for our tests #1

Add more options and optimization for our tests #1

AfzalivE commented Feb 24, 2021 •

edited

Loading

goncalossilva left a comment

goncalossilva Feb 25, 2021

AfzalivE Feb 25, 2021 •

edited

Loading

pedrodanielcsantos Feb 25, 2021

AfzalivE Feb 25, 2021 •

edited

Loading

goncalossilva Feb 27, 2021

pedrodanielcsantos left a comment

pedrodanielcsantos Feb 25, 2021

pedrodanielcsantos Feb 25, 2021

pedrodanielcsantos left a comment

AfzalivE commented Feb 25, 2021

goncalossilva commented Feb 27, 2021

AfzalivE commented Feb 27, 2021 •

edited

Loading

AfzalivE commented Feb 27, 2021 •

edited

Loading

goncalossilva commented Mar 1, 2021

AfzalivE commented Mar 1, 2021 •

edited

Loading

Add more options and optimization for our tests #1

Add more options and optimization for our tests #1

Conversation

AfzalivE commented Feb 24, 2021 • edited Loading

goncalossilva left a comment

Choose a reason for hiding this comment

goncalossilva Feb 25, 2021

Choose a reason for hiding this comment

AfzalivE Feb 25, 2021 • edited Loading

Choose a reason for hiding this comment

pedrodanielcsantos Feb 25, 2021

Choose a reason for hiding this comment

AfzalivE Feb 25, 2021 • edited Loading

Choose a reason for hiding this comment

goncalossilva Feb 27, 2021

Choose a reason for hiding this comment

pedrodanielcsantos left a comment

Choose a reason for hiding this comment

pedrodanielcsantos Feb 25, 2021

Choose a reason for hiding this comment

pedrodanielcsantos Feb 25, 2021

Choose a reason for hiding this comment

pedrodanielcsantos left a comment

Choose a reason for hiding this comment

AfzalivE commented Feb 25, 2021

goncalossilva commented Feb 27, 2021

AfzalivE commented Feb 27, 2021 • edited Loading

AfzalivE commented Feb 27, 2021 • edited Loading

goncalossilva commented Mar 1, 2021

AfzalivE commented Mar 1, 2021 • edited Loading

AfzalivE commented Feb 24, 2021 •

edited

Loading

AfzalivE Feb 25, 2021 •

edited

Loading

AfzalivE Feb 25, 2021 •

edited

Loading

AfzalivE commented Feb 27, 2021 •

edited

Loading

AfzalivE commented Feb 27, 2021 •

edited

Loading

AfzalivE commented Mar 1, 2021 •

edited

Loading