Ensure Tutor works on ARM64 via an easy-to-use plugin

[kdmccormick]

The original title was Ensure Tutor works out-of-the-box on ARM64. We reduced the scope to "via an easy-to-use plugin" for now. We'll follow up with making this an official part of the base installation in : #81

Context

There are a few issues circulating around Open edX development on machines under the ARM64 architecture, notably the new M1 Macs. While many/all of these issues are not caused by Tutor itself, they affect the Tutor dev experience nonetheless. To make it maximally likely that 2U (largely Mac users) fully adopts Tutor, it's important that the tools feels like an "out of the box" experience as much as possible. While ideally this will involve merging fixes to upstream repos, it may also involve workarounds or doc additions in Tutor core or plugins.

Open issues

1. Tutor's openedx image is built for x86-64
   • Raised here: Support for ARM64 overhangio/tutor#510
   • Workaround 1: Use platform: to enable emulation.
     • Mentioned here: Support for ARM64 overhangio/tutor#510
     • Outstanding issue: Emulation incurs a significant performance penalty.
   • Workaround 2: Rebuild the image locally.
     • Enabled here: Fix arm64 image build - open-release/lilac.3 overhangio/tutor#543
     • Documented here: Tutorial: how to run Tutor on ARM-based systems overhangio/tutor#549
     • Outstanding issue: Rebuilding image requires extra steps & extra time.
2. MySQL 5.7 does not provide ARM64 images.
   • Raised here: Support for ARM64 overhangio/tutor#510
   • Workaround 1: Use platform: to enable emulation.
     • Mentioned here: Support for ARM64 overhangio/tutor#510
     • Outstanding issue: Emulation incurs a significant (~2x) performance penalty.
   • Workaround 2: Use MariaDB instead.
     • Documented here: Tutorial: how to run Tutor on ARM-based systems overhangio/tutor#549
     • Outstanding issue: Switching MariaDB requires extra step & differs from production.
   • Workaround 3: Use to MySQL 8 instead.
     • edx-platform migrations currently all pass on MySQL 8.
     • Current plan for Open edX is MySQL 8 compatibility by Nutmeg, with MySQL 5.7 dropped by Olive.
     • Outstanding issue: Using MySQL 8 requires extra step & differs from production.
3. frontend-app-learning does not build on ARM64
   • Outstanding issue: tutor-mfe is broken on ARM64.
   • Raised here: Support for ARM64 overhangio/tutor-mfe#31
   • Potential fix here: build: move pact dependency to devDependencies openedx/frontend-app-learning#823

Acceptance

Ensure that it's nearly as easy (1-2 extras steps max) to set up Tutor on an ARM64 machine as it as on x86-64 machine in all of these configurations:

• Stable
• Nightly, using master
• Nightly, using a custom bind-mounted branch

and using all of these plugins:

• tutor-mfe
• tutor-discovery
• tutor-forum

[ormsbee]

@kdmccormick: So there are at least a few possible pieces of work here, right?

1. Short term (optional): Make it so that Tutor detects the platform's architecture so that it chooses to use MariaDB automatically instead of requiring people set DOCKER_IMAGE_MYSQL=mariadb:10.4.
2. Medium term (by Olive, maybe earlier, but after edx-platform code fixes are in): Change the MySQL version to 8.0, which has proper images for ARM64.
3. Short term: The tutor-mfe ARM64 issue.

[kdmccormick]

Thanks @ormsbee ; I factored your comment into the ticket description.

[kdmccormick]

@regisb Please take a look at the Context, Open Issues, and Acceptance criteria above, and let me know:

• anything I missed in my synopsis
• your thoughts on the goal I listed
• whether there are any paths we could take to make progress on this

Some ideas I've heard so far:

• Provide an ARM64-ready openedx image in the overhangio DockerHub repository.
• Detect whether a user is on ARM64, and if so, switch to MariaDB (edit: or MySQL 8!) by default.

[ormsbee]

@kdmccormick: As a heads up that per @jmbowman, the plan is for edx-platform to be MySQL 8.0 compatible in time for Nutmeg, though it will likely also support 5.7 during that release, with 5.7 support dropped for Olive.

[jmbowman]

edx-platform now has a passing MySQL 8 migrations check: openedx/edx-platform#29784 . The rest of the services haven't been thoroughly checked yet, but I think that resolves all the MySQL 8 compatibility issues that have been identified so far.

[kdmccormick]

That is good to know! It sounds to me like the best workaround for the MySQL issue is to have ARM64 users upgrade early to MySQL 8.

[regisb]

There are two items which are especially difficult to me:

1. Providing arm64 Docker images: to do that, I would have to build the linux and arm64 images synchronously, with distributed workers. Currently, most of my CI does not run on arm64, but on a self-hosted Kubernetes cluster. I could switch to GitHub actions, but I'm very reluctant -- for various reasons, but mostly because of vendor lock-in. Thus, the only solution is emulation. But building the "openedx" image alone takes 3 hours (source). I do not want to delay the publication of Docker images by 3 hours for every release. In any case, it seems to me that arm64 users are mostly interested in the Tutor nightly branch -- for which no Docker image is published, and all users are required to build the images themselves. So I see little point in making drastic changes to the architecture of my CI that would provide only a minor benefit.
2. Getting MFEs to build on arm64: I think that it makes little sense to pretend that Tutor supports arm64 if MFE images do not build on that architecture. But I have no idea what's the root cause of the issue, or how to resolve it. I could use some help here...

For these reasons, I'm not sure what's the best course of action here :-/ Maybe that we should identify the issues that arm64 users face when they follow the tutorial?https://docs.tutor.overhang.io/tutorials/arm64.html

[kdmccormick]

I could switch to GitHub actions, but I'm very reluctant -- for various reasons, but mostly because of vendor lock-in ... I do not want to delay the publication of Docker images by 3 hours for every release.

That makes sense. Thinking out loud: maybe 2U or tCRIL could publish a set of ARM64 images that users could opt into. As I understand it, that wouldn't require any changes on your end or in Tutor. Users would just opt-in to the alternate images by changing the *_DOCKER_IMAGE config settings to different remotes.

In any case, it seems to me that arm64 users are mostly interested in the Tutor nightly branch -- for which no Docker image is published, and all users are required to build the images themselves.

I have been running Nightly using the official nightly images instead of building my own. Is that the wrong way to do it?

I think that it makes little sense to pretend that Tutor supports arm64 if MFE images do not build on that architecture.

I agree: this would need to be fixed upstream. I did link a potential upstream solution above.

[regisb]

I have been running Nightly using the official nightly images instead of building my own. Is that the wrong way to do it?

OK you know what? I had forgotten that these images existed 😅 this is indeed the "right" way to do it. These images are updated nightly by a cron job, which means that it should be fairly easy to make that cron job also build the arm64 "openedx" images. The 3 hour delay is not such a big deal in that situation. Note however that this cron job does not build images for plugins. For instance, the "openedx-discovery" plugin does not have any "nightly" image: https://hub.docker.com/r/overhangio/openedx-discovery/tags

This changes my perspective completely. Currently, my CI builds images on Kubernetes, but honestly it's kind of a pain to build docker-in-docker. I would be interested in setting up a build cluster with both linux/amd64 and linux/arm64 platforms (Hetzner provides Mac Mini hosting, for instance). This would resolve the question of building the Docker images. The MySQL issue is an easy fix. Provided the MFE issue is also resolved, we should be able to improve the arm64 user experience.

[yarons]

Hey guys, I really like this initiative and I'm wondering if I could help.
I created some images to support running openedx on ARM64 with images pushed to the ghcr.io docker registry.

I just wanted to point out some weird things that I ran into along the way:

1. Ubuntu does support MySQL 5.7 on ARM but not as part of the official repository so I had to install the .deb files with dpkg which is pretty terrible but it works, well, at least until the grand switchover to 8.
2. Building multiarch openedx container in parallel takes a lot of time (3 hours as mentioned) so I was wondering if it would be better to just split the process to 2 images built on different containers/machines and combining the end results with docker manifest before pushing, I'm not sure how to do it but I'm still trying to figure this out and docker manifest is still considered experimental.
3. Splitting the buildx multiarch to several building proccesses will allow identifyting the weak points and possibly change the specific build steps to run faster, another thing that docker and buildx support is the use of environment variables TARGETPLATFORM and BUILDPLATFORM.
4. I've also noticed some special "run-on-arch" that we can use to run the AMD64 and ARM64 on respective platform and decrease the building time (basically the binfmt translation time).
5. We've also discussed the mongo situation, as long and there's no plan to support older versions no special works is needed, if there's a plan to support openedX that requires preparations of mongo 3 docker for ARM64, Canonical provides .deb files for mongo 3 via Launchpad, same thing that we encountered in MySQL 5.7.

This is my repo and most of the magic happens in the GitHub Actions section:
https://github.com/yarons/tutor-arm-docker
https://github.com/yarons/mysql-arm-docker (This docker has a lot of certificate acquisition code that can be removed safely but I left it so it'll look and feel a bit like the official MySQL docker for AMD64).

I'm aware that @regisb is highly against GitHub Actions due to vendor lock-in but it should be relatively easy to create a working example on GitHub Actions and switch over to a more open platform (Do you have something in mind? Does GitLab CI/CD platform good enough?)

Thank you!

[yarons]

BTW I managed to run tutor on ARM64 in AWS (Sadly with MariaDB), it requires some tweaking to the current Dockerfiles and images (see bugs 590 and 591 on tutor's repo).

Changing the dockerize tarball to the right architecture and fixing the permissions docker (building it with docker images build permission before running the quickstart/init) should suffice.

If you need any further assistance I'd love to help.

[kdmccormick]

Wow, thank you @yarons! I will take a closer look next week, but it first glance I think what you have will be really helpful.

Did you end up trying to run Tutor with your custom MySQL 5.7 ARM64 image before switching to MariaDB?

[yarons]

Wow, thank you @yarons! I will take a closer look next week, but it first glance I think what you have will be really helpful.

Did you end up trying to run Tutor with your custom MySQL 5.7 ARM64 image before switching to MariaDB?

Not yet but thanks for the idea, I did it for @regisb , I'll try to use it for my implementation as well.

[yarons]

@kdmccormick I'm not sure it's still relevant but I ran it with my own MySQL 5.7 Docker and it worked great.

Thanks.

[regisb]

Can someone with a Mac please confirm that we can close this issue? All images should build properly on the master/nightly branches.

[yarons]

@regisb This is highly important as all the build process was done on Ubuntu ARM64 and never tested with an M1 CPU before.

[bradenmacdonald]

Can someone with a Mac please confirm that we can close this issue? All images should build properly on the master/nightly branches.

@regisb I can confirm that with the current tutor nightly + edx-platform master, the base and dev images build properly :)
However I am having some issue with npm install and libsass in particular, so I haven't been able to build the static assets. I'll investigate.

Edit: since the images built OK this must be some issue with my bind-mounted copy of edx-platform, which takes effect after the image gets built. So I assume the image is fine.

[kdmccormick]

My read on the status of this issue is:

• the tutor-mfe sub-issues have been resolved 🎉
• the image-building and mysql sub-issues are resolved by installing @bradenmacdonald 's plugin: https://github.com/open-craft/tutor-contrib-arm64

Before we can say "ARM64 works out-of-the box", though, we'll want to make sure that:

• Braden's plugin has a bona fide maintainer
• Braden's plugin is mentioned in the docs in a very obvious location
  • Perhaps: It installed by-default as an official plugin, so that ARM64 just need to enable it.
• ARM64 images are regularly built and pushed

These issues are discussed a bit in this PR: overhangio/tutor#650

[bradenmacdonald]

@kdmccormick Did people try out my plugin at the conference? I haven't heard any feedback so don't have a sense of how helpful it has been.

[kdmccormick]

@bradenmacdonald Yes, a few people used it and it seemed to work great!

[ianonavy]

First time trying to run Tutor, and I own an M1 Mac. I tried the plugin just now. Works great!

[kdmccormick]

I'm closing this to capture the fact that we now have Tutor working on ARM via an easy-to-use plugin 💯

As I mentioned in this comment we have some more work to do, which I'm splitting out into a new issue: #81