fix: broken mysql after palm upgrade

This fix is for a rather serious issue that affects users who upgrade
from Olive to Palm. The client mysql charset and collation was
incorrectly set to utf8mb4, while the server stil runs utf8mb3. Only
users who run the mysql container are affected.

To resolve this issue, we explicitely configure the client to use the
utf8mb3 charset/collation.

Important note: users who have somehow managed to upgrade from olive to
Palm before may find themselves in an undefined state. They might have
to fix their mysql data manually. Same thing for users who launched Palm
from scratch; although, according to my preliinary tests, they should be
able to downgrade their connection from utf8mb4 to utf8mb3 without
issue.

In addition, we upgrade to mysql 8.1.0. Among many other fixes, this
avoids a server restart after the upgrade:

> An in-place upgrade from MySQL 5.7 to MySQL 8.0, without a server
> restart, could result in unexpected errors when executing queries on
> tables. This fix eliminates the need to restart the server between the
> upgrade and queries. (Bug #35410528)

https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-34.html

See also the 8.1.0 release notes:

https://dev.mysql.com/doc/relnotes/mysql/8.1/en/news-8-1-0.html

Close #887.

changelog.d/20230815_064652_regis_fix_mysql_upgrade_charset.md

@@ -0,0 +1,2 @@
+- 💥[Bugfix] Fix mysql crash after upgrade to Palm. After an upgrade to Palm, the mysql client run by Django defaults to a utf8mb4 character set and collation, but the mysql server still runs with utf8mb3. This causes broken data during migration from Olive to Palm, and more generally when data is written to the database. To resolve this issue, we explicitely set the utf8mb3 charset and collation in the client. Users who were running Palm might have to fix their data manually. In the future we will upgrade the mysql server to utf8mb4. (by @regisb)
+- [Improvement] We upgrade to MySQL 8.1.0 to avoid having to restart the server after the upgrade.

docs/configuration.rst

@@ -81,7 +81,7 @@ This configuration parameter defines which MongoDB Docker image to use.
 
 .. https://hub.docker.com/_/mysql/tags?page=1&name=8.0
 
-- ``DOCKER_IMAGE_MYSQL`` (default: ``"docker.io/mysql:8.0.33"``)
+- ``DOCKER_IMAGE_MYSQL`` (default: ``"docker.io/mysql:8.1.0"``)
 
 This configuration parameter defines which MySQL Docker image to use.
 

tests/commands/test_images.py

@@ -49,7 +49,7 @@ def test_images_pull_all_vendor_images(self, image_pull: Mock) -> None:
         self.assertIsNone(result.exception)
         self.assertEqual(0, result.exit_code)
         # Note: we should update this tag whenever the mysql image is updated
-        image_pull.assert_called_once_with("docker.io/mysql:8.0.33")
+        image_pull.assert_called_once_with("docker.io/mysql:8.1.0")
 
     def test_images_printtag_image(self) -> None:
         result = self.invoke(["images", "printtag", "openedx"])

tutor/commands/jobs.py

@@ -241,7 +241,7 @@ def sqlshell(args: list[str]) -> t.Iterable[tuple[str, str]]:
     Extra arguments will be passed to the `mysql` command verbatim. For instance, to
     show tables from the "openedx" database, run `do sqlshell openedx -e 'show tables'`.
     """
-    command = "mysql --user={{ MYSQL_ROOT_USERNAME }} --password={{ MYSQL_ROOT_PASSWORD }} --host={{ MYSQL_HOST }} --port={{ MYSQL_PORT }}"
+    command = "mysql --user={{ MYSQL_ROOT_USERNAME }} --password={{ MYSQL_ROOT_PASSWORD }} --host={{ MYSQL_HOST }} --port={{ MYSQL_PORT }} --default-character-set=utf8mb3"
     if args:
         command += " " + shlex.join(args)  # pylint: disable=protected-access
     yield ("lms", command)

tutor/templates/apps/openedx/config/partials/auth.yml

@@ -17,5 +17,8 @@ DATABASES:
     ATOMIC_REQUESTS: true
     OPTIONS:
       init_command: "SET sql_mode='STRICT_TRANS_TABLES'"
+      {%- if RUN_MYSQL %}
+      charset: "utf8mb3"
+      {%- endif %}
 EMAIL_HOST_USER: "{{ SMTP_USERNAME }}"
 EMAIL_HOST_PASSWORD: "{{ SMTP_PASSWORD }}"

tutor/templates/config/defaults.yml

@@ -15,7 +15,7 @@ DOCKER_IMAGE_OPENEDX_DEV: "openedx-dev:{{ TUTOR_VERSION }}"
 DOCKER_IMAGE_CADDY: "docker.io/caddy:2.6.4"
 DOCKER_IMAGE_ELASTICSEARCH: "docker.io/elasticsearch:7.17.9"
 DOCKER_IMAGE_MONGODB: "docker.io/mongo:4.4.22"
-DOCKER_IMAGE_MYSQL: "docker.io/mysql:8.0.33"
+DOCKER_IMAGE_MYSQL: "docker.io/mysql:8.1.0"
 DOCKER_IMAGE_PERMISSIONS: "{{ DOCKER_REGISTRY }}overhangio/openedx-permissions:{{ TUTOR_VERSION }}"
 DOCKER_IMAGE_REDIS: "docker.io/redis:7.0.11"
 DOCKER_IMAGE_SMTP: "docker.io/devture/exim-relay:4.96-r1-0"

tutor/templates/local/docker-compose.yml

@@ -40,7 +40,7 @@ services:
   {% if RUN_MYSQL -%}
   mysql:
     image: {{ DOCKER_IMAGE_MYSQL }}
-    command: mysqld --character-set-server=utf8 --collation-server=utf8_general_ci
+    command: mysqld --character-set-server=utf8mb3 --collation-server=utf8mb3_general_ci
     restart: unless-stopped
     user: "999:999"
     volumes: