Mariadb max_packet_allowed variable exposed #295

g7morris · 2023-10-26T21:40:02Z

Trying to make max_packet_allowed a variable instead of a value not exposed to the user.

Despite the mariadb container using the base image which I assumed uses confd, the environmental variable which I set in the isle-dc project .env file of DB_MAX_ALLOWED_PACKET=268435456 doesn't seem to work.

Am I missing something?

Just curious, I'll keep plugging along.

Hope you are well.

Cheers ,
Gavin

g7morris · 2023-10-27T14:12:18Z

I think I see what is missing. I'm not accustomed to s6 overlay version 3 yet and can see I'm missing a chunk of definitions to fire. I'll work on filling in the missing bits. Ah learning. ;)

g7morris · 2023-10-27T15:39:58Z

Okay a little progress on creating the S6 structure.

I went ahead to at least give max_packet_allowed a default value of 256MB for when the container is first started (Dockerfile ENV) however confd & s6 don't seem to be "firing" or pulling the value from the project .env file.

@nigelgbanks Any pointers on this per se?

nigelgbanks · 2023-10-29T17:54:58Z

the environmental variable which I set in the isle-dc project .env file of DB_MAX_ALLOWED_PACKET=268435456 doesn't seem to work.

That file is for docker-compose.yml file and is not included in the container environment variables. You'd have to specify the environment variable on the mariadb service here:

https://github.com/Islandora-Devops/isle-dc/blob/development/build/docker-compose/docker-compose.mariadb.yml#L11

For more information, the relevant docker-compose documentation.

I'll submit a pull request that sorts this out and exposes it, following the conventions of the this repo.

g7morris · 2023-10-29T22:38:29Z

Rather than making a new MR which ignores the work here, can you confirm what elements here are in the right direction please?

From your comments sounds like I just need to add the following below to the project docker-compose.yml as that was the missing part?

My intent isn't to make you redo my work here per se, just to learn and practice enough on how to expose variables properly.

  mariadb:
    image: ${REPOSITORY}/mariadb:${TAG}
    environment: 
      DB_MAX_ALLOWED_PACKET: ${DB_MAX_ALLOWED_PACKET:-268435456}

I'll test shortly.

Thanks for taking a look!

g7morris · 2023-10-29T22:51:28Z

Seems to work, the value in the mariadb-server.cnf file now reflects the value within the .env file when changed and if not uses the default 256MB.

Are there other settings files or techniques needed?

nigelgbanks · 2023-10-30T16:12:49Z

Rather than making a new MR which ignores the work here, can you confirm what elements here are in the right direction please?

Sure, I was opting for a separate pull as every line will change, but I'll put notes on this pull.

nigelgbanks

Also rebase on main to get passing tests as the latest version of docker compose introduced a breaking change, that has been fixed on main.

nigelgbanks · 2023-10-30T16:17:29Z

mariadb/Dockerfile

@@ -22,5 +22,7 @@ RUN --mount=type=cache,id=mariadb-apk-${TARGETARCH},sharing=locked,target=/var/c
 # base image. Set to 10 minutes just incase it ran on very old or overallocated
 # hardware.
 ENV S6_CMD_WAIT_FOR_SERVICES_MAXTIME=600000
+# Default value of 256 MB in bytes
+ENV DB_MAX_ALLOWED_PACKET=268435456


Please change to the following, and remove the comment.

ENV MYSQL_MAX_ALLOWED_PACKET=16777216

Prefix to match the other environment variables provided by the image, see the README.md.

Also, change the default value to be the same default as is provided by MariaDB (16777216) to not change the existing behavior. See the documentation here.

Also add documentation for the new environment variable to the image's README.md.

For example:

Environment Variable Default Description

MYSQL_ROOT_PASSWORD The database root user password. Defaults to DB_ROOT_PASSWORD

MYSQL_ROOT_USER The database root user (used to create the site database). Defaults to DB_ROOT_USER

MYSQL_MAX_ALLOWED_PACKET 16777216 Max packet length to send to or receive from the server, documentation

nigelgbanks · 2023-10-30T16:19:12Z

mariadb/rootfs/etc/confd/conf.d/mariadb-server.toml

@@ -0,0 +1,7 @@
+[template]
+src = "mariadb-server.tmpl"


Following convention, elsewhere in the repository the src should be named mariadb-server.cnf.tmpl.

Also rename the corresponding file in the template folder.

Oh, same goes for this .toml file, it should become mariadb-server.cnf.toml

nigelgbanks · 2023-10-30T16:20:59Z

mariadb/rootfs/etc/confd/conf.d/mariadb-server.toml

+[template]
+src = "mariadb-server.tmpl"
+dest = "/etc/my.cnf.d/mariadb-server.cnf"
+uid = 100


uid should be 0

nigelgbanks · 2023-10-30T16:21:14Z

mariadb/rootfs/etc/confd/conf.d/mariadb-server.toml

+src = "mariadb-server.tmpl"
+dest = "/etc/my.cnf.d/mariadb-server.cnf"
+uid = 100
+gid = 1000


gid should also be 0.

nigelgbanks · 2023-10-30T16:21:42Z

mariadb/rootfs/etc/confd/conf.d/mariadb-server.toml

+dest = "/etc/my.cnf.d/mariadb-server.cnf"
+uid = 100
+gid = 1000
+mode = "0664"


mode should be 0644.

nigelgbanks · 2023-10-30T16:22:43Z

mariadb/rootfs/etc/confd/templates/mariadb-server.tmpl

@@ -8,7 +8,8 @@
 # this is only for the mysqld standalone daemon
 [mysqld]
 # skip-networking
-
+# Allow for max_allowed_packet to be set in the project .env
+max_allowed_packet ={{ getenv "DB_MAX_ALLOWED_PACKET" }}


Remove the comment as it is not accurate.

Change the line to:

max_allowed_packet={{ getenv "MYSQL_MAX_ALLOWED_PACKET" }}

Also don't forget to rename the file as is stated in an earlier comment.

nigelgbanks · 2023-10-30T16:22:55Z

mariadb/rootfs/etc/s6-overlay/s6-rc.d/set-max-packet/type

@@ -0,0 +1 @@
+oneshot


Delete this file.

nigelgbanks · 2023-10-30T16:23:02Z

mariadb/rootfs/etc/s6-overlay/s6-rc.d/set-max-packet/up

@@ -0,0 +1 @@
+/etc/s6-overlay/scripts/set-max-packet.sh


Delete this file.

nigelgbanks · 2023-10-30T16:23:09Z

mariadb/rootfs/etc/s6-overlay/scripts/set-max-packet.sh

@@ -0,0 +1,8 @@
+#!/command/with-contenv bash


Delete this file.

nigelgbanks · 2023-10-30T16:23:26Z

mariadb/rootfs/etc/s6-overlay/s6-rc.d/set-max-packet/dependencies.d/ready

Delete this file.

nigelgbanks · 2023-10-30T16:26:22Z

From your comments sounds like I just need to add the following below to the project docker-compose.yml as that was the missing part?

My intent isn't to make you redo my work here per se, just to learn and practice enough on how to expose variables properly.
  mariadb:
    image: ${REPOSITORY}/mariadb:${TAG}
    environment: 
      DB_MAX_ALLOWED_PACKET: ${DB_MAX_ALLOWED_PACKET:-268435456}
I'll test shortly.

Thanks for taking a look!

Yup, that was what I was saying. 👍

… env values ignored

g7morris · 2023-10-30T19:04:42Z

Thanks for all the responses, comments and great insights here @nigelgbanks I'll get to work today with atomic fixes and commits, along with the appropriate resolutions. I appreciate you taking the time to help me learn and get the right changes in.

g7morris · 2023-10-30T20:34:10Z

Hi @nigelgbanks

Think we're ready for another review!

I've pushed all of the changes and here is a list below of what I did in order.

Thanks again!

Rebase done - Mariadb max_packet_allowed variable exposed #295 (review)
- On branch - on local - isle-buildkit git:(matomo-upgrade-4151)
- git fetch origin
- git rebase origin/main
  - Successfully rebased and updated refs/heads/mariadb_max_packet
- git push origin HEAD --force

Enumerating objects: 35, done.
Counting objects: 100% (35/35), done.
Delta compression using up to 8 threads
Compressing objects: 100% (20/20), done.
Writing objects: 100% (27/27), 2.93 KiB | 2.93 MiB/s, done.
Total 27 (delta 6), reused 2 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (6/6), completed with 4 local objects.
remote: 
remote: GitHub found 6 vulnerabilities on Islandora-Devops/isle-buildkit's default branch (6 moderate). To find out more, visit:
remote:      https://github.com/Islandora-Devops/isle-buildkit/security/dependabot
remote: 
To https://github.com/Islandora-Devops/isle-buildkit.git
 + c92f255...672b564 HEAD -> mariadb_max_packet (forced update)

Dockerfile - Mariadb max_packet_allowed variable exposed #295 (comment)
- DONE - variable name change along with ENV
- DONE - README.md updated to reflect new default variable
Confd - Mariadb max_packet_allowed variable exposed #295 (comment)
- DONE - renamed confd tmpl & toml files according to project conventions
- DONE - uid is now 0
- DONE - guid is now 0
- DONE - mode is now 0644
- DONE - comment is removed
- DONE - variable is renamed to MYSQL_MAX_ALLOWED_PACKET
S6 -
- DONE - removed type file
- DONE - removed up file
- DONE - removed script
- DONE - removed ready file
- DONE - removed mariadb/rootfs/etc/s6-overlay/s6-rc.d/set-max-packet/dependencies.d/
- DONE - removed mariadb/rootfs/etc/s6-overlay/s6-rc.d/set-max-packet

g7morris · 2023-10-30T21:19:05Z

Hi @nigelgbanks

I think we're in business.

Just tested all of this on a local isle-dc project and it seems to work nicely.

FYI, one just can't exceed 1 GB as a max value per docs https://mariadb.com/docs/server/ref/mdb/system-variables/max_allowed_packet/.

Test 1 - Use default settings of 16 MB

Ran make bake mariadb for new mariadb Docker image
Only one edit indocker-compose.yml to change the Docker image.

mariadb:
     image: islandora/mariadb:local

make up - Start containers with new mariadb image
docker-compose exec mariadb bash - shell into the container
mysql - hop onto command line
Ran this query SHOW VARIABLES WHERE variable_name = 'max_allowed_packet'; and got the default value.

MariaDB [(none)]> SHOW VARIABLES WHERE variable_name = 'max_allowed_packet';
+--------------------+----------+
| Variable_name      | Value    |
+--------------------+----------+
| max_allowed_packet | 16777216 |
+--------------------+----------+
1 row in set (0.004 sec)

Test 2 - Change settings in docker-compose.yml to 900 MB

Ran make bake mariadb for new mariadb Docker image
Edited indocker-compose.yml the following changes for the new Docker image and the new env.

mariadb:
     image: islandora/mariadb:local
    environment:
        MYSQL_MAX_ALLOWED_PACKET: 943718400

make up - Start containers with new mariadb image
docker-compose exec mariadb bash - shell into the container
mysql - hop onto command line
Ran this query SHOW VARIABLES WHERE variable_name = 'max_allowed_packet'; and got the newly updated value!

MariaDB [(none)]> SHOW VARIABLES WHERE variable_name = 'max_allowed_packet';
+--------------------+----------+
| Variable_name      | Value    |
+--------------------+----------+
| max_allowed_packet | 943718400 |
+--------------------+----------+
1 row in set (0.004 sec)

Please note: That one can also use the following:

 mariadb:
    #image: ${REPOSITORY}/mariadb:${TAG}
    image: islandora/mariadb:local
    environment:
      MYSQL_MAX_ALLOWED_PACKET: ${MYSQL_MAX_ALLOWED_PACKET:-268435456}

with a line in the .env file MYSQL_MAX_ALLOWED_PACKET=943718400 and this will work great as well.

g7morris requested a review from nigelgbanks October 26, 2023 21:40

nigelgbanks requested changes Oct 30, 2023

View reviewed changes

Gavin Morris added 2 commits October 30, 2023 13:06

Mariadb max_packet_allowed - Draft

3fdce2d

Draft 2 - max_packet_now has default env but still NOT dynamic. Confd…

672b564

… env values ignored

g7morris force-pushed the mariadb_max_packet branch from c92f255 to 672b564 Compare October 30, 2023 17:07

Variable rename and README update per Comment 1

06ef6f9

Gavin Morris added 3 commits October 30, 2023 16:18

Renamed confd cnf files with correct uid settings

84177c6

Remove errant comment

e651ccb

Remove three unneeded files

c0d927f

g7morris self-assigned this Oct 30, 2023

g7morris added the mariadb label Oct 30, 2023

Remove extra set-max-packet directories and files

4aa186d

Gavin Morris added 2 commits October 30, 2023 16:41

Missing toml value fixed

983ffc2

Quotes for perm values removed by mistake

22f6437

g7morris changed the title ~~Mariadb max_packet_allowed - Draft~~ Mariadb max_packet_allowed variable exposed Oct 30, 2023

nigelgbanks approved these changes Nov 1, 2023

View reviewed changes

nigelgbanks merged commit 36ec971 into main Nov 1, 2023
76 checks passed

nigelgbanks deleted the mariadb_max_packet branch November 1, 2023 21:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mariadb max_packet_allowed variable exposed #295

Mariadb max_packet_allowed variable exposed #295

g7morris commented Oct 26, 2023

g7morris commented Oct 27, 2023

g7morris commented Oct 27, 2023

nigelgbanks commented Oct 29, 2023

g7morris commented Oct 29, 2023

g7morris commented Oct 29, 2023

nigelgbanks commented Oct 30, 2023

nigelgbanks left a comment

nigelgbanks Oct 30, 2023

nigelgbanks Oct 30, 2023

nigelgbanks Oct 30, 2023

nigelgbanks Oct 30, 2023

nigelgbanks Oct 30, 2023

nigelgbanks Oct 30, 2023

nigelgbanks Oct 30, 2023

nigelgbanks Oct 30, 2023

nigelgbanks Oct 30, 2023

nigelgbanks Oct 30, 2023

nigelgbanks Oct 30, 2023

nigelgbanks Oct 30, 2023

nigelgbanks commented Oct 30, 2023

g7morris commented Oct 30, 2023

g7morris commented Oct 30, 2023

g7morris commented Oct 30, 2023

Environment Variable	Default	Description
MYSQL_ROOT_PASSWORD		The database root user password. Defaults to `DB_ROOT_PASSWORD`
MYSQL_ROOT_USER		The database root user (used to create the site database). Defaults to `DB_ROOT_USER`
MYSQL_MAX_ALLOWED_PACKET	16777216	Max packet length to send to or receive from the server, documentation

		@@ -0,0 +1 @@
		oneshot

		@@ -0,0 +1 @@
		/etc/s6-overlay/scripts/set-max-packet.sh

Mariadb max_packet_allowed variable exposed #295

Mariadb max_packet_allowed variable exposed #295

Conversation

g7morris commented Oct 26, 2023

g7morris commented Oct 27, 2023

g7morris commented Oct 27, 2023

nigelgbanks commented Oct 29, 2023

g7morris commented Oct 29, 2023

g7morris commented Oct 29, 2023

nigelgbanks commented Oct 30, 2023

nigelgbanks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nigelgbanks commented Oct 30, 2023

g7morris commented Oct 30, 2023

g7morris commented Oct 30, 2023

g7morris commented Oct 30, 2023

Test 1 - Use default settings of 16 MB

Test 2 - Change settings in docker-compose.yml to 900 MB