HDDS-11631. Add Run with Docker Compose guide #101

ptlrs · 2024-10-31T05:46:54Z

What changes were proposed in this pull request?

Added Run with Docker Compose guide to the Developer Guide section

Please describe your PR in detail:

Added a Run with Docker compose guide
Added new keywords to fix spelling linter errors
Added a placeholder page to document docker/docker-compose related issues
Added mermaid package to render images via pnpm add which updated package.json and pnpm-lock.yaml
Updated the pnpm version to fix incompatible lock file version error when running docker-compose

What is the link to the Apache Jira?

https://issues.apache.org/jira/browse/HDDS-11631

How was this patch tested?

Ran the website locally via docker-compose
CI:
~~https://github.com/ptlrs/ozone-site/actions/runs/11606604449~~
https://github.com/ptlrs/ozone-site/actions/runs/11845847017

errose28

Thanks for adding this @ptlrs. There's some more information we can add to this page too:

Provide a quick summary of the various compose clusters available for HA and security.
Explain how to make configuration changes with the docker-config file.

errose28 · 2024-11-06T00:11:59Z

cspell.yaml

@@ -48,6 +48,7 @@ words:
 - ratis
 - OM
 - SCM
+- DN


Unlike OM and SCM we don't usually use DN as an abbreviation for datanode when talking about Ozone in an official context, so it was deliberately omitted here. It might be better to use an inline ignore to skip it in the mermaid code block.

docs/06-troubleshooting/15-docker.md

errose28 · 2024-11-06T00:13:26Z

docs/08-developer-guide/02-run/02-docker-compose.md

@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
 - Define the Ozone runner image at `ozone-docker-runner` and its purpose to wrap Ozone binaries.
 - How to define which image gets used in the cluster (runner or other pre-built image).
 - Changing configurations in Docker Compose (`docker-config` file, transformation.py, etc).
+


We can delete the TODO above this section since we are filling in the page in this PR.

errose28 · 2024-11-06T00:17:41Z

package.json

    "@docusaurus/plugin-pwa": "3.3.2",
+    "@docusaurus/preset-classic": "3.3.2",
+    "@docusaurus/theme-mermaid": "3.3.2",


I checked out this change and ran docker compse up --build to build an image with the new dependencies but got this error. Not sure exactly what the cause is:

ozone-site-site-1 | ozone-site-site-1 | > [email protected] start /ozone-site ozone-site-site-1 | > docusaurus start --port 3001 "--host" "0.0.0.0" ozone-site-site-1 | ozone-site-site-1 | [INFO] Starting the development server... ozone-site-site-1 | ozone-site-site-1 | [ERROR] Error: Docusaurus was unable to resolve the "@docusaurus/theme-mermaid" theme. Make sure one of the following packages are installed: ozone-site-site-1 | - @docusaurus/theme-mermaid ozone-site-site-1 | - @docusaurus/docusaurus-theme-theme-mermaid ozone-site-site-1 | at resolveModuleName (/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/plugins/moduleShorthand.js:41:15) ozone-site-site-1 | at normalizeShorthand (/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/plugins/configs.js:76:60) ozone-site-site-1 | at /ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/plugins/configs.js:90:63 ozone-site-site-1 | at Array.map (<anonymous>) ozone-site-site-1 | at loadPluginConfigs (/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/plugins/configs.js:90:48) ozone-site-site-1 | at async initPlugins (/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/plugins/init.js:42:27) ozone-site-site-1 | at async /ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:136:36 ozone-site-site-1 | at async loadSite (/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/site.js:127:45) ozone-site-site-1 | at async createReloadableSite (/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/commands/start/utils.js:46:16) ozone-site-site-1 | at async Command.start (/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/commands/start/start.js:22:28) ozone-site-site-1 | [INFO] Docusaurus version: 3.3.2 ozone-site-site-1 | Node version: v20.18.0 ozone-site-site-1 | ELIFECYCLE Command failed with exit code 1. ozone-site-site-1 exited with code 1

I saw that I faced the same problem after uninstalling docusaurus/theme-mermaid.
I was able to resolve it by reinstalling the package again on my mac.
Next, I ran docker compse up --build and then ran it with no issues.

errose28 · 2024-11-06T00:18:07Z

package.json

@@ -1,5 +1,5 @@
 {
-  "packageManager": "pnpm@8.7.1",
+  "packageManager": "pnpm@9.12.3",


Can you clarify why we need the pnpm version bump? Is mermaid incompatible with older versions?

With [email protected], running docker-compose fails with the following error:

0.280 ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-8.7.1.tgz 1.109 ERR_PNPM_LOCKFILE_BREAKING_CHANGE Lockfile /ozone-site/pnpm-lock.yaml not compatible with current pnpm 1.109 1.109 Run with the --force parameter to recreate the lockfile. ------ failed to solve: process "/bin/sh -c pnpm install --prod --frozen-lockfile" did not complete successfully: exit code: 1 `docker-compose` process finished with exit code 17

I added the docusaurus/theme-mermaid package via pnpm add @docusaurus/[email protected] on my mac. This updated the pnpm-lock.yaml and the package.json files.

errose28 · 2024-11-06T00:30:30Z

docs/08-developer-guide/02-run/02-docker-compose.md

+
+This will pull the official Apache Ozone images from Docker Hub.
+
+## Architecture Diagram


To avoid duplicate content I think we should leave this information for the page dedicated to Ozone architecture. Default ports will be covered in docs/administrator-guide/configuration/basic/network/. This may mean we don't need to add mermaid in this PR.

As a new user, it is not immediately apparent how the Ozone services will be divvied across containers. It could be the case that all Ozone services run within one container. There are several possible configurations.

I added the mermaid diagram to visualize how many containers will be available and how each Ozone service maps to those containers.

It just so happens that each Ozone service is a separate container and the container diagram looks similar to what could easily have been an Ozone architecture diagram. Perhaps the title and wording add to the confusion.

If it is still a cause of confusion, I can remove the diagram. Not a problem.

errose28 · 2024-11-06T00:36:46Z

docs/08-developer-guide/02-run/02-docker-compose.md

+
+## Running Ozone
+
+You can run Ozone either using your locally built version or using official pre-built Docker images.


I think we should organize this a little differently. We can have one section on running docker-compose from a pre-build Ozone tarball, regardless of where the tarball came from (built from source or Ozone release). Then we can mention the pre-built docker images for each release separately if people want to use those for other things.

I don't see why someone would pull just the compose file from the tarball and manually pair it with a docker image built from the same version.

errose28 · 2024-11-06T00:40:57Z

docs/08-developer-guide/02-run/02-docker-compose.md

+
+### Port Mappings
+
+The default setup exposes the following ports:


I'm not sure the RPC connections will work with clients outside the compose network. The configuration and hostnames would need to be configured manually at least. HTTP endpoints should work out of the box though.

Default ports will be covered in docs/administrator-guide/configuration/basic/network/

Removed the port mappings section here

errose28 · 2024-11-06T00:41:22Z

docs/08-developer-guide/02-run/02-docker-compose.md

+
+```bash
+# Start the cluster
+docker-compose up -d


Lets use docker compose since v2 is the standard going forward.

errose28 · 2024-11-06T00:43:11Z

docs/08-developer-guide/02-run/02-docker-compose.md

+docker-compose logs -f [service_name]
+
+# Scale data nodes
+docker-compose up -d --scale datanode=3


This only works on clusters that provide a service called datanode. Some of the compose clusters like topology and upgrade that need to use individual datanodes differently give each datanode a different service.

adoroszlai

I suggest writing the page on Docker for users first. We should refer developers to that guide for basics.

errose28 · 2024-11-14T22:52:17Z

Good suggestion Attila. Rishabh and I talked on a call and he had the idea of creating a new compose file with all the configs baked so that new users can pull just that one file instead of a whole release tarball where the configs live outside the docker compose definition. For now when writing the docker quick start we can refer to the current process of pulling a full release though.

ptlrs added 2 commits October 30, 2024 21:58

HDDS-11631.Added Running Ozone From Docker Compose guide

c31c52f

HDDS-11631. Fix linter errors

8554c03

github-actions bot added the website-v2 Tasks for the new ozone website targeting the HDDS-9225-website-v2 branch label Oct 31, 2024

errose28 self-requested a review November 1, 2024 17:31

errose28 added the docs Changes updating documentation on the website label Nov 1, 2024

errose28 reviewed Nov 6, 2024

View reviewed changes

ptlrs added 2 commits November 12, 2024 23:48

HDDS-11631. Fix review comments

8d51361

HDDS-11631. Fix review comments

3ac2f6d

ptlrs requested a review from errose28 November 14, 2024 21:32

adoroszlai reviewed Nov 14, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-11631. Add Run with Docker Compose guide #101

HDDS-11631. Add Run with Docker Compose guide #101

ptlrs commented Oct 31, 2024 •

edited

Loading

errose28 left a comment

errose28 Nov 6, 2024

ptlrs Nov 13, 2024

errose28 Nov 6, 2024

ptlrs Nov 14, 2024

errose28 Nov 6, 2024

ptlrs Nov 14, 2024

errose28 Nov 6, 2024

ptlrs Nov 13, 2024

errose28 Nov 6, 2024

ptlrs Nov 13, 2024

errose28 Nov 6, 2024

ptlrs Nov 14, 2024

errose28 Nov 6, 2024 •

edited

Loading

ptlrs Nov 13, 2024

errose28 Nov 6, 2024

ptlrs Nov 13, 2024

errose28 Nov 6, 2024

ptlrs Nov 14, 2024

adoroszlai left a comment

errose28 commented Nov 14, 2024

@@ @@ -48,6 +48,7 @@ words: @@
               - ratis
               - OM
               - SCM
+              - DN


		This will pull the official Apache Ozone images from Docker Hub.

		## Architecture Diagram


		## Running Ozone

		You can run Ozone either using your locally built version or using official pre-built Docker images.


		### Port Mappings

		The default setup exposes the following ports:

HDDS-11631. Add Run with Docker Compose guide #101

Are you sure you want to change the base?

HDDS-11631. Add Run with Docker Compose guide #101

Conversation

ptlrs commented Oct 31, 2024 • edited Loading

What changes were proposed in this pull request?

What is the link to the Apache Jira?

How was this patch tested?

errose28 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

errose28 Nov 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adoroszlai left a comment

Choose a reason for hiding this comment

errose28 commented Nov 14, 2024

ptlrs commented Oct 31, 2024 •

edited

Loading

errose28 Nov 6, 2024 •

edited

Loading