doc: migrate 0.50.0

docs/content/docs/components/cdn.md

@@ -4,36 +4,138 @@ weight: 3
 ---
 
 ## What's CDN
-CDN is a service dedicated to receive and store CDS's logs. In a near future, it will also be able to manage artifacts and cache used by your jobs. 
+
+CDN is a service dedicated to receive and store CDS's logs and artifacts. 
 
 CDN stores the list of all known log or artifact items in a Postgres database and communicates with storage backends to store the contents of those items.
 These backends are call units and there are two types of units in CDN:
 
 * Buffer unit: To store logs and artifacts of incoming jobs, these units are designed to be fast for read / write operations, but with limited capacity.
 
-* Storage Unit: to store complete job's logs and artifact.
+* Storage Unit: to store complete job's logs and artifacts.
 
-When logs or file are received by CDN, it will first store these items in its buffer. Then, when the item is fully received, it will be moved to one of the configured storage units.
+When logs or file are received by CDN from a cds worker, it will first store these items in its buffer. Then, when the item is fully received, it will be moved to one of the configured storage units.
 If the CDN service is configured with multiple storage units, each unit periodically checks for missing items and synchronizes these items from other units.
 
+CDS UI and CLI communicate with CDN to get entire logs, or stream them.
+
+## Supported units
+* Buffer (type: log): Redis.
+* Buffer (type: file): Local, NFS
+* Storage: Local, Swift, S3, Webdav
 
 ## Configuration
-Like any other CDS service, CDN requires to be authenticated with a consumer. The required scopes are Service, Worker and RunExecution.
+Like any other CDS service, CDN requires to be authenticated with a consumer. The required scopes are `Service`, `Worker` and `RunExecution`.
+
+You can generate a configuration file with the `engine` binary:
+
+```sh
+$ engine config new cdn > cds-configuration.toml
+```
 
 You must have at least one storage unit, one file buffer and one log buffer to be able to run CDN.
 
-## Supported units
-* Buffer (type: log): Redis.
-* Buffer (type: file): Local.
-* Storage: Local, Swift, S3, Webdav, CDS (cds unit is used for migration purpose and will be removed in future release).
+### CDN artifact configuration
 
+#### Storage Unit Buffer
 
-## Use case
+You must have a `storageUnits.buffers` , one for the type `log`, another for the type `file`.
 
-Workers and hatcheries communicate with CDN, sending step logs and service log.
+Type `log`:
 
-![CDN_RECEIVE](/images/cdn_logs_receive.png?width=600px)
+```toml
+      [cdn.storageUnits.buffers.redis]
+        bufferType = "log"
 
-CDS UI and CLI communicate with CDN to get entire logs, or stream them.
+        [cdn.storageUnits.buffers.redis.redis]
+          host = "aaa@instance0,instance1,instance2"
+          password = "your-password"
+```
+
+Type `file`:
+
+```toml
+      [cdn.storageUnits.buffers.local-buffer]
+
+        # it can be 'log' to receive logs or 'file' to receive artifacts
+        bufferType = "file"
+
+        [cdn.storageUnits.buffers.local-buffer.local]
+          path = "/var/lib/cds-engine/cdn-buffer"
+```
+
+To multi-instanciate the cdn service, you can use a NFS for the bufferType file, example:
+
+```toml
+      [cdn.storageUnits.buffers.buffer-nfs]
+        bufferType = "file"
+
+        [cdn.storageUnits.buffers.buffer-nfs.nfs]
+          host = "w.x.y.z"
+          targetPartition = "/zpool-partition/cdn"
+          userID = 0
+          groupID = 0
+
+          [[cdn.storageUnits.buffers.buffer-nfs.nfs.encryption]]
+            Cipher = "aes-gcm"
+            Identifier = "nfs-buffer-id"
+            ## enter a key here, 32 lenght
+            Key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+            Sealed = false   
+```
+
+#### Storage Units Storage
+
+The storage unit 'storage' store the artifacts. 
+You can use `Local`, `Swift`, `S3`, `Webdav`
+
+Example of storage unit `local`:
+
+```toml
+    [cdn.storageUnits.storages]
+
+      [cdn.storageUnits.storages.local]
+
+        # flag to disabled backend synchronization
+        disableSync = false
+
+        # global bandwith shared by the sync processes (in Mb)
+        syncBandwidth = 128
+
+        # number of parallel sync processes
+        syncParallel = 2
+
+        [cdn.storageUnits.storages.local.local]
+          path = "/tmp/cds/local-storage"
+
+          [[cdn.storageUnits.storages.local.local.encryption]]
+            Cipher = "aes-gcm"
+            Identifier = "cdn-storage-local"
+            LocatorSalt = "xxxxxxxxx"
+            SecretValue = "xxxxxxxxxxxxxxxxx"
+            Timestamp = 0
+```
+
+Example of storage unit `swift`:
+```
+    [cdn.storageUnits.storages]
+
+      [cdn.storageUnits.storages.swift]
+        syncParallel = 6
+        syncBandwidth = 1000
+
+        [cdn.storageUnits.storages.swift.swift]
+          address = "https://xxx.yyy.zzz/v3"
+          username = "foo"
+          password = "your-password-here"
+          tenant = "your-tenant-here"
+          domain = "Default"
+          region = "XXX"
+          containerPrefix = "prod"
 
-![CDN_GET](/images/cdn_logs_get.png?width=600px)
+          [[cdn.storageUnits.storages.swift.swift.encryption]]
+            Cipher = "aes-gcm"
+            Identifier = "swift-backend-id"
+            LocatorSalt = "XXXXXXXX"
+            SecretValue = "XXXXXXXXXXXXXXXX"
+```

docs/content/hosting/upgrade/migrate_0.50.md

@@ -0,0 +1,123 @@
+---
+title: "Migrate 0.50"
+weight: 1
+---
+
+## Migrate an existing instance
+
+Before upgrading your CDS Instance:
+- You have to backup your databases: cds and cdn databases.
+- You have to install the version 0.49.0 if you use the 0.48 version.
+- The backend cds-backend have to be removed from the cdn configuration.
+
+## PostgreSQL
+
+PostgreSQL 14 min is now recommanded
+
+## CDN Service
+
+This release introduced the CDN Artifacts. This means that all artifacts upload / download will be done through the CDN service.
+This is not enabled by default, you have to enable that with a feature flipping. The version 0.51 will use the CDN service to manage artifacts by default.
+
+### How to configure the new CDN artifact service?
+
+Some configuration have to be added to the cdn service to manage artifacts.
+
+You have to add a `storageUnits.buffers` with the type "file".
+```toml
+      [cdn.storageUnits.buffers.local-buffer]
+
+        # it can be 'log' to receive logs or 'file' to receive artifacts
+        bufferType = "file"
+
+        [cdn.storageUnits.buffers.local-buffer.local]
+          path = "/var/lib/cds-engine/cdn-buffer"
+```
+
+To multi-instanciate the cdn service, you can use a NFS for the bufferType file, example:
+
+```toml
+      [cdn.storageUnits.buffers.buffer-nfs]
+        bufferType = "file"
+
+        [cdn.storageUnits.buffers.buffer-nfs.nfs]
+          host = "w.x.y.z"
+          targetPartition = "/zpool-partition/cdn"
+          userID = 0
+          groupID = 0
+
+          [[cdn.storageUnits.buffers.buffer-nfs.nfs.encryption]]
+            Cipher = "aes-gcm"
+            Identifier = "nfs-buffer-id"
+            ## enter a key here, 32 lenght
+            Key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+            Sealed = false   
+```
+
+## How to enable the CDS Artifact?
+
+Make sure you have configured cdn in the paragraph above, then enable it with this command:
+
+```sh
+cat <<EOF > feature.yaml
+name: cdn-artifact
+rule: return true
+EOF
+cdsctl admin feature import feature.yaml
+```
+
+In the next release (0.51.0), you don't need to use the feature flipping `cdn-artifact` to enable cdn artifact.
+
+### How to migrate existing artifacts to the new backend?
+
+If you want to migrate some artifacts to the new backend, you can use cdsctl:
+```sh
+$ cdsctl workflow artifact cdn-migrate <project_key> <workflow_name> <run_number>
+```
+This command will migrate artifacts for one workflow run.
+
+## Workflow Runs : Purge and Retention
+
+The default retention rule have to be added in api configuration:
+
+```toml
+  [api.workflow]
+
+    # Default rule for workflow run retention policy, this rule can be overridden on each workflow.
+    # Example: 'return run_days_before < 365' keeps runs for one year.
+    defaultRetentionPolicy = "return run_days_before < 365"
+```
+Documentation: https://ovh.github.io/cds/docs/concepts/workflow/retention/
+
+In the next release (0.51.0), you don't need to use the feature flipping `workflow-retention-policy` to enable the new purge system.
+
+## Spawning worker : MaxAttemptsNumberBeforeFailure
+
+A new hatchery configuration attribute is available to control the maximum attempts to start a same job.
+
+Example on the `hatchery.local`:
+
+```toml
+      [hatchery.local.commonConfiguration.provision]
+
+        # Maximum attempts to start a same job. -1 to disable failing jobs when to many attempts
+        # maxAttemptsNumberBeforeFailure = 5
+```
+
+## CDS Binaries lazy loading
+
+`downloadFromGitHub` and `supportedOSArch` are added. This will allow you to download cds workers / cdsctl from GitHub if it's not in already downloaded into the `directory`.
+
+```toml
+  [api.download]
+
+    # this directory contains cds binaries. If it's empty, cds will download binaries from GitHub (property downloadFromGitHub) or from an artifactory instance (property artifactory) to it
+    directory = "/var/lib/cds-engine"
+
+    # allow downloading binaries from GitHub
+    downloadFromGitHub = true
+
+    # example: ["darwin/amd64","darwin/arm64","linux/amd64","windows/amd64"]. If empty, all os / arch are supported: windows,darwin,linux,freebsd,openbsd and amd64,arm,386,arm64,ppc64le
+    supportedOSArch = []
+```
+