From 7b60a926f87d16d81634fbfb19df27282db2bfe3 Mon Sep 17 00:00:00 2001 From: Nikolaj Volgushev Date: Thu, 15 Feb 2024 12:02:13 +0100 Subject: [PATCH] Docs for hot-reloadable remote cluster credentials (#105483) Docs PR to accompany https://github.com/elastic/elasticsearch/pull/103215. Resolves: ES-7625 --- .../cluster/remote-clusters-api-key.asciidoc | 21 +++++++++-------- .../remote-clusters-migration.asciidoc | 20 ++++++++-------- .../cluster/remote-clusters-settings.asciidoc | 5 +++- .../remote-clusters-troubleshooting.asciidoc | 23 ++++++++----------- docs/reference/setup/secure-settings.asciidoc | 1 + 5 files changed, 37 insertions(+), 33 deletions(-) diff --git a/docs/reference/modules/cluster/remote-clusters-api-key.asciidoc b/docs/reference/modules/cluster/remote-clusters-api-key.asciidoc index 73ec6966ec049..b95ebdf143a57 100644 --- a/docs/reference/modules/cluster/remote-clusters-api-key.asciidoc +++ b/docs/reference/modules/cluster/remote-clusters-api-key.asciidoc @@ -31,7 +31,7 @@ In this model, cross-cluster operations use <> (remote cluster interface) for communication between clusters. A remote cluster must enable this port for local clusters to connect. Configure Transport Layer Security (TLS) for this port to maximize security (as explained -in <>). +in <>). The local cluster must trust the remote cluster on the remote cluster interface. This means that the local cluster trusts the remote cluster's certificate @@ -65,15 +65,15 @@ information, refer to https://www.elastic.co/subscriptions. ===== On the remote cluster // tag::remote-cluster-steps[] -. Enable the remote cluster server on every node of the remote cluster. In +. Enable the remote cluster server on every node of the remote cluster. In `elasticsearch.yml`: -.. Set <> to +.. Set <> to `true`. .. Configure the bind and publish address for remote cluster server traffic, for example using <>. Without configuring the address, remote cluster traffic may be bound to the local interface, and remote clusters running on other machines can't connect. -.. Optionally, configure the remote server port using +.. Optionally, configure the remote server port using <> (defaults to `9443`). . Next, generate a certificate authority (CA) and a server certificate/key pair. On one of the nodes of the remote cluster, from the directory where {es} has @@ -86,8 +86,8 @@ been installed: ./bin/elasticsearch-certutil ca --pem --out=cross-cluster-ca.zip --pass CA_PASSWORD ---- + -Replace `CA_PASSWORD` with the password you want to use for the CA. You can -remove the `--pass` option and its argument if you are not deploying to a +Replace `CA_PASSWORD` with the password you want to use for the CA. You can +remove the `--pass` option and its argument if you are not deploying to a production environment. .. Unzip the generated `cross-cluster-ca.zip` file. This compressed file @@ -100,7 +100,7 @@ contains the following content: |_ ca.key ---- -.. Generate a certificate and private key pair for the nodes in the remote +.. Generate a certificate and private key pair for the nodes in the remote cluster: + [source,sh] @@ -183,13 +183,16 @@ Replace `ALIAS` with the same name that you will use to create the remote cluste later. When prompted, enter the encoded cross-cluster API key created on the remote cluster earlier. -. Restart the local cluster to load the keystore change. +. Restart the local cluster to load changes to the keystore and settings. + +**Note:** If you are configuring only the cross-cluster API key, you can call the <> API, instead of restarting the cluster. +Configuring the `remote_cluster_client` settings in `elasticsearch.yml` still requires a restart. [[remote-clusters-connect-api-key]] ==== Connect to a remote cluster :trust-mechanism: api-key include::remote-clusters-connect.asciidoc[] -:!trust-mechanism: +:!trust-mechanism: include::{es-repo-dir}/security/authentication/remote-clusters-privileges-api-key.asciidoc[leveloffset=+1] diff --git a/docs/reference/modules/cluster/remote-clusters-migration.asciidoc b/docs/reference/modules/cluster/remote-clusters-migration.asciidoc index 9db7c4a0257ad..e205d7cb141fe 100644 --- a/docs/reference/modules/cluster/remote-clusters-migration.asciidoc +++ b/docs/reference/modules/cluster/remote-clusters-migration.asciidoc @@ -51,7 +51,7 @@ include::remote-clusters-api-key.asciidoc[tag=remote-cluster-steps] [[remote-clusters-migration-stop]] ==== Stop cross-cluster operations -On the local cluster, stop any persistent tasks that refer to the remote +On the local cluster, stop any persistent tasks that refer to the remote cluster: * Use the <> API to stop any transforms. @@ -74,13 +74,13 @@ roles used for cross-cluster operations. You should be able to copy these privileges from the original roles on the remote cluster, where they are defined under the certification based security model. ** The roles on the local cluster can't exceed the `access` privilege granted by -the cross-cluster API key. Any extra local privileges will be suppressed by the +the cross-cluster API key. Any extra local privileges will be suppressed by the cross-cluster API key's privileges. ** No update is needed if the {ccr} or {ccs} tasks have been configured with a `superuser` role. The `superuser` role is automatically updated to allow access to all remote indices. ** Tasks that are run as regular users with named roles are immediately updated -with the new privileges. A task will load a new definition the next time it +with the new privileges. A task will load a new definition the next time it runs. ** You need to restart tasks that are run using an API key (done in a later step). @@ -123,7 +123,7 @@ created on the remote cluster earlier. . If you've dynamically configured the remote cluster (via the cluster settings API): -.. Restart the local cluster to load changes to the keystore. +.. Restart the local cluster to load changes to the keystore and settings. .. Re-add the remote cluster. Use the same remote cluster alias, and change the transport port into the remote cluster port. For example: @@ -188,7 +188,7 @@ remote cluster: ---- // TEST[skip:TODO] <1> The remote cluster is connected. -<2> If present, indicates the remote cluster has connected using API key +<2> If present, indicates the remote cluster has connected using API key authentication. [[remote-clusters-migration-resume]] @@ -204,7 +204,7 @@ task will update the task with the updated API key. * Use the <> API to start any transforms. * Use the <> API to open any anomaly detection jobs. * Use the <> API to resume any auto-follow {ccr}. -* Use the <> API to resume any manual {ccr} or +* Use the <> API to resume any manual {ccr} or existing indices that were created from the auto-follow pattern. [[remote-clusters-migration-disable-cert]] @@ -232,8 +232,8 @@ or distributed, is no longer trusted. Another solution is to apply IP filters to the transport interface, blocking traffic from outside the cluster. -. Optionally, delete any roles on the remote cluster that were only used for -cross-cluster operations. These roles are no longer used under the API key based +. Optionally, delete any roles on the remote cluster that were only used for +cross-cluster operations. These roles are no longer used under the API key based security model. [[remote-clusters-migration-rollback]] @@ -252,7 +252,7 @@ the migration. . On each node, remove the `remote_cluster_client.ssl.*` settings from `elasticsearch.yml`. -. Restart the local cluster to apply changes to the keystore and +. Restart the local cluster to apply changes to the keystore and `elasticsearch.yml`. . On the local cluster, apply the original remote cluster settings. If the @@ -263,4 +263,4 @@ remote cluster connection has been configured statically (using the local cluster has connected to the remote cluster. The response should have `"connected": true` and not have `"cluster_credentials": "::es_redacted::"`. -. Restart any persistent tasks that you've stopped earlier. \ No newline at end of file +. Restart any persistent tasks that you've stopped earlier. diff --git a/docs/reference/modules/cluster/remote-clusters-settings.asciidoc b/docs/reference/modules/cluster/remote-clusters-settings.asciidoc index 85e63918e6fa9..bba8c7ffb3491 100644 --- a/docs/reference/modules/cluster/remote-clusters-settings.asciidoc +++ b/docs/reference/modules/cluster/remote-clusters-settings.asciidoc @@ -65,7 +65,8 @@ mode are described separately. is used as the fallback setting. -`cluster.remote..credentials` (<>):: +`cluster.remote..credentials` (<>, <>):: +[[remote-cluster-credentials-setting]] beta:[] Per cluster setting for configuring <>. @@ -75,6 +76,8 @@ beta:[] The presence (or not) of this setting determines which model a remote cluster uses. If present, the remote cluster uses the API key based model. Otherwise, it uses the certificate based model. + If the setting is added, removed, or updated in the <> and reloaded via the + <> API, the cluster will automatically rebuild its connection to the remote. [[remote-cluster-sniff-settings]] ==== Sniff mode remote cluster settings diff --git a/docs/reference/modules/cluster/remote-clusters-troubleshooting.asciidoc b/docs/reference/modules/cluster/remote-clusters-troubleshooting.asciidoc index 5dc6ab8c08c88..f7b08b40bb7ef 100644 --- a/docs/reference/modules/cluster/remote-clusters-troubleshooting.asciidoc +++ b/docs/reference/modules/cluster/remote-clusters-troubleshooting.asciidoc @@ -31,13 +31,13 @@ incoming cross-cluster requests by default, while it is ready to send outgoing cross-cluster requests. Ensure you've enabled the remote cluster server on every node of the remote cluster. In `elasticsearch.yml`: -* Set <> to +* Set <> to `true`. * Configure the bind and publish address for remote cluster server traffic, for example using <>. Without configuring the address, remote cluster traffic may be bound to the local interface, and remote clusters running on other machines can't connect. -* Optionally, configure the remote server port using +* Optionally, configure the remote server port using <> (defaults to `9443`). [[remote-clusters-troubleshooting-common-issues]] @@ -73,7 +73,7 @@ org.elasticsearch.transport.ConnectTransportException: [][192.168.0.42:9443] *co ====== Resolution * Check the host and port for the remote cluster are correct. -* Ensure the <> on the remote cluster. * Ensure no firewall is blocking the communication. @@ -108,11 +108,11 @@ cause of the failure. For example: * Is the remote cluster certificate not signed by a trusted CA? This is the most likely cause. -* Is hostname verification failing? +* Is hostname verification failing? * Is the certificate expired? Once you know the cause, you should be able to fix it by adjusting the remote -cluster related SSL settings on either the local cluster or the remote cluster. +cluster related SSL settings on either the local cluster or the remote cluster. Often, the issue is on the local cluster. For example, fix it by configuring necessary trusted CAs (`xpack.security.remote_cluster_client.ssl.certificate_authorities`). @@ -275,7 +275,7 @@ This does not show up in the logs of the remote cluster. ====== Resolution Add the cross-cluster API key to {es} keystore on every node of the local -cluster. Restart the local cluster to reload the keystore. +cluster. Use the <> API to reload the keystore. [[remote-clusters-troubleshooting-wrong-api-key-type]] ===== Using the wrong API key type @@ -302,8 +302,7 @@ This does not show up in the logs of the remote cluster. Ask the remote cluster administrator to create and distribute a <>. Replace the existing API key in the {es} keystore with this cross-cluster API key on every -node of the local cluster. Restart the local cluster for keystore changes to -take effect. +node of the local cluster. Use the <> API to reload the keystore. [[remote-clusters-troubleshooting-non-valid-api-key]] ===== Invalid API key @@ -334,15 +333,14 @@ The remote cluster logs `Authentication using apikey failed`: Ask the remote cluster administrator to create and distribute a <>. Replace the existing API key in the {es} keystore with this cross-cluster API key on every -node of the local cluster. Restart the local cluster for keystore changes to -take effect. +node of the local cluster. Use the <> API to reload the keystore. [[remote-clusters-troubleshooting-insufficient-privileges]] ===== API key or local user has insufficient privileges The effective permission for a local user running requests on a remote cluster is determined by the intersection of the cross-cluster API key's privileges and -the local user's `remote_indices` privileges. +the local user's `remote_indices` privileges. ====== Symptom @@ -366,8 +364,7 @@ This does not show up in any logs. create and distribute a <>. Replace the existing API key in the {es} keystore with this cross-cluster API key on every -node of the local cluster. Restart the local cluster for keystore changes to -take effect. +node of the local cluster. Use the <> API to reload the keystore. [[remote-clusters-troubleshooting-no-remote_indices-privileges]] ===== Local user has no `remote_indices` privileges diff --git a/docs/reference/setup/secure-settings.asciidoc b/docs/reference/setup/secure-settings.asciidoc index 028e766d063df..cb88be94e17b6 100644 --- a/docs/reference/setup/secure-settings.asciidoc +++ b/docs/reference/setup/secure-settings.asciidoc @@ -64,3 +64,4 @@ There are reloadable secure settings for: * <> * <> * <> +* <>