Docs for hot reloadable remote cluster credentials

docs/reference/modules/cluster/remote-clusters-api-key.asciidoc

@@ -31,7 +31,7 @@ In this model, cross-cluster operations use <<remote_cluster.port,a dedicated
 server port>> (remote cluster interface) for communication between clusters. A
 remote cluster must enable this port for local clusters to connect. Configure
 Transport Layer Security (TLS) for this port to maximize security (as explained
-in <<remote-clusters-security-api-key>>). 
+in <<remote-clusters-security-api-key>>).
 
 The local cluster must trust the remote cluster on the remote cluster interface.
 This means that the local cluster trusts the remote cluster's certificate
@@ -65,15 +65,15 @@ information, refer to https://www.elastic.co/subscriptions.
 ===== On the remote cluster
 
 // tag::remote-cluster-steps[]
-. Enable the remote cluster server on every node of the remote cluster. In 
+. Enable the remote cluster server on every node of the remote cluster. In
 `elasticsearch.yml`:
-.. Set <<remote-cluster-network-settings,`remote_cluster_server.enabled`>> to 
+.. Set <<remote-cluster-network-settings,`remote_cluster_server.enabled`>> to
 `true`.
 .. Configure the bind and publish address for remote cluster server traffic, for
 example using <<remote-cluster-network-settings,`remote_cluster.host`>>. Without
 configuring the address, remote cluster traffic may be bound to the local
 interface, and remote clusters running on other machines can't connect.
-.. Optionally, configure the remote server port using 
+.. Optionally, configure the remote server port using
 <<remote_cluster.port,`remote_cluster.port`>> (defaults to `9443`).
 . Next, generate a certificate authority (CA) and a server certificate/key pair.
 On one of the nodes of the remote cluster, from the directory where {es} has
@@ -86,8 +86,8 @@ been installed:
 ./bin/elasticsearch-certutil ca --pem --out=cross-cluster-ca.zip --pass CA_PASSWORD
 ----
 +
-Replace `CA_PASSWORD` with the password you want to use for the CA. You can 
-remove the `--pass` option and its argument if you are not deploying to a 
+Replace `CA_PASSWORD` with the password you want to use for the CA. You can
+remove the `--pass` option and its argument if you are not deploying to a
 production environment.
 
 .. Unzip the generated `cross-cluster-ca.zip` file. This compressed file
@@ -100,7 +100,7 @@ contains the following content:
 |_ ca.key
 ----
 
-.. Generate a certificate and private key pair for the nodes in the remote 
+.. Generate a certificate and private key pair for the nodes in the remote
 cluster:
 +
 [source,sh]
@@ -183,13 +183,16 @@ Replace `ALIAS` with the same name that you will use to create the remote cluste
 later. When prompted, enter the encoded cross-cluster API key created on the
 remote cluster earlier.
 
-. Restart the local cluster to load the keystore change.
+. Restart the local cluster to load changes to the keystore and settings.
+
+**Note:** If you are configuring only the cross-cluster API key, you can call the <<cluster-nodes-reload-secure-settings>> API, instead of restarting the cluster.
+Configuring the `remote_cluster_client` settings in `elasticsearch.yml` still requires a restart.
 
 [[remote-clusters-connect-api-key]]
 ==== Connect to a remote cluster
 
 :trust-mechanism: api-key
 include::remote-clusters-connect.asciidoc[]
-:!trust-mechanism: 
+:!trust-mechanism:
 
 include::{es-repo-dir}/security/authentication/remote-clusters-privileges-api-key.asciidoc[leveloffset=+1]

docs/reference/modules/cluster/remote-clusters-migration.asciidoc

@@ -51,7 +51,7 @@ include::remote-clusters-api-key.asciidoc[tag=remote-cluster-steps]
 [[remote-clusters-migration-stop]]
 ==== Stop cross-cluster operations
 
-On the local cluster, stop any persistent tasks that refer to the remote 
+On the local cluster, stop any persistent tasks that refer to the remote
 cluster:
 
 * Use the <<stop-transform>> API to stop any transforms.
@@ -74,13 +74,13 @@ roles used for cross-cluster operations. You should be able to copy these
 privileges from the original roles on the remote cluster, where they are defined
 under the certification based security model.
 ** The roles on the local cluster can't exceed the `access` privilege granted by
-the cross-cluster API key. Any extra local privileges will be suppressed by the 
+the cross-cluster API key. Any extra local privileges will be suppressed by the
 cross-cluster API key's privileges.
 ** No update is needed if the {ccr} or {ccs} tasks have been configured with a
 `superuser` role. The `superuser` role is automatically updated to allow access
 to all remote indices.
 ** Tasks that are run as regular users with named roles are immediately updated
-with the new privileges. A task will load a new definition the next time it 
+with the new privileges. A task will load a new definition the next time it
 runs.
 ** You need to restart tasks that are run using an API key (done in a later
 step).
@@ -123,7 +123,7 @@ created on the remote cluster earlier.
 . If you've dynamically configured the remote cluster (via the cluster settings
 API):
 
-.. Restart the local cluster to load changes to the keystore.
+.. Restart the local cluster to load changes to the keystore and settings.
 
 .. Re-add the remote cluster. Use the same remote cluster alias, and change the
 transport port into the remote cluster port. For example:
@@ -188,7 +188,7 @@ remote cluster:
 ----
 // TEST[skip:TODO]
 <1> The remote cluster is connected.
-<2> If present, indicates the remote cluster has connected using API key 
+<2> If present, indicates the remote cluster has connected using API key
 authentication.
 
 [[remote-clusters-migration-resume]]
@@ -204,7 +204,7 @@ task will update the task with the updated API key.
 * Use the <<start-transform>> API to start any transforms.
 * Use the <<ml-open-job>> API to open any anomaly detection jobs.
 * Use the <<ccr-post-resume-follow>> API to resume any auto-follow {ccr}.
-* Use the <<ccr-resume-auto-follow-pattern>> API to resume any manual {ccr} or 
+* Use the <<ccr-resume-auto-follow-pattern>> API to resume any manual {ccr} or
 existing indices that were created from the auto-follow pattern.
 
 [[remote-clusters-migration-disable-cert]]
@@ -232,8 +232,8 @@ or distributed, is no longer trusted.
 Another solution is to apply IP filters to the transport interface, blocking
 traffic from outside the cluster.
 
-. Optionally, delete any roles on the remote cluster that were only used for 
-cross-cluster operations. These roles are no longer used under the API key based 
+. Optionally, delete any roles on the remote cluster that were only used for
+cross-cluster operations. These roles are no longer used under the API key based
 security model.
 
 [[remote-clusters-migration-rollback]]
@@ -252,7 +252,7 @@ the migration.
 . On each node, remove the `remote_cluster_client.ssl.*` settings from
 `elasticsearch.yml`.
 
-. Restart the local cluster to apply changes to the keystore and 
+. Restart the local cluster to apply changes to the keystore and
 `elasticsearch.yml`.
 
 . On the local cluster, apply the original remote cluster settings. If the
@@ -263,4 +263,4 @@ remote cluster connection has been configured statically (using the
 local cluster has connected to the remote cluster. The response should have
 `"connected": true` and not have `"cluster_credentials": "::es_redacted::"`.
 
-. Restart any persistent tasks that you've stopped earlier.
+. Restart any persistent tasks that you've stopped earlier.

docs/reference/modules/cluster/remote-clusters-settings.asciidoc

@@ -65,7 +65,7 @@ mode are described separately.
   is used as the fallback setting.
 
 
-`cluster.remote.<cluster_alias>.credentials` (<<secure-settings,Secure>>)::
+`cluster.remote.<cluster_alias>.credentials` (<<secure-settings,Secure>>, <<reloadable-secure-settings,Reloadable>>)::
 
 beta:[]
   Per cluster setting for configuring <<remote-clusters-api-key,remote clusters with the API Key based model>>.
@@ -75,6 +75,8 @@ beta:[]
   The presence (or not) of this setting determines which model a remote cluster uses.
   If present, the remote cluster uses the API key based model.
   Otherwise, it uses the certificate based model.
+  If the setting is added, removed, or updated in the <<secure-settings,{es} keystore>> and reloaded via the
+  <<cluster-nodes-reload-secure-settings>> API, the cluster will automatically rebuild its connection to the remote.
 
 [[remote-cluster-sniff-settings]]
 ==== Sniff mode remote cluster settings