Merge branch 'main' into custom-roles-poc-es

docs/reference/modules/cluster/remote-clusters-api-key.asciidoc

@@ -31,7 +31,7 @@ In this model, cross-cluster operations use <<remote_cluster.port,a dedicated
 server port>> (remote cluster interface) for communication between clusters. A
 remote cluster must enable this port for local clusters to connect. Configure
 Transport Layer Security (TLS) for this port to maximize security (as explained
-in <<remote-clusters-security-api-key>>). 
+in <<remote-clusters-security-api-key>>).
 
 The local cluster must trust the remote cluster on the remote cluster interface.
 This means that the local cluster trusts the remote cluster's certificate
@@ -65,15 +65,15 @@ information, refer to https://www.elastic.co/subscriptions.
 ===== On the remote cluster
 
 // tag::remote-cluster-steps[]
-. Enable the remote cluster server on every node of the remote cluster. In 
+. Enable the remote cluster server on every node of the remote cluster. In
 `elasticsearch.yml`:
-.. Set <<remote-cluster-network-settings,`remote_cluster_server.enabled`>> to 
+.. Set <<remote-cluster-network-settings,`remote_cluster_server.enabled`>> to
 `true`.
 .. Configure the bind and publish address for remote cluster server traffic, for
 example using <<remote-cluster-network-settings,`remote_cluster.host`>>. Without
 configuring the address, remote cluster traffic may be bound to the local
 interface, and remote clusters running on other machines can't connect.
-.. Optionally, configure the remote server port using 
+.. Optionally, configure the remote server port using
 <<remote_cluster.port,`remote_cluster.port`>> (defaults to `9443`).
 . Next, generate a certificate authority (CA) and a server certificate/key pair.
 On one of the nodes of the remote cluster, from the directory where {es} has
@@ -86,8 +86,8 @@ been installed:
 ./bin/elasticsearch-certutil ca --pem --out=cross-cluster-ca.zip --pass CA_PASSWORD
 ----
 +
-Replace `CA_PASSWORD` with the password you want to use for the CA. You can 
-remove the `--pass` option and its argument if you are not deploying to a 
+Replace `CA_PASSWORD` with the password you want to use for the CA. You can
+remove the `--pass` option and its argument if you are not deploying to a
 production environment.
 
 .. Unzip the generated `cross-cluster-ca.zip` file. This compressed file
@@ -100,7 +100,7 @@ contains the following content:
 |_ ca.key
 ----
 
-.. Generate a certificate and private key pair for the nodes in the remote 
+.. Generate a certificate and private key pair for the nodes in the remote
 cluster:
 +
 [source,sh]
@@ -183,13 +183,16 @@ Replace `ALIAS` with the same name that you will use to create the remote cluste
 later. When prompted, enter the encoded cross-cluster API key created on the
 remote cluster earlier.
 
-. Restart the local cluster to load the keystore change.
+. Restart the local cluster to load changes to the keystore and settings.
+
+**Note:** If you are configuring only the cross-cluster API key, you can call the <<cluster-nodes-reload-secure-settings>> API, instead of restarting the cluster.
+Configuring the `remote_cluster_client` settings in `elasticsearch.yml` still requires a restart.
 
 [[remote-clusters-connect-api-key]]
 ==== Connect to a remote cluster
 
 :trust-mechanism: api-key
 include::remote-clusters-connect.asciidoc[]
-:!trust-mechanism: 
+:!trust-mechanism:
 
 include::{es-repo-dir}/security/authentication/remote-clusters-privileges-api-key.asciidoc[leveloffset=+1]

docs/reference/modules/cluster/remote-clusters-migration.asciidoc

@@ -51,7 +51,7 @@ include::remote-clusters-api-key.asciidoc[tag=remote-cluster-steps]
 [[remote-clusters-migration-stop]]
 ==== Stop cross-cluster operations
 
-On the local cluster, stop any persistent tasks that refer to the remote 
+On the local cluster, stop any persistent tasks that refer to the remote
 cluster:
 
 * Use the <<stop-transform>> API to stop any transforms.
@@ -74,13 +74,13 @@ roles used for cross-cluster operations. You should be able to copy these
 privileges from the original roles on the remote cluster, where they are defined
 under the certification based security model.
 ** The roles on the local cluster can't exceed the `access` privilege granted by
-the cross-cluster API key. Any extra local privileges will be suppressed by the 
+the cross-cluster API key. Any extra local privileges will be suppressed by the
 cross-cluster API key's privileges.
 ** No update is needed if the {ccr} or {ccs} tasks have been configured with a
 `superuser` role. The `superuser` role is automatically updated to allow access
 to all remote indices.
 ** Tasks that are run as regular users with named roles are immediately updated
-with the new privileges. A task will load a new definition the next time it 
+with the new privileges. A task will load a new definition the next time it
 runs.
 ** You need to restart tasks that are run using an API key (done in a later
 step).
@@ -123,7 +123,7 @@ created on the remote cluster earlier.
 . If you've dynamically configured the remote cluster (via the cluster settings
 API):
 
-.. Restart the local cluster to load changes to the keystore.
+.. Restart the local cluster to load changes to the keystore and settings.
 
 .. Re-add the remote cluster. Use the same remote cluster alias, and change the
 transport port into the remote cluster port. For example:
@@ -188,7 +188,7 @@ remote cluster:
 ----
 // TEST[skip:TODO]
 <1> The remote cluster is connected.
-<2> If present, indicates the remote cluster has connected using API key 
+<2> If present, indicates the remote cluster has connected using API key
 authentication.
 
 [[remote-clusters-migration-resume]]
@@ -204,7 +204,7 @@ task will update the task with the updated API key.
 * Use the <<start-transform>> API to start any transforms.
 * Use the <<ml-open-job>> API to open any anomaly detection jobs.
 * Use the <<ccr-post-resume-follow>> API to resume any auto-follow {ccr}.
-* Use the <<ccr-resume-auto-follow-pattern>> API to resume any manual {ccr} or 
+* Use the <<ccr-resume-auto-follow-pattern>> API to resume any manual {ccr} or
 existing indices that were created from the auto-follow pattern.
 
 [[remote-clusters-migration-disable-cert]]
@@ -232,8 +232,8 @@ or distributed, is no longer trusted.
 Another solution is to apply IP filters to the transport interface, blocking
 traffic from outside the cluster.
 
-. Optionally, delete any roles on the remote cluster that were only used for 
-cross-cluster operations. These roles are no longer used under the API key based 
+. Optionally, delete any roles on the remote cluster that were only used for
+cross-cluster operations. These roles are no longer used under the API key based
 security model.
 
 [[remote-clusters-migration-rollback]]
@@ -252,7 +252,7 @@ the migration.
 . On each node, remove the `remote_cluster_client.ssl.*` settings from
 `elasticsearch.yml`.
 
-. Restart the local cluster to apply changes to the keystore and 
+. Restart the local cluster to apply changes to the keystore and
 `elasticsearch.yml`.
 
 . On the local cluster, apply the original remote cluster settings. If the
@@ -263,4 +263,4 @@ remote cluster connection has been configured statically (using the
 local cluster has connected to the remote cluster. The response should have
 `"connected": true` and not have `"cluster_credentials": "::es_redacted::"`.
 
-. Restart any persistent tasks that you've stopped earlier.
+. Restart any persistent tasks that you've stopped earlier.

docs/reference/modules/cluster/remote-clusters-settings.asciidoc

@@ -65,7 +65,8 @@ mode are described separately.
   is used as the fallback setting.
 
 
-`cluster.remote.<cluster_alias>.credentials` (<<secure-settings,Secure>>)::
+`cluster.remote.<cluster_alias>.credentials` (<<secure-settings,Secure>>, <<reloadable-secure-settings,Reloadable>>)::
+[[remote-cluster-credentials-setting]]
 
 beta:[]
   Per cluster setting for configuring <<remote-clusters-api-key,remote clusters with the API Key based model>>.
@@ -75,6 +76,8 @@ beta:[]
   The presence (or not) of this setting determines which model a remote cluster uses.
   If present, the remote cluster uses the API key based model.
   Otherwise, it uses the certificate based model.
+  If the setting is added, removed, or updated in the <<secure-settings,{es} keystore>> and reloaded via the
+  <<cluster-nodes-reload-secure-settings>> API, the cluster will automatically rebuild its connection to the remote.
 
 [[remote-cluster-sniff-settings]]
 ==== Sniff mode remote cluster settings

docs/reference/modules/cluster/remote-clusters-troubleshooting.asciidoc

@@ -31,13 +31,13 @@ incoming cross-cluster requests by default, while it is ready to send outgoing
 cross-cluster requests. Ensure you've enabled the remote cluster server on every
 node of the remote cluster. In `elasticsearch.yml`:
 
-* Set <<remote-cluster-network-settings,`remote_cluster_server.enabled`>> to 
+* Set <<remote-cluster-network-settings,`remote_cluster_server.enabled`>> to
 `true`.
 * Configure the bind and publish address for remote cluster server traffic, for
 example using <<remote-cluster-network-settings,`remote_cluster.host`>>. Without
 configuring the address, remote cluster traffic may be bound to the local
 interface, and remote clusters running on other machines can't connect.
-* Optionally, configure the remote server port using 
+* Optionally, configure the remote server port using
 <<remote_cluster.port,`remote_cluster.port`>> (defaults to `9443`).
 
 [[remote-clusters-troubleshooting-common-issues]]
@@ -73,7 +73,7 @@ org.elasticsearch.transport.ConnectTransportException: [][192.168.0.42:9443] *co
 ====== Resolution
 
 * Check the host and port for the remote cluster are correct.
-* Ensure the <<remote-clusters-troubleshooting-enable-server,remote cluster 
+* Ensure the <<remote-clusters-troubleshooting-enable-server,remote cluster
 server is enabled>> on the remote cluster.
 * Ensure no firewall is blocking the communication.
 
@@ -108,11 +108,11 @@ cause of the failure. For example:
 
 * Is the remote cluster certificate not signed by a trusted CA? This is the most
 likely cause.
-* Is hostname verification failing? 
+* Is hostname verification failing?
 * Is the certificate expired?
 
 Once you know the cause, you should be able to fix it by adjusting the remote
-cluster related SSL settings on either the local cluster or the remote cluster. 
+cluster related SSL settings on either the local cluster or the remote cluster.
 
 Often, the issue is on the local cluster. For example, fix it by configuring necessary
 trusted CAs (`xpack.security.remote_cluster_client.ssl.certificate_authorities`).
@@ -275,7 +275,7 @@ This does not show up in the logs of the remote cluster.
 ====== Resolution
 
 Add the cross-cluster API key to {es} keystore on every node of the local
-cluster. Restart the local cluster to reload the keystore.
+cluster. Use the <<cluster-nodes-reload-secure-settings>> API to reload the keystore.
 
 [[remote-clusters-troubleshooting-wrong-api-key-type]]
 ===== Using the wrong API key type
@@ -302,8 +302,7 @@ This does not show up in the logs of the remote cluster.
 Ask the remote cluster administrator to create and distribute a
 <<security-api-create-cross-cluster-api-key,cross-cluster API key>>. Replace the
 existing API key in the {es} keystore with this cross-cluster API key on every
-node of the local cluster. Restart the local cluster for keystore changes to
-take effect.
+node of the local cluster. Use the <<cluster-nodes-reload-secure-settings>> API to reload the keystore.
 
 [[remote-clusters-troubleshooting-non-valid-api-key]]
 ===== Invalid API key
@@ -334,15 +333,14 @@ The remote cluster logs `Authentication using apikey failed`:
 Ask the remote cluster administrator to create and distribute a
 <<security-api-create-cross-cluster-api-key,cross-cluster API key>>. Replace the
 existing API key in the {es} keystore with this cross-cluster API key on every
-node of the local cluster. Restart the local cluster for keystore changes to
-take effect.
+node of the local cluster. Use the <<cluster-nodes-reload-secure-settings>> API to reload the keystore.
 
 [[remote-clusters-troubleshooting-insufficient-privileges]]
 ===== API key or local user has insufficient privileges
 
 The effective permission for a local user running requests on a remote cluster
 is determined by the intersection of the cross-cluster API key's privileges and
-the local user's `remote_indices` privileges. 
+the local user's `remote_indices` privileges.
 
 ====== Symptom
 
@@ -366,8 +364,7 @@ This does not show up in any logs.
 create and distribute a
 <<security-api-create-cross-cluster-api-key,cross-cluster API key>>. Replace the
 existing API key in the {es} keystore with this cross-cluster API key on every
-node of the local cluster. Restart the local cluster for keystore changes to
-take effect.
+node of the local cluster. Use the <<cluster-nodes-reload-secure-settings>> API to reload the keystore.
 
 [[remote-clusters-troubleshooting-no-remote_indices-privileges]]
 ===== Local user has no `remote_indices` privileges

docs/reference/setup/secure-settings.asciidoc

@@ -64,3 +64,4 @@ There are reloadable secure settings for:
 * <<ref-jwt-settings, JWT realm>>
 * <<ref-ad-settings, Active Directory realm>>
 * <<ref-ldap-settings, LDAP realm>>
+* <<remote-cluster-credentials-setting, Remote cluster credentials for the API key based security model>>

server/src/internalClusterTest/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreIT.java

@@ -267,7 +267,7 @@ public void testRestoreIndexWithMissingShards() throws Exception {
                 .setIndices("test-idx-all", "test-idx-none", "test-idx-some", "test-idx-closed")
                 .setWaitForCompletion(true)
         );
-        assertThat(sne.getMessage(), containsString("Indices don't have primary shards"));
+        assertThat(sne.getMessage(), containsString("the following indices have unassigned primary shards"));
 
         if (randomBoolean()) {
             logger.info("checking snapshot completion using status");

server/src/internalClusterTest/java/org/elasticsearch/snapshots/SharedClusterSnapshotRestoreIT.java

@@ -769,7 +769,7 @@ public void testUnallocatedShards() {
             SnapshotException.class,
             clusterAdmin().prepareCreateSnapshot("test-repo", "test-snap").setWaitForCompletion(true).setIndices("test-idx")
         );
-        assertThat(sne.getMessage(), containsString("Indices don't have primary shards"));
+        assertThat(sne.getMessage(), containsString("the following indices have unassigned primary shards"));
         assertThat(getRepositoryData("test-repo"), is(RepositoryData.EMPTY));
     }
 

server/src/main/java/org/elasticsearch/TransportVersions.java

@@ -197,7 +197,7 @@ static TransportVersion def(int id) {
      * Reference to the minimum transport version that can be used with CCS.
      * This should be the transport version used by the previous minor release.
      */
-    public static final TransportVersion MINIMUM_CCS_VERSION = V_8_11_X;
+    public static final TransportVersion MINIMUM_CCS_VERSION = V_8_12_0;
 
     static final NavigableMap<Integer, TransportVersion> VERSION_IDS = getAllVersionIds(TransportVersions.class);
 

server/src/main/java/org/elasticsearch/common/ReferenceDocs.java

@@ -70,6 +70,7 @@ public enum ReferenceDocs {
     BOOTSTRAP_CHECK_TOKEN_SSL,
     BOOTSTRAP_CHECK_SECURITY_MINIMAL_SETUP,
     CONTACT_SUPPORT,
+    UNASSIGNED_SHARDS,
     // this comment keeps the ';' on the next line so every entry above has a trailing ',' which makes the diff for adding new links cleaner
     ;
 

server/src/main/java/org/elasticsearch/indices/recovery/StatelessPrimaryRelocationAction.java

@@ -8,7 +8,6 @@
 
 package org.elasticsearch.indices.recovery;
 
-import org.elasticsearch.TransportVersions;
 import org.elasticsearch.action.ActionRequest;
 import org.elasticsearch.action.ActionRequestValidationException;
 import org.elasticsearch.action.ActionResponse;
@@ -49,11 +48,7 @@ public Request(StreamInput in) throws IOException {
             shardId = new ShardId(in);
             targetNode = new DiscoveryNode(in);
             targetAllocationId = in.readString();
-            if (in.getTransportVersion().onOrAfter(TransportVersions.V_8_11_X)) {
-                clusterStateVersion = in.readVLong();
-            } else {
-                clusterStateVersion = 0L; // temporary bwc: do not wait for cluster state to be applied
-            }
+            clusterStateVersion = in.readVLong();
         }
 
         @Override
@@ -68,9 +63,7 @@ public void writeTo(StreamOutput out) throws IOException {
             shardId.writeTo(out);
             targetNode.writeTo(out);
             out.writeString(targetAllocationId);
-            if (out.getTransportVersion().onOrAfter(TransportVersions.V_8_11_X)) {
-                out.writeVLong(clusterStateVersion);
-            } // temporary bwc: just omit it, the receiver doesn't wait for a cluster state anyway
+            out.writeVLong(clusterStateVersion);
         }
 
         public long recoveryId() {

server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java

@@ -59,6 +59,7 @@
 import org.elasticsearch.cluster.service.MasterService;
 import org.elasticsearch.cluster.service.MasterServiceTaskQueue;
 import org.elasticsearch.common.Priority;
+import org.elasticsearch.common.ReferenceDocs;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.UUIDs;
 import org.elasticsearch.common.collect.ImmutableOpenMap;
@@ -111,6 +112,7 @@
 import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
+import java.util.TreeSet;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.CopyOnWriteArrayList;
 import java.util.concurrent.Executor;
@@ -4017,14 +4019,18 @@ private SnapshotsInProgress createSnapshot(
                 repositoryName
             );
             if (request.partial() == false) {
-                Set<String> missing = new HashSet<>();
+                Set<String> missing = new TreeSet<>(); // sorted for more usable message
                 for (Map.Entry<ShardId, ShardSnapshotStatus> entry : shards.entrySet()) {
                     if (entry.getValue().state() == ShardState.MISSING) {
                         missing.add(entry.getKey().getIndex().getName());
                     }
                 }
                 if (missing.isEmpty() == false) {
-                    throw new SnapshotException(snapshot, "Indices don't have primary shards " + missing);
+                    throw new SnapshotException(snapshot, Strings.format("""
+                        the following indices have unassigned primary shards \
+                        and cannot be included in a snapshot unless [partial] is set to [true]: %s; \
+                        for help with troubleshooting unassigned shards see %s
+                        """, missing, ReferenceDocs.UNASSIGNED_SHARDS));
                 }
             }
             final var newEntry = SnapshotsInProgress.startedEntry(

server/src/main/resources/org/elasticsearch/common/reference-docs-links.json

@@ -30,5 +30,6 @@
   "BOOTSTRAP_CHECK_TLS": "bootstrap-checks-xpack.html#bootstrap-checks-tls",
   "BOOTSTRAP_CHECK_TOKEN_SSL": "bootstrap-checks-xpack.html#_token_ssl_check",
   "BOOTSTRAP_CHECK_SECURITY_MINIMAL_SETUP": "security-minimal-setup.html",
-  "CONTACT_SUPPORT": "troubleshooting.html#troubleshooting-contact-support"
+  "CONTACT_SUPPORT": "troubleshooting.html#troubleshooting-contact-support",
+  "UNASSIGNED_SHARDS": "red-yellow-cluster-status.html"
 }