diff --git a/src/current/_data/releases.yml b/src/current/_data/releases.yml index 79fee4eb1f2..6294647a51a 100644 --- a/src/current/_data/releases.yml +++ b/src/current/_data/releases.yml @@ -7787,13 +7787,6 @@ docker_arm_limited_access: false source: true previous_release: v24.1.9 - cloud_only: true - cloud_only_message_short: 'Available only for select CockroachDB Cloud clusters' - cloud_only_message: > - This version is currently available only for select - CockroachDB Cloud clusters. To request to upgrade - a CockroachDB self-hosted cluster to this version, - [contact support](https://support.cockroachlabs.com/hc/requests/new). - release_name: v24.2.8 major_version: v24.2 @@ -7821,13 +7814,6 @@ docker_arm_limited_access: false source: true previous_release: v24.2.7 - cloud_only: true - cloud_only_message_short: 'Available only for select CockroachDB Cloud clusters' - cloud_only_message: > - This version is currently available only for select - CockroachDB Cloud clusters. To request to upgrade - a CockroachDB self-hosted cluster to this version, - [contact support](https://support.cockroachlabs.com/hc/requests/new). - release_name: v24.3.3 major_version: v24.3 @@ -7855,13 +7841,6 @@ docker_arm_limited_access: false source: true previous_release: v24.3.2 - cloud_only: true - cloud_only_message_short: 'Available only for select CockroachDB Cloud clusters' - cloud_only_message: > - This version is currently available only for select - CockroachDB Cloud clusters. To request to upgrade - a CockroachDB self-hosted cluster to this version, - [contact support](https://support.cockroachlabs.com/hc/requests/new). - release_name: v23.2.19 major_version: v23.2 @@ -7889,10 +7868,31 @@ docker_arm_limited_access: false source: true previous_release: v23.2.18 - cloud_only: true - cloud_only_message_short: 'Available only for select CockroachDB Cloud clusters' - cloud_only_message: > - This version is currently available only for select - CockroachDB Cloud clusters. To request to upgrade - a CockroachDB self-hosted cluster to this version, - [contact support](https://support.cockroachlabs.com/hc/requests/new). + + +- release_name: v25.1.0-alpha.3 + major_version: v25.1 + release_date: '2025-01-15' + release_type: Testing + go_version: go1.22.8 + sha: 31e84cb3a57c52a779ff0982c95fb26646b54926 + has_sql_only: true + has_sha256sum: true + mac: + mac_arm: true + mac_arm_experimental: true + mac_arm_limited_access: false + windows: true + linux: + linux_arm: true + linux_arm_experimental: false + linux_arm_limited_access: false + linux_intel_fips: true + linux_arm_fips: false + docker: + docker_image: cockroachdb/cockroach-unstable + docker_arm: true + docker_arm_experimental: false + docker_arm_limited_access: false + source: true + previous_release: v25.1.0-alpha.2 diff --git a/src/current/_includes/releases/v25.1/v25.1.0-alpha.3.md b/src/current/_includes/releases/v25.1/v25.1.0-alpha.3.md new file mode 100644 index 00000000000..d31f0793583 --- /dev/null +++ b/src/current/_includes/releases/v25.1/v25.1.0-alpha.3.md @@ -0,0 +1,72 @@ +## v25.1.0-alpha.3 + +Release Date: January 15, 2025 + +{% include releases/new-release-downloads-docker-image.md release=include.release %} + +

Backward-incompatible changes

+ +- Several metrics are redundant and have been removed. The following list maps each removed metric to an existing, identical metric. [#138786][#138786] + - Removed `sql.schema_changer.running`, which is redundant with `jobs.schema_change.currently_running`. + - Removed `sql.schema_changer.successes`, which is redundant with `jobs.schema_change.resume_completed`. + - Removed `sql.schema_changer.retry_errors`, which is redundant with `jobs.schema_change.resume_retry_error`. + - Removed `sql.schema_changer.permanent_errors`, which is redundant with `jobs.schema_change.resume_failed`. + +

General changes

+ +- When changefeeds are created with a resolved option lower than the `min_checkpoint_frequency` option, a notice is now printed, since this may lead to unexpected behavior. [#138181][#138181] +- CockroachDB binaries are now built with profile-guided optimization (PGO) enabled. [#138700][#138700] + +

SQL language changes

+ +- Users can now always see and control (pause/resume/cancel) jobs that they own. [#138178][#138178] +- CockroachDB now provides different options for `CREATE LOGICALLY REPLICATED TABLE`: `UNIDIRECTIONAL` and `BIDIRECTIONAL ON`. These options are used for `CREATE LOGICALLY REPLICATED TABLE`, but not `CREATE LOGICAL REPLICATION STREAM`. [#138244][#138244] +- `CHANGEFEED`s using named external connections now automatically update their configuration when the connection configuration changes. [#138237][#138237] +- Added support for `DO` statements embedded within PL/pgSQL routines and other `DO` statements. `DO` statements execute a block of code inline as an anonymous function. Currently, only a PL/pgSQL body is allowed. [#138709][#138709] +- Added support for `DO` statements in SQL, which allow a PL/pgSQL code block to be executed inline. [#138709][#138709] + +

Operational changes

+ +- If a row-level TTL job is scheduled to run and the previous scheduled job for that table is still running, the scheduled run will now be skipped rather than waiting for the previous job to complete. [#138336][#138336] +- Schema object identifiers (e.g., database names, schema names, table names, and function names) are no longer redacted when logging statements in the `EXEC` or `SQL_SCHEMA` channels. If redaction of these names is required, then the new cluster setting `sql.log.redact_names.enabled` can be set to `true`. The default value of the setting is `false`. [#136897][#136897] +- Object identifiers such as table names, schema names, function names, and type names are no longer redacted in the `SQL_SCHEMA` log channel. [#136897][#136897] +- Changed the default value of the cluster setting `admission.l0_file_count_overload_threshold` to `4000`. [#138699][#138699] +- Introduced a metric, `sql.schema_changer.object_count`, that counts the number of schema objects in the cluster. [#138786][#138786] +- Renamed the `changefeed.min_highwater_advance` cluster setting to `changefeed.resolved_timestamp.min_update_interval` to more accurately reflect its function. Its description in the automatically generated documentation has also been updated. The previous name remains usable for backward compatibility. [#138673][#138673] + +

DB Console changes

+ +- Added a `/debug/pprof/fgprof` endpoint to capture off-CPU stack traces. Use of this endpoint will have a noticeable impact on performance while the endpoint is being triggered. [#138779][#138779] + +

Bug fixes

+ +- In the v2 **Databases > Table** page, the `CREATE` statement will now show up as expected for tables with custom schema names. [#138294][#138294] +- Fixed issues with the virtual index scan on `crdb_internal.create_type_statements`, ensuring consistent results when querying user-defined types (UDTs) across databases. [#138295][#138295] +- Queries that perform a cast from the string representation of an array containing geometry or geography types to a SQL array type will now succeed. [#138557][#138557] +- Fixed a bug that disregarded tuple labels in some cases. This could cause unexpected behavior, such as when converting a tuple to JSON with `to_jsonb`. See [#136167][#136167] for more details. This incorrect removal of tuple labels bug was introduced in v22.1.0, and changes in v24.3.0 made unexpected behavior due to the bug more likely. [#138791][#138791] + +- Fixed a bug where locks were taken on the system tables `system.users` and `system.role_options` even when `allow_role_memberships_to_change_during_transaction` was set. Now, users are able to create and drop users quickly when `allow_role_memberships_to_change_during_transaction` is set, even if there are contending transactions on `system.users` and `system.role_options`. [#137940][#137940] + +[#136167]: https://github.com/cockroachdb/cockroach/pull/136167 +[#136897]: https://github.com/cockroachdb/cockroach/pull/136897 +[#137750]: https://github.com/cockroachdb/cockroach/pull/137750 +[#137940]: https://github.com/cockroachdb/cockroach/pull/137940 +[#138178]: https://github.com/cockroachdb/cockroach/pull/138178 +[#138181]: https://github.com/cockroachdb/cockroach/pull/138181 +[#138223]: https://github.com/cockroachdb/cockroach/pull/138223 +[#138237]: https://github.com/cockroachdb/cockroach/pull/138237 +[#138244]: https://github.com/cockroachdb/cockroach/pull/138244 +[#138294]: https://github.com/cockroachdb/cockroach/pull/138294 +[#138295]: https://github.com/cockroachdb/cockroach/pull/138295 +[#138332]: https://github.com/cockroachdb/cockroach/pull/138332 +[#138334]: https://github.com/cockroachdb/cockroach/pull/138334 +[#138336]: https://github.com/cockroachdb/cockroach/pull/138336 +[#138343]: https://github.com/cockroachdb/cockroach/pull/138343 +[#138557]: https://github.com/cockroachdb/cockroach/pull/138557 +[#138673]: https://github.com/cockroachdb/cockroach/pull/138673 +[#138699]: https://github.com/cockroachdb/cockroach/pull/138699 +[#138700]: https://github.com/cockroachdb/cockroach/pull/138700 +[#138709]: https://github.com/cockroachdb/cockroach/pull/138709 +[#138779]: https://github.com/cockroachdb/cockroach/pull/138779 +[#138786]: https://github.com/cockroachdb/cockroach/pull/138786 +[#138791]: https://github.com/cockroachdb/cockroach/pull/138791 diff --git a/src/current/_includes/v23.2/known-limitations/generic-query-plan-limitations.md b/src/current/_includes/v23.2/known-limitations/generic-query-plan-limitations.md new file mode 100644 index 00000000000..e28e66d5f32 --- /dev/null +++ b/src/current/_includes/v23.2/known-limitations/generic-query-plan-limitations.md @@ -0,0 +1,2 @@ +- Because [generic query plans]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-cache) use lookup joins instead of the scans and revscans used by custom query plans, generic query plans do not perform as well as custom query plans in some cases. [#128916](https://github.com/cockroachdb/cockroach/issues/128916) +- [Generic query plans]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-type) are not included in the [plan cache]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-cache). This means a generic query plan built and optimized for a prepared statement in one session cannot be used by another session. To reuse generic query plans for maximum performance, a prepared statement should be executed multiple times instead of prepared and executed once. [#128911](https://github.com/cockroachdb/cockroach/issues/128911) \ No newline at end of file diff --git a/src/current/_includes/v23.2/misc/enterprise-features.md b/src/current/_includes/v23.2/misc/enterprise-features.md index 1dca40ae466..b07cbed978e 100644 --- a/src/current/_includes/v23.2/misc/enterprise-features.md +++ b/src/current/_includes/v23.2/misc/enterprise-features.md @@ -6,6 +6,7 @@ Feature | Description [Multi-Region Capabilities]({% link {{ page.version.version }}/multiregion-overview.md %}) | Row-level control over where your data is stored to help you reduce read and write latency and meet regulatory requirements. [PL/pgSQL]({% link {{ page.version.version }}/plpgsql.md %}) | Use a procedural language in [user-defined functions]({% link {{ page.version.version }}/user-defined-functions.md %}) and [stored procedures]({% link {{ page.version.version }}/stored-procedures.md %}) to improve performance and enable more complex queries. [Node Map]({% link {{ page.version.version }}/enable-node-map.md %}) | Visualize the geographical distribution of a cluster by plotting its node localities on a world map. +[Generic query plans]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-type) | Improve performance for prepared statements by enabling generic plans that eliminate most of the query latency attributed to planning. ## Recovery and streaming diff --git a/src/current/_includes/v23.2/misc/session-vars.md b/src/current/_includes/v23.2/misc/session-vars.md index 95bc51591d7..3aa7bc8224b 100644 --- a/src/current/_includes/v23.2/misc/session-vars.md +++ b/src/current/_includes/v23.2/misc/session-vars.md @@ -56,6 +56,7 @@ | `optimizer_use_multicol_stats` | If `on`, the optimizer uses collected multi-column statistics for cardinality estimation. | `on` | No | Yes | | `optimizer_use_not_visible_indexes` | If `on`, the optimizer uses not visible indexes for planning. | `off` | No | Yes | | `pg_trgm.similarity_threshold` | The threshold above which a [`%`]({% link {{ page.version.version }}/functions-and-operators.md %}#operators) string comparison returns `true`. The value must be between `0` and `1`. For more information, see [Trigram Indexes]({% link {{ page.version.version }}/trigram-indexes.md %}). | `0.3` | Yes | Yes | +| `plan_cache_mode` | The type of plan that is cached in the [query plan cache]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-cache): `auto`, `force_generic_plan`, or `force_custom_plan`.

For more information, refer to [Query plan type]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-type). | `force_custom_plan` | Yes | Yes | `prefer_lookup_joins_for_fks` | If `on`, the optimizer prefers [`lookup joins`]({% link {{ page.version.version }}/joins.md %}#lookup-joins) to [`merge joins`]({% link {{ page.version.version }}/joins.md %}#merge-joins) when performing [`foreign key`]({% link {{ page.version.version }}/foreign-key.md %}) checks. | `off` | Yes | Yes | | `reorder_joins_limit` | Maximum number of joins that the optimizer will attempt to reorder when searching for an optimal query execution plan.

For more information, see [Join reordering]({% link {{ page.version.version }}/cost-based-optimizer.md %}#join-reordering). | `8` | Yes | Yes | | `require_explicit_primary_keys` | If `on`, CockroachDB throws an error for all tables created without an explicit primary key defined. | `off` | Yes | Yes | diff --git a/src/current/_includes/v24.1/known-limitations/generic-query-plan-limitations.md b/src/current/_includes/v24.1/known-limitations/generic-query-plan-limitations.md new file mode 100644 index 00000000000..e28e66d5f32 --- /dev/null +++ b/src/current/_includes/v24.1/known-limitations/generic-query-plan-limitations.md @@ -0,0 +1,2 @@ +- Because [generic query plans]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-cache) use lookup joins instead of the scans and revscans used by custom query plans, generic query plans do not perform as well as custom query plans in some cases. [#128916](https://github.com/cockroachdb/cockroach/issues/128916) +- [Generic query plans]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-type) are not included in the [plan cache]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-cache). This means a generic query plan built and optimized for a prepared statement in one session cannot be used by another session. To reuse generic query plans for maximum performance, a prepared statement should be executed multiple times instead of prepared and executed once. [#128911](https://github.com/cockroachdb/cockroach/issues/128911) \ No newline at end of file diff --git a/src/current/_includes/v24.1/misc/enterprise-features.md b/src/current/_includes/v24.1/misc/enterprise-features.md index ee8f6eac8fe..0b9cef07fd8 100644 --- a/src/current/_includes/v24.1/misc/enterprise-features.md +++ b/src/current/_includes/v24.1/misc/enterprise-features.md @@ -7,6 +7,7 @@ Feature | Description [Multi-Region Capabilities]({% link {{ page.version.version }}/multiregion-overview.md %}) | Row-level control over where your data is stored to help you reduce read and write latency and meet regulatory requirements. [PL/pgSQL]({% link {{ page.version.version }}/plpgsql.md %}) | Use a procedural language in [user-defined functions]({% link {{ page.version.version }}/user-defined-functions.md %}) and [stored procedures]({% link {{ page.version.version }}/stored-procedures.md %}) to improve performance and enable more complex queries. [Node Map]({% link {{ page.version.version }}/enable-node-map.md %}) | Visualize the geographical distribution of a cluster by plotting its node localities on a world map. +[Generic query plans]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-type) | Improve performance for prepared statements by enabling generic plans that eliminate most of the query latency attributed to planning. ## Recovery and streaming diff --git a/src/current/_includes/v24.1/misc/session-vars.md b/src/current/_includes/v24.1/misc/session-vars.md index 2d93817e285..ad38e3247bc 100644 --- a/src/current/_includes/v24.1/misc/session-vars.md +++ b/src/current/_includes/v24.1/misc/session-vars.md @@ -56,9 +56,10 @@ | `optimizer_use_lock_op_for_serializable` | If `on`, the optimizer uses a `Lock` operator to construct query plans for `SELECT` statements using the [`FOR UPDATE` and `FOR SHARE`]({% link {{ page.version.version }}/select-for-update.md %}) clauses. This setting only affects `SERIALIZABLE` transactions. `READ COMMITTED` transactions are evaluated with the `Lock` operator regardless of the setting. | `off` | Yes | Yes | | `optimizer_use_multicol_stats` | If `on`, the optimizer uses collected multi-column statistics for cardinality estimation. | `on` | No | Yes | | `optimizer_use_not_visible_indexes` | If `on`, the optimizer uses not visible indexes for planning. | `off` | No | Yes | -| `optimizer_use_virtual_computed_column_stats` | If `on`, the optimizer uses table statistics on [virtual computed columns]({% link {{ page.version.version }}/computed-columns.md %}#virtual-computed-columns). | `on` | Yes | Yes | -| `plpgsql_use_strict_into` | If `on`, PL/pgSQL [`SELECT ... INTO` and `RETURNING ... INTO` statements]({% link {{ page.version.version }}/plpgsql.md %}#assign-a-result-to-a-variable) behave as though the `STRICT` option is specified. This causes the SQL statement to error if it does not return exactly one row. | `off` | Yes | Yes | +| `optimizer_use_virtual_computed_column_stats` | If `on`, the optimizer uses table statistics on [virtual computed columns]({% link {{ page.version.version }}/computed-columns.md %}#virtual-computed-columns). | `on` | Yes | Yes | `pg_trgm.similarity_threshold` | The threshold above which a [`%`]({% link {{ page.version.version }}/functions-and-operators.md %}#operators) string comparison returns `true`. The value must be between `0` and `1`. For more information, see [Trigram Indexes]({% link {{ page.version.version }}/trigram-indexes.md %}). | `0.3` | Yes | Yes | +| `plan_cache_mode` | The type of plan that is cached in the [query plan cache]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-cache): `auto`, `force_generic_plan`, or `force_custom_plan`.

For more information, refer to [Query plan type]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-type). | `force_custom_plan` | Yes | Yes +| `plpgsql_use_strict_into` | If `on`, PL/pgSQL [`SELECT ... INTO` and `RETURNING ... INTO` statements]({% link {{ page.version.version }}/plpgsql.md %}#assign-a-result-to-a-variable) behave as though the `STRICT` option is specified. This causes the SQL statement to error if it does not return exactly one row. | `off` | Yes | Yes | | `prefer_lookup_joins_for_fks` | If `on`, the optimizer prefers [`lookup joins`]({% link {{ page.version.version }}/joins.md %}#lookup-joins) to [`merge joins`]({% link {{ page.version.version }}/joins.md %}#merge-joins) when performing [`foreign key`]({% link {{ page.version.version }}/foreign-key.md %}) checks. | `off` | Yes | Yes | | `reorder_joins_limit` | Maximum number of joins that the optimizer will attempt to reorder when searching for an optimal query execution plan.

For more information, see [Join reordering]({% link {{ page.version.version }}/cost-based-optimizer.md %}#join-reordering). | `8` | Yes | Yes | | `require_explicit_primary_keys` | If `on`, CockroachDB throws an error for all tables created without an explicit primary key defined. | `off` | Yes | Yes | diff --git a/src/current/cockroachcloud/cloud-api.md b/src/current/cockroachcloud/cloud-api.md index 2d443fd1a96..aa3490acb53 100644 --- a/src/current/cockroachcloud/cloud-api.md +++ b/src/current/cockroachcloud/cloud-api.md @@ -8,7 +8,7 @@ cloud: true The Cloud API is a [REST interface](https://wikipedia.org/wiki/Representational_state_transfer) that allows you programmatic access to manage the lifecycle of clusters within your organization. -This document pertains to the `latest` version of the API's `v1` endpoints, `2024-09-16`. For more detailed coverage of API endpoints for this version and prior verisons, refer to the [API reference documentation](https://www.cockroachlabs.com/docs/api/cloud/v1). +This document pertains to the `latest` version of the API's `v1` endpoints, `2024-09-16`. For more detailed coverage of API endpoints for this version and prior versions, refer to the [API reference documentation](https://www.cockroachlabs.com/docs/api/cloud/v1). To manage clusters and other resources in CockroachDB Cloud, you can also use the [CockroachDB Cloud Terraform provider]({% link cockroachcloud/provision-a-cluster-with-terraform.md %}), which implements the API. @@ -799,6 +799,7 @@ If the request was successful, the client will receive a list of SQL users. ~~~ Where: + - `` is the SQL username of the user. - `` is the token to use for retrieving the next page of results, if any. @@ -863,6 +864,7 @@ curl --request DELETE \ ~~~ Where: + - `{cluster_id}` is the unique ID of this cluster. {{site.data.alerts.callout_info}} The cluster ID used in the Cloud API is different from the routing ID used when [connecting to clusters]({% link cockroachcloud/connect-to-your-cluster.md %}). @@ -899,6 +901,7 @@ curl --request PUT \ ~~~ Where: + - `{cluster_id}` is the unique ID of this cluster. {{site.data.alerts.callout_info}} The cluster ID used in the Cloud API is different from the routing ID used when [connecting to clusters]({% link cockroachcloud/connect-to-your-cluster.md %}). diff --git a/src/current/images/v24.3/wal-failover-behavior.png b/src/current/images/v24.3/wal-failover-behavior.png new file mode 100644 index 00000000000..d6eb3b295f0 Binary files /dev/null and b/src/current/images/v24.3/wal-failover-behavior.png differ diff --git a/src/current/images/v24.3/wal-failover-overview.png b/src/current/images/v24.3/wal-failover-overview.png new file mode 100644 index 00000000000..b737986463a Binary files /dev/null and b/src/current/images/v24.3/wal-failover-overview.png differ diff --git a/src/current/images/v25.1/wal-failover-behavior.png b/src/current/images/v25.1/wal-failover-behavior.png new file mode 100644 index 00000000000..d6eb3b295f0 Binary files /dev/null and b/src/current/images/v25.1/wal-failover-behavior.png differ diff --git a/src/current/images/v25.1/wal-failover-overview.png b/src/current/images/v25.1/wal-failover-overview.png new file mode 100644 index 00000000000..b737986463a Binary files /dev/null and b/src/current/images/v25.1/wal-failover-overview.png differ diff --git a/src/current/jekyll-algolia-dev/lib/jekyll/algolia/indexer.rb b/src/current/jekyll-algolia-dev/lib/jekyll/algolia/indexer.rb index a09ba7741f0..fc6ddf547c7 100644 --- a/src/current/jekyll-algolia-dev/lib/jekyll/algolia/indexer.rb +++ b/src/current/jekyll-algolia-dev/lib/jekyll/algolia/indexer.rb @@ -357,6 +357,24 @@ def self.update_synonyms synonyms: ['materialized view', 'material view'] }, false) + index.save_synonym('session setting', { + objectID: 'session setting', + type: 'synonym', + synonyms: ['session setting', 'session variable', 'session settings', 'session variables', 'session var', 'session vars'] + }, false) + + index.save_synonym('cluster setting', { + objectID: 'cluster setting', + type: 'synonym', + synonyms: ['cluster setting', 'cluster variable', 'cluster settings', 'cluster variables', 'cluster var', 'cluster vars'] + }, false) + + index.save_synonym('molt', { + objectID: 'molt', + type: 'synonym', + synonyms: ['molt', 'migrate off legacy technology'] + }, false) + return end diff --git a/src/current/releases/index.md b/src/current/releases/index.md index f7dd618a343..5054fbb77a3 100644 --- a/src/current/releases/index.md +++ b/src/current/releases/index.md @@ -110,12 +110,13 @@ As of 2024, CockroachDB is released under a staged delivery process. New release ### Recent releases -| Version | Release Type | GA date | Latest patch release | -| :---: | :---: | :---: | :---: | -| [v24.2](#v24-2) | Innovation | 2024-08-12 | v24.2.4 | -| [v24.1](#v24-1) | Regular | 2024-05-20 | v24.1.6 (LTS) | -| [v23.2](#v23-2) | Regular | 2024-02-05 | v23.2.13 (LTS) | -| [v23.1](#v23-1) | Regular | 2023-05-15 | v23.1.28 (LTS) | +| Version | Release Type | GA date | +| :---: | :---: | :---: | +| [v24.3](#v24-3) | Regular | 2024-11-18 | +| [v24.2](#v24-2) | Innovation | 2024-08-12 | +| [v24.1](#v24-1) | Regular | 2024-05-20 | +| [v23.2](#v23-2) | Regular | 2024-02-05 | +| [v23.1](#v23-1) | Regular | 2023-05-15 | ### Upcoming releases @@ -123,7 +124,6 @@ The following releases and their descriptions represent proposed plans that are | Version | Release Type | Expected GA date | | :---: | :---: | :---: | -| v24.3 | Regular | 2024-11-18 | | v25.1 | Innovation | 2025 Q1 | | v25.2 | Regular | 2025 Q2 | | v25.3 | Innovation | 2025 Q3 | diff --git a/src/current/v23.2/cockroachdb-feature-availability.md b/src/current/v23.2/cockroachdb-feature-availability.md index c7209230d49..db17aa1709c 100644 --- a/src/current/v23.2/cockroachdb-feature-availability.md +++ b/src/current/v23.2/cockroachdb-feature-availability.md @@ -43,6 +43,10 @@ Any feature made available in a phase prior to GA is provided without any warran **The following features are in preview** and are subject to change. To share feedback and/or issues, contact [Support](https://support.cockroachlabs.com/hc). {{site.data.alerts.end}} +### Generic query plans + +[Generic query plans]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-type) are generated and optimized once without considering specific placeholder values, and are not regenerated on subsequent executions, unless the plan becomes stale due to [schema changes]({% link {{ page.version.version }}/online-schema-changes.md %}) or new [table statistics]({% link {{ page.version.version }}/cost-based-optimizer.md %}#table-statistics) and must be re-optimized. This approach eliminates most of the query latency attributed to planning. + ### CockroachDB Cloud Folders [Organizing CockroachDB {{ site.data.products.cloud }} clusters using folders]({% link cockroachcloud/folders.md %}) is in preview. Folders allow you to organize and manage access to your clusters according to your organization's requirements. For example, you can create top-level folders for each business unit in your organization, and within those folders, organize clusters by geographic location and then by level of maturity, such as production, staging, and testing. diff --git a/src/current/v23.2/cost-based-optimizer.md b/src/current/v23.2/cost-based-optimizer.md index b8af245f5a2..294d9e75f2a 100644 --- a/src/current/v23.2/cost-based-optimizer.md +++ b/src/current/v23.2/cost-based-optimizer.md @@ -277,27 +277,87 @@ Only tables with `ZONE` [survivability]({% link {{ page.version.version }}/multi ## Query plan cache -CockroachDB uses a cache for the query plans generated by the optimizer. This can lead to faster query execution since the database can reuse a query plan that was previously calculated, rather than computing a new plan each time a query is executed. +CockroachDB caches some of the query plans generated by the optimizer. The query plan cache is used for the following types of statements: + +- Prepared statements. +- Non-prepared statements using identical constant values. + +Caching query plans leads to faster query execution: rather than generating a new plan each time a query is executed, CockroachDB reuses a query plan that was previously generated. The query plan cache is enabled by default. To disable it, execute the following statement: {% include_cached copy-clipboard.html %} ~~~ sql -> SET CLUSTER SETTING sql.query_cache.enabled = false; +SET CLUSTER SETTING sql.query_cache.enabled = false; ~~~ -Only the following statements use the plan cache: +The following statements can use the plan cache: [`SELECT`]({% link {{ page.version.version }}/select-clause.md %}), [`INSERT`]({% link {{ page.version.version }}/insert.md %}), [`UPDATE`]({% link {{ page.version.version }}/update.md %}), [`UPSERT`]({% link {{ page.version.version }}/upsert.md %}), and [`DELETE`]({% link {{ page.version.version }}/delete.md %}). -- [`SELECT`]({% link {{ page.version.version }}/select-clause.md %}) -- [`INSERT`]({% link {{ page.version.version }}/insert.md %}) -- [`UPDATE`]({% link {{ page.version.version }}/update.md %}) -- [`UPSERT`]({% link {{ page.version.version }}/upsert.md %}) -- [`DELETE`]({% link {{ page.version.version }}/delete.md %}) +Two types of plans can be cached: custom and generic. Refer to [Query plan type](#query-plan-type). -The optimizer can use cached plans if they are: +### Query plan type -- Prepared statements. -- Non-prepared statements using identical constant values. +Two types of plans can be cached: + +- *Custom* query plans are generated for a given query structure and optimized for specific placeholder values, and are re-optimized on subsequent executions. By default, the optimizer uses custom plans. Custom plans are included in the [plan cache](#query-plan-cache). +- {% include_cached new-in.html version="v23.2.12" %} *Generic* query plans are generated and optimized once without considering specific placeholder values, and are **not** regenerated on subsequent executions, unless the plan becomes stale due to [schema changes]({% link {{ page.version.version }}/online-schema-changes.md %}) or new [table statistics](#table-statistics) and must be re-optimized. This approach eliminates most of the query latency attributed to planning. For example: + + ~~~ sql + PREPARE p AS SELECT ...; + SET plan_cache_mode = force_generic_plan; + EXECUTE p; -- The query plan is generated and optimized. + EXECUTE p; -- The query plan is reused without re-optimizing. + EXECUTE p; -- The query plan is reused without re-optimizing. + ~~~ + + Generic plans are **not** included in the plan cache, but are cached per session. This means that they must still be re-optimized each time a session prepares a statement using a generic plan. To reuse generic query plans for maximum performance, a prepared statement should be executed multiple times instead of prepared and executed once. + + This feature is in [preview]({% link {{ page.version.version }}/cockroachdb-feature-availability.md %}) and is subject to change. + + {{site.data.alerts.callout_success}} + Generic query plans will only benefit workloads that use prepared statements, which are issued via explicit `PREPARE` statements or by client libraries using the [PostgreSQL extended wire protocol](https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-FLOW-EXT-QUERY). Generic query plans are most beneficial for queries with high planning times, such as queries with many [joins]({% link {{ page.version.version }}/joins.md %}). For more information on reducing planning time for such queries, refer to [Reduce planning time for queries with many joins](#reduce-planning-time-for-queries-with-many-joins). + {{site.data.alerts.end}} + +To change the type of plan that is cached, use the [`plan_cache_mode`]({% link {{ page.version.version }}/session-variables.md %}#plan-cache-mode) session setting. This setting applies when a statement is executed, not when it is prepared. Statements are therefore not associated with a specific query plan type when they are prepared. + +The following modes can be set: + +- `force_custom_plan` (default): Force the use of custom plans. +- `force_generic_plan`: Force the use of generic plans. +- `auto`: Automatically determine whether to use custom or generic query plans for prepared statements. Custom plans are used for the first five statement executions. Subsequent executions use a generic plan if its estimated cost is not significantly higher than the average cost of the preceding custom plans. + +{{site.data.alerts.callout_info}} +Generic plans are always used for non-prepared statements that do not contain placeholders or [stable functions]({% link {{ page.version.version }}/functions-and-operators.md %}#function-volatility), regardless of the `plan_cache_mode` setting. +{{site.data.alerts.end}} + +In some cases, generic query plans are less efficient than custom plans. For this reason, Cockroach Labs recommends setting `plan_cache_mode` to `auto` instead of `force_generic_plan`. Under the `auto` setting, the optimizer avoids bad generic plans by falling back to custom plans. For example: + +Set `plan_cache_mode` to `auto` at the session level: + +{% include_cached copy-clipboard.html %} +~~~ sql +SET plan_cache_mode = auto +~~~ + +At the [database level]({% link {{ page.version.version }}/alter-database.md %}#set-session-variable): + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER DATABASE db SET plan_cache_mode = auto; +~~~ + +At the [role level]({% link {{ page.version.version }}/alter-role.md %}#set-default-session-variable-values-for-a-role): + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER ROLE db_user SET plan_cache_mode = auto; +~~~ + +To verify the plan type used by a query, check the [`EXPLAIN ANALYZE`]({% link {{ page.version.version }}/explain-analyze.md %}) output for the query. + +- If a generic query plan is optimized for the current execution, the `plan type` in the output is `generic, re-optimized`. +- If a generic query plan is reused for the current execution without performing optimization, the `plan type` in the output is `generic, reused`. +- If a custom query plan is used for the current execution, the `plan type` in the output is `custom`. ## Join reordering @@ -309,7 +369,7 @@ To change this setting, which is controlled by the `reorder_joins_limit` [sessio {% include_cached copy-clipboard.html %} ~~~ sql -> SET reorder_joins_limit = 0; +SET reorder_joins_limit = 0; ~~~ To disable this feature, set the variable to `0`. You can configure the default `reorder_joins_limit` session setting with the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `sql.defaults.reorder_joins_limit`, which has a default value of `8`. @@ -328,6 +388,7 @@ The cost-based optimizer explores multiple join orderings to find the lowest-cos - To limit the size of the subtree that can be reordered, set the `reorder_joins_limit` [session variable]({% link {{ page.version.version }}/set-vars.md %}) to a lower value, for example: + {% include_cached copy-clipboard.html %} ~~~ sql SET reorder_joins_limit = 2; ~~~ diff --git a/src/current/v23.2/explain-analyze.md b/src/current/v23.2/explain-analyze.md index b6c99e4a66f..aa99643f59b 100644 --- a/src/current/v23.2/explain-analyze.md +++ b/src/current/v23.2/explain-analyze.md @@ -212,6 +212,7 @@ EXPLAIN ANALYZE SELECT city, AVG(revenue) FROM rides GROUP BY city; execution time: 8ms distribution: full vectorized: true + plan type: custom rows decoded from KV: 500 (88 KiB, 1 gRPC calls) cumulative time spent in KV: 6ms maximum memory usage: 240 KiB @@ -262,6 +263,7 @@ EXPLAIN ANALYZE SELECT * FROM vehicles JOIN rides ON rides.vehicle_id = vehicles execution time: 5ms distribution: local vectorized: true + plan type: custom rows decoded from KV: 515 (90 KiB, 2 gRPC calls) cumulative time spent in KV: 4ms maximum memory usage: 580 KiB @@ -335,6 +337,7 @@ EXPLAIN ANALYZE (VERBOSE) SELECT city, AVG(revenue) FROM rides GROUP BY city; execution time: 5ms distribution: full vectorized: true + plan type: custom rows decoded from KV: 500 (88 KiB, 500 KVs, 1 gRPC calls) cumulative time spent in KV: 4ms maximum memory usage: 240 KiB @@ -397,6 +400,7 @@ EXPLAIN ANALYZE (DISTSQL) SELECT city, AVG(revenue) FROM rides GROUP BY city; execution time: 4ms distribution: full vectorized: true + plan type: custom rows decoded from KV: 500 (88 KiB, 1 gRPC calls) cumulative time spent in KV: 3ms maximum memory usage: 240 KiB @@ -475,6 +479,7 @@ EXPLAIN ANALYZE (REDACT) SELECT * FROM rides WHERE revenue > 90 ORDER BY revenue execution time: 6ms distribution: full vectorized: true + plan type: custom rows decoded from KV: 500 (88 KiB, 1 gRPC calls) cumulative time spent in KV: 4ms maximum memory usage: 280 KiB diff --git a/src/current/v23.2/known-limitations.md b/src/current/v23.2/known-limitations.md index 0f8f62c8885..217f7c03e9e 100644 --- a/src/current/v23.2/known-limitations.md +++ b/src/current/v23.2/known-limitations.md @@ -76,6 +76,10 @@ This page describes newly identified limitations in the CockroachDB {{page.relea {% include {{ page.version.version }}/known-limitations/physical-cluster-replication.md %} - {% include {{ page.version.version }}/known-limitations/pcr-scheduled-changefeeds.md %} +### Generic query plans + +{% include {{ page.version.version }}/known-limitations/generic-query-plan-limitations.md %} + ## Unresolved limitations ### Limitations for user-defined functions (UDFs) diff --git a/src/current/v23.2/migrate-from-avro.md b/src/current/v23.2/migrate-from-avro.md index 8e161b48acd..fb47d9b6868 100644 --- a/src/current/v23.2/migrate-from-avro.md +++ b/src/current/v23.2/migrate-from-avro.md @@ -163,7 +163,7 @@ There are additional import [options][option] you can use when importing binary - `records_terminated_by`, which specifies the unicode character used to indicate new lines in the input binary or JSON file (default: `\n`). {{site.data.alerts.callout_info}} -The following example uses sample data generated by [Avro tools](https://github.com/cockroachdb/cockroach/tree/master/pkg/ccl/importccl/testdata/avro). +The following example uses sample data generated by [Avro tools](https://github.com/cockroachdb/cockroach/tree/master/pkg/sql/importer/testdata/avro). {{site.data.alerts.end}} For example, to import the data from `simple-schema.json` into a `simple` table, first [create the table]({% link {{ page.version.version }}/create-table.md %}) to import into. Then run `IMPORT INTO` with the following options: diff --git a/src/current/v24.1/cockroachdb-feature-availability.md b/src/current/v24.1/cockroachdb-feature-availability.md index 7ed53ccab27..db6ae600630 100644 --- a/src/current/v24.1/cockroachdb-feature-availability.md +++ b/src/current/v24.1/cockroachdb-feature-availability.md @@ -43,6 +43,10 @@ Any feature made available in a phase prior to GA is provided without any warran **The following features are in preview** and are subject to change. To share feedback and/or issues, contact [Support](https://support.cockroachlabs.com/hc). {{site.data.alerts.end}} +### Generic query plans + +[Generic query plans]({% link {{ page.version.version }}/cost-based-optimizer.md %}#query-plan-type) are generated and optimized once without considering specific placeholder values, and are not regenerated on subsequent executions, unless the plan becomes stale due to [schema changes]({% link {{ page.version.version }}/online-schema-changes.md %}) or new [table statistics]({% link {{ page.version.version }}/cost-based-optimizer.md %}#table-statistics) and must be re-optimized. This approach eliminates most of the query latency attributed to planning. + ### CockroachDB Cloud Folders [Organizing CockroachDB {{ site.data.products.cloud }} clusters using folders]({% link cockroachcloud/folders.md %}) is in preview. Folders allow you to organize and manage access to your clusters according to your organization's requirements. For example, you can create top-level folders for each business unit in your organization, and within those folders, organize clusters by geographic location and then by level of maturity, such as production, staging, and testing. diff --git a/src/current/v24.1/cost-based-optimizer.md b/src/current/v24.1/cost-based-optimizer.md index fdfd805a323..f680fdb8a80 100644 --- a/src/current/v24.1/cost-based-optimizer.md +++ b/src/current/v24.1/cost-based-optimizer.md @@ -277,27 +277,87 @@ Only tables with `ZONE` [survivability]({% link {{ page.version.version }}/multi ## Query plan cache -CockroachDB uses a cache for the query plans generated by the optimizer. This can lead to faster query execution since the database can reuse a query plan that was previously calculated, rather than computing a new plan each time a query is executed. +CockroachDB caches some of the query plans generated by the optimizer. The query plan cache is used for the following types of statements: + +- Prepared statements. +- Non-prepared statements using identical constant values. + +Caching query plans leads to faster query execution: rather than generating a new plan each time a query is executed, CockroachDB reuses a query plan that was previously generated. The query plan cache is enabled by default. To disable it, execute the following statement: {% include_cached copy-clipboard.html %} ~~~ sql -> SET CLUSTER SETTING sql.query_cache.enabled = false; +SET CLUSTER SETTING sql.query_cache.enabled = false; ~~~ -Only the following statements use the plan cache: +The following statements can use the plan cache: [`SELECT`]({% link {{ page.version.version }}/select-clause.md %}), [`INSERT`]({% link {{ page.version.version }}/insert.md %}), [`UPDATE`]({% link {{ page.version.version }}/update.md %}), [`UPSERT`]({% link {{ page.version.version }}/upsert.md %}), and [`DELETE`]({% link {{ page.version.version }}/delete.md %}). -- [`SELECT`]({% link {{ page.version.version }}/select-clause.md %}) -- [`INSERT`]({% link {{ page.version.version }}/insert.md %}) -- [`UPDATE`]({% link {{ page.version.version }}/update.md %}) -- [`UPSERT`]({% link {{ page.version.version }}/upsert.md %}) -- [`DELETE`]({% link {{ page.version.version }}/delete.md %}) +Two types of plans can be cached: custom and generic. Refer to [Query plan type](#query-plan-type). -The optimizer can use cached plans if they are: +### Query plan type -- Prepared statements. -- Non-prepared statements using identical constant values. +Two types of plans can be cached: + +- *Custom* query plans are generated for a given query structure and optimized for specific placeholder values, and are re-optimized on subsequent executions. By default, the optimizer uses custom plans. Custom plans are included in the [plan cache](#query-plan-cache). +- {% include_cached new-in.html version="v24.1.5" %}*Generic* query plans are generated and optimized once without considering specific placeholder values, and are **not** regenerated on subsequent executions, unless the plan becomes stale due to [schema changes]({% link {{ page.version.version }}/online-schema-changes.md %}) or new [table statistics](#table-statistics) and must be re-optimized. This approach eliminates most of the query latency attributed to planning. For example: + + ~~~ sql + PREPARE p AS SELECT ...; + SET plan_cache_mode = force_generic_plan; + EXECUTE p; -- The query plan is generated and optimized. + EXECUTE p; -- The query plan is reused without re-optimizing. + EXECUTE p; -- The query plan is reused without re-optimizing. + ~~~ + + Generic plans are **not** included in the plan cache, but are cached per session. This means that they must still be re-optimized each time a session prepares a statement using a generic plan. To reuse generic query plans for maximum performance, a prepared statement should be executed multiple times instead of prepared and executed once. + + This feature is in [preview]({% link {{ page.version.version }}/cockroachdb-feature-availability.md %}) and is subject to change. + + {{site.data.alerts.callout_success}} + Generic query plans will only benefit workloads that use prepared statements, which are issued via explicit `PREPARE` statements or by client libraries using the [PostgreSQL extended wire protocol](https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-FLOW-EXT-QUERY). Generic query plans are most beneficial for queries with high planning times, such as queries with many [joins]({% link {{ page.version.version }}/joins.md %}). For more information on reducing planning time for such queries, refer to [Reduce planning time for queries with many joins](#reduce-planning-time-for-queries-with-many-joins). + {{site.data.alerts.end}} + +To change the type of plan that is cached, use the [`plan_cache_mode`]({% link {{ page.version.version }}/session-variables.md %}#plan-cache-mode) session setting. This setting applies when a statement is executed, not when it is prepared. Statements are therefore not associated with a specific query plan type when they are prepared. + +The following modes can be set: + +- `force_custom_plan` (default): Force the use of custom plans. +- `force_generic_plan`: Force the use of generic plans. +- `auto`: Automatically determine whether to use custom or generic query plans for prepared statements. Custom plans are used for the first five statement executions. Subsequent executions use a generic plan if its estimated cost is not significantly higher than the average cost of the preceding custom plans. + +{{site.data.alerts.callout_info}} +Generic plans are always used for non-prepared statements that do not contain placeholders or [stable functions]({% link {{ page.version.version }}/functions-and-operators.md %}#function-volatility), regardless of the `plan_cache_mode` setting. +{{site.data.alerts.end}} + +In some cases, generic query plans are less efficient than custom plans. For this reason, Cockroach Labs recommends setting `plan_cache_mode` to `auto` instead of `force_generic_plan`. Under the `auto` setting, the optimizer avoids bad generic plans by falling back to custom plans. For example: + +Set `plan_cache_mode` to `auto` at the session level: + +{% include_cached copy-clipboard.html %} +~~~ sql +SET plan_cache_mode = auto +~~~ + +At the [database level]({% link {{ page.version.version }}/alter-database.md %}#set-session-variable): + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER DATABASE db SET plan_cache_mode = auto; +~~~ + +At the [role level]({% link {{ page.version.version }}/alter-role.md %}#set-default-session-variable-values-for-a-role): + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER ROLE db_user SET plan_cache_mode = auto; +~~~ + +To verify the plan type used by a query, check the [`EXPLAIN ANALYZE`]({% link {{ page.version.version }}/explain-analyze.md %}) output for the query. + +- If a generic query plan is optimized for the current execution, the `plan type` in the output is `generic, re-optimized`. +- If a generic query plan is reused for the current execution without performing optimization, the `plan type` in the output is `generic, reused`. +- If a custom query plan is used for the current execution, the `plan type` in the output is `custom`. ## Join reordering @@ -309,7 +369,7 @@ To change this setting, which is controlled by the `reorder_joins_limit` [sessio {% include_cached copy-clipboard.html %} ~~~ sql -> SET reorder_joins_limit = 0; +SET reorder_joins_limit = 0; ~~~ To disable this feature, set the variable to `0`. You can configure the default `reorder_joins_limit` session setting with the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `sql.defaults.reorder_joins_limit`, which has a default value of `8`. @@ -328,6 +388,7 @@ The cost-based optimizer explores multiple join orderings to find the lowest-cos - To limit the size of the subtree that can be reordered, set the `reorder_joins_limit` [session variable]({% link {{ page.version.version }}/set-vars.md %}) to a lower value, for example: + {% include_cached copy-clipboard.html %} ~~~ sql SET reorder_joins_limit = 2; ~~~ diff --git a/src/current/v24.1/explain-analyze.md b/src/current/v24.1/explain-analyze.md index 1d1b5ff6a54..0e27a1e4c29 100644 --- a/src/current/v24.1/explain-analyze.md +++ b/src/current/v24.1/explain-analyze.md @@ -212,6 +212,7 @@ EXPLAIN ANALYZE SELECT city, AVG(revenue) FROM rides GROUP BY city; execution time: 8ms distribution: full vectorized: true + plan type: custom rows decoded from KV: 500 (88 KiB, 1 gRPC calls) cumulative time spent in KV: 6ms maximum memory usage: 240 KiB @@ -262,6 +263,7 @@ EXPLAIN ANALYZE SELECT * FROM vehicles JOIN rides ON rides.vehicle_id = vehicles execution time: 5ms distribution: local vectorized: true + plan type: custom rows decoded from KV: 515 (90 KiB, 2 gRPC calls) cumulative time spent in KV: 4ms maximum memory usage: 580 KiB @@ -335,6 +337,7 @@ EXPLAIN ANALYZE (VERBOSE) SELECT city, AVG(revenue) FROM rides GROUP BY city; execution time: 5ms distribution: full vectorized: true + plan type: custom rows decoded from KV: 500 (88 KiB, 500 KVs, 1 gRPC calls) cumulative time spent in KV: 4ms maximum memory usage: 240 KiB @@ -397,6 +400,7 @@ EXPLAIN ANALYZE (DISTSQL) SELECT city, AVG(revenue) FROM rides GROUP BY city; execution time: 4ms distribution: full vectorized: true + plan type: custom rows decoded from KV: 500 (88 KiB, 1 gRPC calls) cumulative time spent in KV: 3ms maximum memory usage: 240 KiB @@ -475,6 +479,7 @@ EXPLAIN ANALYZE (REDACT) SELECT * FROM rides WHERE revenue > 90 ORDER BY revenue execution time: 6ms distribution: full vectorized: true + plan type: custom rows decoded from KV: 500 (88 KiB, 1 gRPC calls) cumulative time spent in KV: 4ms maximum memory usage: 280 KiB diff --git a/src/current/v24.1/known-limitations.md b/src/current/v24.1/known-limitations.md index d3cae5543d9..4805bd5b937 100644 --- a/src/current/v24.1/known-limitations.md +++ b/src/current/v24.1/known-limitations.md @@ -35,6 +35,10 @@ Limitations will be added as they are discovered. {% include {{ page.version.version }}/known-limitations/fast-cutback-latest-timestamp.md %} +### Generic query plans + +{% include {{ page.version.version }}/known-limitations/generic-query-plan-limitations.md %} + ## Limitations from {{ previous_version }} and earlier This section describes limitations from previous CockroachDB versions that still impact {{ page.version.version }}. diff --git a/src/current/v24.1/migrate-from-avro.md b/src/current/v24.1/migrate-from-avro.md index 8557ce40cea..39b7bdbc9aa 100644 --- a/src/current/v24.1/migrate-from-avro.md +++ b/src/current/v24.1/migrate-from-avro.md @@ -161,7 +161,7 @@ There are additional import [options][option] you can use when importing binary - `records_terminated_by`, which specifies the unicode character used to indicate new lines in the input binary or JSON file (default: `\n`). {{site.data.alerts.callout_info}} -The following example uses sample data generated by [Avro tools](https://github.com/cockroachdb/cockroach/tree/master/pkg/ccl/importccl/testdata/avro). +The following example uses sample data generated by [Avro tools](https://github.com/cockroachdb/cockroach/tree/master/pkg/sql/importer/testdata/avro). {{site.data.alerts.end}} For example, to import the data from `simple-schema.json` into a `simple` table, first [create the table]({% link {{ page.version.version }}/create-table.md %}) to import into. Then run `IMPORT INTO` with the following options: diff --git a/src/current/v24.2/migrate-from-avro.md b/src/current/v24.2/migrate-from-avro.md index 8557ce40cea..39b7bdbc9aa 100644 --- a/src/current/v24.2/migrate-from-avro.md +++ b/src/current/v24.2/migrate-from-avro.md @@ -161,7 +161,7 @@ There are additional import [options][option] you can use when importing binary - `records_terminated_by`, which specifies the unicode character used to indicate new lines in the input binary or JSON file (default: `\n`). {{site.data.alerts.callout_info}} -The following example uses sample data generated by [Avro tools](https://github.com/cockroachdb/cockroach/tree/master/pkg/ccl/importccl/testdata/avro). +The following example uses sample data generated by [Avro tools](https://github.com/cockroachdb/cockroach/tree/master/pkg/sql/importer/testdata/avro). {{site.data.alerts.end}} For example, to import the data from `simple-schema.json` into a `simple` table, first [create the table]({% link {{ page.version.version }}/create-table.md %}) to import into. Then run `IMPORT INTO` with the following options: diff --git a/src/current/v24.3/migrate-from-avro.md b/src/current/v24.3/migrate-from-avro.md index 8557ce40cea..39b7bdbc9aa 100644 --- a/src/current/v24.3/migrate-from-avro.md +++ b/src/current/v24.3/migrate-from-avro.md @@ -161,7 +161,7 @@ There are additional import [options][option] you can use when importing binary - `records_terminated_by`, which specifies the unicode character used to indicate new lines in the input binary or JSON file (default: `\n`). {{site.data.alerts.callout_info}} -The following example uses sample data generated by [Avro tools](https://github.com/cockroachdb/cockroach/tree/master/pkg/ccl/importccl/testdata/avro). +The following example uses sample data generated by [Avro tools](https://github.com/cockroachdb/cockroach/tree/master/pkg/sql/importer/testdata/avro). {{site.data.alerts.end}} For example, to import the data from `simple-schema.json` into a `simple` table, first [create the table]({% link {{ page.version.version }}/create-table.md %}) to import into. Then run `IMPORT INTO` with the following options: diff --git a/src/current/v24.3/wal-failover.md b/src/current/v24.3/wal-failover.md index 1966284e31a..ee3925c2c89 100644 --- a/src/current/v24.3/wal-failover.md +++ b/src/current/v24.3/wal-failover.md @@ -27,6 +27,10 @@ When a disk stalls on a node, it could be due to complete hardware failure or it WAL failover uses a secondary disk to fail over WAL writes to when transient disk stalls occur. This limits the write impact to a few hundreds of milliseconds (the [failover threshold, which is configurable](#unhealthy-op-threshold)). Note that WAL failover **only preserves availability of writes**. If reads to the underlying storage are also stalled, operations that read and do not find data in the block cache or page cache will stall. +The following diagram shows how WAL failover works at a high level. For more information about the WAL, memtables, and SSTables, refer to the [Architecture » Storage Layer documentation]({% link {{ page.version.version }}/architecture/storage-layer.md %}). + +WAL failover overview diagram + ## Create and configure a cluster to be ready for WAL failover The steps to provision a cluster that has a single data store versus a multi-store cluster are slightly different. In this section, we will provide high-level instructions for setting up each of these configurations. We will use [GCE](https://cloud.google.com/compute/docs) as the environment. You will need to translate these instructions into the steps used by the deployment tools in your environment. @@ -371,6 +375,10 @@ If a disk stalls for longer than the duration of [`COCKROACH_ENGINE_MAX_SYNC_DUR In a [multi-store](#multi-store-config) cluster, if a disk for a store has a transient stall, WAL will failover to the second store's disk. When the stall on the first disk clears, the WAL will failback to the first disk. WAL failover will daisy-chain from store _A_ to store _B_ to store _C_. +The following diagram shows the behavior of WAL writes during a disk stall with and without WAL failover enabled. + +how long WAL writes take during a disk stall with and without WAL failover enabled + ## FAQs ### 1. What are the benefits of WAL failover? diff --git a/src/current/v25.1/migrate-from-avro.md b/src/current/v25.1/migrate-from-avro.md index 8557ce40cea..39b7bdbc9aa 100644 --- a/src/current/v25.1/migrate-from-avro.md +++ b/src/current/v25.1/migrate-from-avro.md @@ -161,7 +161,7 @@ There are additional import [options][option] you can use when importing binary - `records_terminated_by`, which specifies the unicode character used to indicate new lines in the input binary or JSON file (default: `\n`). {{site.data.alerts.callout_info}} -The following example uses sample data generated by [Avro tools](https://github.com/cockroachdb/cockroach/tree/master/pkg/ccl/importccl/testdata/avro). +The following example uses sample data generated by [Avro tools](https://github.com/cockroachdb/cockroach/tree/master/pkg/sql/importer/testdata/avro). {{site.data.alerts.end}} For example, to import the data from `simple-schema.json` into a `simple` table, first [create the table]({% link {{ page.version.version }}/create-table.md %}) to import into. Then run `IMPORT INTO` with the following options: diff --git a/src/current/v25.1/wal-failover.md b/src/current/v25.1/wal-failover.md index eac722c8024..bcac4d00518 100644 --- a/src/current/v25.1/wal-failover.md +++ b/src/current/v25.1/wal-failover.md @@ -27,6 +27,10 @@ When a disk stalls on a node, it could be due to complete hardware failure or it WAL failover uses a secondary disk to fail over WAL writes to when transient disk stalls occur. This limits the write impact to a few hundreds of milliseconds (the [failover threshold, which is configurable](#unhealthy-op-threshold)). Note that WAL failover **only preserves availability of writes**. If reads to the underlying storage are also stalled, operations that read and do not find data in the block cache or page cache will stall. +The following diagram shows how WAL failover works at a high level. For more information about the WAL, memtables, and SSTables, refer to the [Architecture » Storage Layer documentation]({% link {{ page.version.version }}/architecture/storage-layer.md %}). + +WAL failover overview diagram + ## Create and configure a cluster to be ready for WAL failover The steps to provision a cluster that has a single data store versus a multi-store cluster are slightly different. In this section, we will provide high-level instructions for setting up each of these configurations. We will use [GCE](https://cloud.google.com/compute/docs) as the environment. You will need to translate these instructions into the steps used by the deployment tools in your environment. @@ -371,6 +375,10 @@ If a disk stalls for longer than the duration of [`COCKROACH_ENGINE_MAX_SYNC_DUR In a [multi-store](#multi-store-config) cluster, if a disk for a store has a transient stall, WAL will failover to the second store's disk. When the stall on the first disk clears, the WAL will failback to the first disk. WAL failover will daisy-chain from store _A_ to store _B_ to store _C_. +The following diagram shows the behavior of WAL writes during a disk stall with and without WAL failover enabled. + +how long WAL writes take during a disk stall with and without WAL failover enabled + ## FAQs ### 1. What are the benefits of WAL failover?