Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add docs for GLOBAL_ASYNC_QUERIES (SIP-39) #12573

Merged
merged 1 commit into from
Jan 19, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ little bit helps, and credit will always be given.
- [Adding a DB migration](#adding-a-db-migration)
- [Merging DB migrations](#merging-db-migrations)
- [SQL Lab Async](#sql-lab-async)
- [Async Chart Queries](#async-chart-queries)
- [Chart Parameters](#chart-parameters)
- [Datasource & Chart Type](#datasource--chart-type)
- [Time](#time)
Expand Down Expand Up @@ -972,6 +973,29 @@ Note that:
to your production environment, and use the similar broker as well as
results backend configuration

### Async Chart Queries

It's possible to configure database queries for charts to operate in `async` mode. This is especially useful for dashboards with many charts that may otherwise be affected by browser connection limits. To enable async queries for dashboards and Explore, the following dependencies are required:

- Redis 5.0+ (the feature utilizes [Redis Streams](https://redis.io/topics/streams-intro))
- Cache backends enabled via the `CACHE_CONFIG` and `DATA_CACHE_CONFIG` config settings
- Celery workers configured and running to process async tasks

The following configuration settings are available for async queries (see config.py for default values)

- `GLOBAL_ASYNC_QUERIES` (feature flag) - enable or disable async query operation
- `GLOBAL_ASYNC_QUERIES_REDIS_CONFIG` - Redis connection info
- `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_PREFIX` - the prefix used with Redis Streams
- `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT` - the maximum number of events for each user-specific event stream (FIFO eviction)
- `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT_FIREHOSE` - the maximum number of events for all users (FIFO eviction)
- `GLOBAL_ASYNC_QUERIES_JWT_COOKIE_NAME` - the async query feature uses a [JWT](https://tools.ietf.org/html/rfc7519) cookie for authentication, this setting is the cookie's name
- `GLOBAL_ASYNC_QUERIES_JWT_COOKIE_SECURE` - JWT cookie secure option
- `GLOBAL_ASYNC_QUERIES_JWT_SECRET` - JWT's use a secret key to sign and validate the contents. This value should be at least 32 bytes and have sufficient randomness for proper security
- `GLOBAL_ASYNC_QUERIES_TRANSPORT` - currently the only available option is (HTTP) `polling`, but support for a WebSocket will be added in future versions
- `GLOBAL_ASYNC_QUERIES_POLLING_DELAY` - the time (in ms) between polling requests

More information on the async query feature can be found in [SIP-39](https://github.com/apache/superset/issues/9190).

## Chart Parameters

Chart parameters are stored as a JSON encoded string the `slices.params` column and are often referenced throughout the code as form-data. Currently the form-data is neither versioned nor typed as thus is somewhat free-formed. Note in the future there may be merit in using something like [JSON Schema](https://json-schema.org/) to both annotate and validate the JSON object in addition to using a Mypy `TypedDict` (introduced in Python 3.8) for typing the form-data in the backend. This section serves as a potential primer for that work.
Expand Down
26 changes: 26 additions & 0 deletions docs/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1113,6 +1113,32 @@ serialization. This can be disabled by setting ``RESULTS_BACKEND_USE_MSGPACK = F
in your configuration, should any issues arise. Please clear your existing results
cache store when upgrading an existing environment.

**Async queries for dashboards and Explore**

It's also possible to configure database queries for charts to operate in `async` mode.
This is especially useful for dashboards with many charts that may otherwise be affected
by browser connection limits. To enable async queries for dashboards and Explore, the
following dependencies are required:

- Redis 5.0+ (the feature utilizes `Redis Streams <https://redis.io/topics/streams-intro>`_)
- Cache backends enabled via the ``CACHE_CONFIG`` and ``DATA_CACHE_CONFIG`` config settings
- Celery workers configured and running to process async tasks

The following configuration settings are available for async queries (see config.py for default values)

- ``GLOBAL_ASYNC_QUERIES`` (feature flag) - enable or disable async query operation
- ``GLOBAL_ASYNC_QUERIES_REDIS_CONFIG`` - Redis connection info
- ``GLOBAL_ASYNC_QUERIES_REDIS_STREAM_PREFIX`` - the prefix used with Redis Streams
- ``GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT`` - the maximum number of events for each user-specific event stream (FIFO eviction)
- ``GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT_FIREHOSE`` - the maximum number of events for all users (FIFO eviction)
- ``GLOBAL_ASYNC_QUERIES_JWT_COOKIE_NAME`` - the async query feature uses a `JWT <https://tools.ietf.org/html/rfc7519>`_ cookie for authentication, this setting is the cookie's name
- ``GLOBAL_ASYNC_QUERIES_JWT_COOKIE_SECURE`` - JWT cookie secure option
- ``GLOBAL_ASYNC_QUERIES_JWT_SECRET`` - JWT's use a secret key to sign and validate the contents. This value should be at least 32 bytes and have sufficient randomness for proper security
- ``GLOBAL_ASYNC_QUERIES_TRANSPORT`` - currently the only available option is (HTTP) `polling`, but support for a WebSocket will be added in future versions
- ``GLOBAL_ASYNC_QUERIES_POLLING_DELAY`` - the time (in ms) between polling requests

More information on the async query feature can be found in `SIP-39 <https://github.com/apache/superset/issues/9190>`_.

**Important notes**

* It is important that all the worker nodes and web servers in
Expand Down