-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(chart): Supporting custom SQL as temporal x-axis column with filter #25126
Conversation
… while applying granularity
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the fix - one comment about using a pre-existing util. Also, could we add a test to make sure this works as expected? There should be several old integration tests for the chart data endpoint that can be repurposed for this test. Let me know if you need help setting it up.
@zephyring also, let's add a proper title + description to explain the change here. |
/testenv up |
@kgabryje Ephemeral environment spinning up at http://54.149.185.45:8080. Credentials are |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM - one last nit about using a type guard to detect adhoc columns, other than that LGTM
@@ -128,6 +128,8 @@ def _apply_granularity( | |||
|
|||
if granularity := query_object.granularity: | |||
filter_to_remove = None | |||
if is_adhoc_column(x_axis): # type: ignore |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm curious why we need # type: ignore
here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the x_axis is from form_data and it's ANY
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It may be a good idea to broaden the scope of the type guard to Any
so it can handle arbitrary types.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the type guard is better to scope down rather than scope up right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@zephyring @villebro I think this PR introduced another potential bug in Adhoc column as X-Axis, you should sent a probe query to ensure x-axis is a temporal column.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hi @zhaoyongjie can you share more on what potential bug this may've introduced?
@@ -1007,6 +1007,8 @@ def adhoc_column_to_sqla( # pylint: disable=too-many-locals | |||
qry = sa.select([sqla_column]).limit(1).select_from(tbl) | |||
sql = self.database.compile_sqla_query(qry) | |||
col_desc = get_columns_description(self.database, self.schema, sql) | |||
if not col_desc: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dealing with case where custom_sql
can be empty string and col_desc is empty
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should handle this on frontend and not let users save a custom sql that's an empty string
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kgabryje I agree - however, as it will be lots of work to clean up old metadata, I think we can maybe have both?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should handle this edge case in the backend anyways. I will be nice to prevent user from saving empty custom sql in the frontend and show some friendly message. But for now if the custom_sql is any arbitrary string the API responds with 500 with error message directly from DB engine, which is "fine" and readable for users but just this empty string it throws index out of bound error
to user which is not helpful.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can create a separate PR for UI messaging and prevention though
Ephemeral environment shutdown and build artifacts deleted. |
…er (apache#25126) Co-authored-by: Kamil Gabryjelski <[email protected]>
…er (#25126) Co-authored-by: Kamil Gabryjelski <[email protected]>
"granularity": "ds", | ||
"filters": [ | ||
{"col": "ds", "op": "TEMPORAL_RANGE", "val": "No filter"} | ||
], | ||
"extras": { | ||
"having": "", | ||
"where": "", | ||
}, | ||
"applied_time_extras": {}, | ||
"columns": [ | ||
{ | ||
"columnType": "BASE_AXIS", | ||
"datasourceWarning": False, | ||
"expressionType": "SQL", | ||
"label": "My column", | ||
"sqlExpression": "ds", | ||
"timeGrain": "P1W", | ||
} | ||
], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Basically, the "granularity" and "Axis-column" shouldn't be concurrent sent from frontend. the "granularity" is a legacy Druid "concept"(and this concept as be wrapped under the hood in "modern" Druid), so from my original design would like to totally remove "granularity" in Superset, the "granularity" means that a temporal column in columns(or dimensions).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the "time_range" and "filters" in query_object is same logic as before.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Basically, the "granularity" and "Axis-column" shouldn't be concurrent sent from frontend. the "granularity" is a legacy Druid "concept"(and this concept as be wrapped under the hood in "modern" Druid), so from my original design would like to totally remove "granularity" in Superset, the "granularity" means that a temporal column in columns(or dimensions).
I'm trying to fix a bug related to granularity. Basically, the temporal granularity column should be renamed as "__timestamp" like what the legacy time series line chart did before(alias after group by should not be the same with the raw column name in clickhouse) but current chart don't, so granularity is a little different from other temporal columns(at least for users). If the "granularity" should be removed, should this bug be fixed on the frontend?
it seems like the time column as x-axis has been completely unusable for a very long time. thanks to this PR made custom SQL work
* fix: is_select with UNION (apache#25290) (cherry picked from commit bb002d6) * fix: Add explicit ON DELETE CASCADE for dashboard_roles (apache#25320) (cherry picked from commit d54e827) * fix(chart): Supporting custom SQL as temporal x-axis column with filter (apache#25126) Co-authored-by: Kamil Gabryjelski <[email protected]> * fix: Use RLS clause instead of ID for cache key (apache#25229) (cherry picked from commit fba66c6) * fix: Improve the reliability of alerts & reports (apache#25239) (cherry picked from commit f672d5d) * fix: DashboardRoles cascade operation (apache#25349) (cherry picked from commit a971a28) * fix: datetime with timezone excel export (apache#25318) Co-authored-by: Michael S. Molina <[email protected]> (cherry picked from commit 5ebcd2a) * fix: Workaround for Cypress ECONNRESET error (apache#25399) (cherry picked from commit d76ff39) * fix(sqllab): invalid persisted tab state (apache#25308) (apache#25398) * fix: Rename on_delete parameter to ondelete (apache#25424) (cherry picked from commit 893b45f) * fix: preventing save button from flickering in SQL Lab (apache#25106) (cherry picked from commit 296ff17) * fix: chart import (apache#25425) (cherry picked from commit a4d8f36) * fix: swagger UI CSP error (apache#25368) (cherry picked from commit 1716b9f) * fix: smarter date formatter (apache#25404) (cherry picked from commit f0080f9) * fix(sqllab): invalid start date (apache#25437) * fix(nativeFilters): Speed up native filters by removing unnecessary rerenders (apache#25282) Co-authored-by: JUST.in DO IT <[email protected]> (cherry picked from commit a0eeb4d) * fix(SqlLab): make icon placement even (apache#25372) (cherry picked from commit 11b49a6) * fix: Duplicate items when pasting into Select (apache#25447) (cherry picked from commit 7cf96cd) * fix: update the SQLAlchemy model definition at json column for Log table (apache#25445) (cherry picked from commit e83a76a) * fix(helm chart): set chart appVersion to 3.0.0 (apache#25373) * fix(mysql): handle string typed decimal results (apache#24241) (cherry picked from commit 7eab59a) * fix: Styles not loading because of faulty CSP setting (apache#25468) (cherry picked from commit 0cebffd) * fix(sqllab): error with lazy_gettext for tab titles (apache#25469) (cherry picked from commit ddde178) * fix: Address Mypy issue which is causing CI to fail (apache#25494) (cherry picked from commit 36ed617) * chore: Adds 3.0.1 CHANGELOG * fix: Unable to sync columns when database or dataset name contains `+` (apache#25390) (cherry picked from commit dbe0838) * fix(sqllab): Broken query containing 'children' (apache#25490) (cherry picked from commit b92957e) * chore: Expand error detail on screencapture (apache#25519) (cherry picked from commit ba541e8) * fix: tags permissions error message (apache#25516) (cherry picked from commit 50b0816) * fix: Apply normalization to all dttm columns (apache#25147) (cherry picked from commit 58fcd29) * fix: REST API CSRF exempt list (apache#25590) (cherry picked from commit 549abb5) * fix(RLS): Fix Info Tooltip + Button Alignment on RLS Modal (apache#25400) (cherry picked from commit a6d0e6f) * fix: thubmnails loading - Talisman default config (apache#25486) (cherry picked from commit 52f631a) * fix(Presto): catch DatabaseError when testing Presto views (apache#25559) Co-authored-by: Rui Zhao <[email protected]> (cherry picked from commit be3714e) * fix(Charts): Set max row limit + removed the option to use an empty row limit value (apache#25579) (cherry picked from commit f556ef5) * fix(window): unavailable localStorage and sessionStorage (apache#25599) * fix: finestTemporalGrainFormatter (apache#25618) (cherry picked from commit 62bffaf) * fix: revert fix(sqllab): Force trino client async execution (apache#24859) (apache#25541) (cherry picked from commit e56e0de) * chore: Updates 3.0.1 CHANGELOG * fix(sqllab): Mistitled for new tab after rename (apache#25523) (cherry picked from commit a520124) * fix(sqllab): template validation error within comments (apache#25626) (cherry picked from commit b370c66) * fix: avoid 500 errors with SQLLAB_BACKEND_PERSISTENCE (apache#25553) (cherry picked from commit 99f79f5) * fix(import): Make sure query context is overwritten for overwriting imports (apache#25493) (cherry picked from commit a0a0d80) * fix: permalink save/overwrites in explore (apache#25112) Co-authored-by: Elizabeth Thompson <[email protected]> (cherry picked from commit e58a3ab) * fix(header navlinks): link navlinks to path prefix (apache#25495) (cherry picked from commit 51c56dd) * fix: improve upload ZIP file validation (apache#25658) * fix: warning of nth-child (apache#23638) (cherry picked from commit 16cc089) * fix(dremio): Fixes issue with Dremio SQL generation for Charts with Series Limit (apache#25657) (cherry picked from commit be82657) --------- Co-authored-by: Beto Dealmeida <[email protected]> Co-authored-by: John Bodley <[email protected]> Co-authored-by: Zef Lin <[email protected]> Co-authored-by: Kamil Gabryjelski <[email protected]> Co-authored-by: Jack Fragassi <[email protected]> Co-authored-by: Michael S. Molina <[email protected]> Co-authored-by: JUST.in DO IT <[email protected]> Co-authored-by: Jack <[email protected]> Co-authored-by: Daniel Vaz Gaspar <[email protected]> Co-authored-by: Stepan <[email protected]> Co-authored-by: Corbin Bullard <[email protected]> Co-authored-by: Gyuil Han <[email protected]> Co-authored-by: Celalettin Calis <[email protected]> Co-authored-by: Ville Brofeldt <[email protected]> Co-authored-by: ʈᵃᵢ <[email protected]> Co-authored-by: Michael S. Molina <[email protected]> Co-authored-by: mapledan <[email protected]> Co-authored-by: Igor Khrol <[email protected]> Co-authored-by: Rui Zhao <[email protected]> Co-authored-by: Fabien <[email protected]> Co-authored-by: Hugh A. Miles II <[email protected]> Co-authored-by: OskarNS <[email protected]>
…er (apache#25126) Co-authored-by: Kamil Gabryjelski <[email protected]>
…er (apache#25126) Co-authored-by: Kamil Gabryjelski <[email protected]>
SUMMARY
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TESTING INSTRUCTIONS
ADDITIONAL INFORMATION