Flask App factory PR #1 #8418

craig-rueda · 2019-10-21T16:34:36Z

SUMMARY

WIP - This is the first of several PRs which addresses SIP-24

ADDITIONAL INFORMATION

REVIEWERS

# Conflicts: # superset/__init__.py

john-bodley

Thanks @craig-rueda for tackling this. I love the idea of detangling aspects of the Superset app which historically has been problematic when extending due to potential cyclical dependencies.

john-bodley · 2019-10-21T16:41:07Z

superset/app.py

+logger = logging.getLogger(__name__)
+
+
+def create_app():


I generally like this approach as we (Airbnb) use it elsewhere. Any reason for having the SupersetAppInitializer class rather than the pattern suggested in the Flask cookie-cutter template.

Yea, the thinking here is that an end user can simply extend the SupersetAppInitializer and override only the parts they're interested in.

Although I could just do all the work in app:create_app(), it's a bit more OO to break the task of initialization into a set of phases.

I think a good example of this need would be someone that wanted to override something specific, such as the configuration of FAB, but wanted to carry forward the rest of Superset's config. SupersetAppInitializer allows you to compose your config from reusable bits as an end user.

Wondering if you tried to override SupersetAppInitializer in a local config, but it screams circular dependency, though maybe with all the proxying here it's not an issue anymore (!?)

john-bodley · 2019-10-21T16:43:06Z

superset/app.py

+        module_datasource_map.update(self.config.get("ADDITIONAL_MODULE_DS_MAP"))
+        ConnectorRegistry.register_sources(module_datasource_map)
+
+    def configure_cache(self):


Like the cookie-cutter example one could have a method which registers all extensions.

john-bodley · 2019-10-21T16:44:33Z

superset/__init__.py

-module_datasource_map = app.config["DEFAULT_MODULE_DS_MAP"]
-module_datasource_map.update(app.config["ADDITIONAL_MODULE_DS_MAP"])
-ConnectorRegistry.register_sources(module_datasource_map)
+app = create_app()


Do we still need this file? Having logic in __init__.py can be problematic for cyclical dependencies.

We do and we don't. My goal is to clear it out completely... However, the changes required in order to facilitate this will end up being quite a large PR. If you think it's reasonable to "just do it", then I'd be happy to pull that thread.

One thing of note here is that removing app from here will REQUIRE people to register their FLASK_APP env var differently superset.app -> superset.app:create_app()

john-bodley · 2019-10-21T16:46:54Z

superset/app.py

+
+
+    try:
+        # Allow user to override our config completely


I’m not sure we want to allow them to completely override the default config as there’s implicit requirements in the code that certain config variables exist.

john-bodley · 2019-10-21T16:53:37Z

superset/app.py

+                csrf.exempt(ex)
+
+    def register_blueprints(self):
+        for bp in self.config.get("BLUEPRINTS"):


mypy is going to barf at using config.get(...) as the return type is Optional[...]. In reference to my previous comment about the default config, config[...] should be suffice.

Yea, I haven't linted yet - just getting ducks in order in terms of structure

john-bodley · 2019-10-21T16:59:44Z

superset/app.py

+        """
+        pass
+
+    def init_views(self) -> None:


Should we simply make all our views blueprints and register them accordingly? This may simplify things in the future and/or reduce the cyclical dependency issues we often run into. We (Airbnb) have found the model of using blueprints for everything has worked quite successfully and the Flask app really just glues everything together in a somewhat lightweight fashion.

Yep, definitely. I want to pull up all occurrences of things like appbuilder.add_view() to this method. Ideally, the views themselves wouldn't register themselves, but rather this init logic would compose the views we're interested in.

Again, this PR was intended to be the first step in chipping away at this pattern migration. Thoughts on going for it?? @mistercrunch ??

I'm pretty sure that FAB's add_view creates a blueprint for each call

yap, it's ModelView, ModelRestApi class is a blueprint and it's actually created on add_view

superset/extensions.py

superset/utils/manifest_processor.py

superset/utils/results_backend_manager.py

superset/__init__.py

mistercrunch · 2019-10-23T04:30:59Z

Read through, LGTM. This is many steps in the right direction.

# Conflicts: # superset/__init__.py # superset/cli.py # superset/utils/core.py

john-bodley · 2019-11-08T06:52:22Z

setup.py

@@ -110,6 +110,7 @@ def get_git_sha():
    extras_require={
        "bigquery": ["pybigquery>=0.4.10", "pandas_gbq>=0.10.0"],
        "cors": ["flask-cors>=2.0.0"],
+        "flask-testing": ["Flask-Testing==0.7.1"],


Nit. Can we lowercase this like the other packages. Package names are case insensitive.

…ue in tests

codecov-io · 2019-11-15T18:02:28Z

Codecov Report

Merging #8418 into master will increase coverage by 0.07%.
The diff coverage is 83.53%.

@@            Coverage Diff             @@
##           master    #8418      +/-   ##
==========================================
+ Coverage   65.69%   65.77%   +0.07%     
==========================================
  Files         474      477       +3     
  Lines       23583    23726     +143     
  Branches     2571     2571              
==========================================
+ Hits        15494    15606     +112     
- Misses       7920     7951      +31     
  Partials      169      169

Impacted Files	Coverage Δ
superset/utils/core.py	`88.02% <ø> (-0.79%)`	⬇️
superset/forms.py	`95% <ø> (-0.46%)`	⬇️
superset/tasks/celery_app.py	`0% <0%> (-100%)`	⬇️
superset/utils/cache_manager.py	`100% <100%> (ø)`
superset/tasks/schedules.py	`79.89% <100%> (ø)`	⬆️
superset/__init__.py	`100% <100%> (+24.03%)`	⬆️
superset/sql_lab.py	`77.93% <100%> (ø)`	⬆️
superset/tasks/cache.py	`73.68% <100%> (ø)`	⬆️
superset/utils/cache.py	`46.15% <71.42%> (+4.48%)`	⬆️
superset/cli.py	`37.19% <77.55%> (+3.09%)`	⬆️
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b97ae3...d2f4ada. Read the comment docs.

john-bodley

Overall LGTM. Just a few comments.

john-bodley · 2019-11-18T02:59:28Z

superset/__init__.py

+    manifest_processor,
+    results_backend_manager,
+    security_manager as ext_security_manager,
+    talisman as ext_talisman,


Any reason these need to be names ext_?

For clarity at this point. I suppose I could have just imported extensions and then referenced each item upon assignment.

@craig-rueda what I meant was you import talisman as ext_talisman and then on line #59 there is talisman = ext_talisman and thus why not just,

from superset.extensions import talisman

Ah, I didn't know that was valid. Chaining imports :). I'm still a bit of a Python noob...

john-bodley · 2019-11-18T03:07:35Z

superset/bin/superset


 if __name__ == '__main__':
-    cli()
+    superset()


I think we could annex this file and have setup.py reference superset.cli:superset.

superset/utils/manifest_processor.py

tests/access_tests.py

dpgaspar · 2019-11-19T09:13:19Z

superset/app.py

+        appbuilder.indexview = SupersetIndexView
+        appbuilder.base_template = "superset/base.html"
+        appbuilder.security_manager_class = custom_sm
+        appbuilder.update_perms = False


Let FAB handle these:

FAB_UPDATE_PERMS defaults to True in FAB, setting superset's default to True could be done on config.py

FAB_INDEX_VIEW

FAB_BASE_TEMPLATE

We also have FAB_SECURITY_MANAGER_CLASS

This was in master - would prefer to tune configs in a subsequent PR

Actually I think we need to force-set those. Superset is force-pinning FAB configurations as it operates on these assumptions.

superset/app.py

superset/cli.py

dpgaspar · 2019-11-19T09:26:09Z

superset/extensions.py

+
+APP_DIR = os.path.dirname(__file__)
+
+appbuilder = AppBuilder(update_perms=False)


Should we force it? set FAB_UPDATE_PERMS to False on config

This was in master - would prefer to tune configs in a subsequent PR

I think we want to force it. Docs are clear about running superset init which takes care of that, and we have many other CLI commands and things where we don't want to operate against the db, create delays and flood the logs with unrelated operations.

setup.py

superset/app.py

# Conflicts: # tests/sqllab_tests.py

mistercrunch

This LGTM personally. It's a bit risky, but well overdue and a great step forward.

Let's push this forward if other committers are ok with it.

dpgaspar

LGTM amazing refactor!

john-bodley · 2019-12-06T18:25:35Z

@craig-rueda we ran into an issue with Celery using the configuration defined here related to running outside of an app context and need to use,

celery worker --app=superset.tasks.celery_app.app

instead. I was wondering whether you could update UPDATING.md and possibly CONTRIBUTING.md to reflect the change.

craig-rueda added 5 commits October 15, 2019 08:21

First cut at app factory

56e6c1a

Setting things back to master

21e3494

Working with new FLASK_APP

be37d24

Merge branch 'master' into app_factory

9d31fff

# Conflicts: # superset/__init__.py

Still need to refactor Celery

259a307

pull-request-size bot added the size/XL label Oct 21, 2019

john-bodley reviewed Oct 21, 2019

View reviewed changes

craig-rueda mentioned this pull request Oct 22, 2019

[SIP-24] Proposal to introduce Flask app factory pattern #8318

Closed

mistercrunch reviewed Oct 23, 2019

View reviewed changes

superset/__init__.py Show resolved Hide resolved

craig-rueda added 8 commits October 25, 2019 09:48

Merge branch 'master' into app_factory

9f8fe89

CLI mostly working

89bba87

Working on unit tests

fb9cf33

Moving cli stuff around a bit

01aafb5

Merge branch 'master' into app_factory

9bf6333

# Conflicts: # superset/__init__.py # superset/cli.py # superset/utils/core.py

Removing get in config

db25d4d

Defaulting test config

0ab9f0a

Adding flask-testing

921c285

john-bodley reviewed Nov 8, 2019

View reviewed changes

craig-rueda added 7 commits November 8, 2019 09:18

flask-testing casing

095529f

resultsbackend property bug

428655b

Fixing up cli

97e4cda

Quick fix for KV api

9f686cd

Working on save slice

f1318d5

Fixed core_tests

947622b

Fixed utils_tests

91a39f5

dpgaspar added the preset label Nov 13, 2019

craig-rueda added 3 commits November 13, 2019 10:44

Most tests working - still need to dig into remaining app_context iss…

f675ad4

…ue in tests

All tests passing locally - need to update code comments

78990d1

Fixing dashboard tests again

7e37cef

fixing cypress

92e7922

craig-rueda marked this pull request as ready for review November 15, 2019 18:14

john-bodley reviewed Nov 18, 2019

View reviewed changes

craig-rueda added 4 commits November 18, 2019 12:08

Merge remote-tracking branch 'remotes/upstream/master' into app_factory

798d723

moving manifest processor around

b943d68

moving results backend manager around

ab7a69c

Cleaning up __init__ a bit more

f72438a

dpgaspar requested changes Nov 19, 2019

View reviewed changes

craig-rueda added 7 commits November 19, 2019 08:28

Addressing PR comments

b30ff82

Addressing PR comments

b333044

Blacking

a9ecc7a

Fixes for running celery worker

ccd0338

Tuning isort

03a3e4a

Merge remote-tracking branch 'remotes/origin/master' into app_factory

cf0dac2

# Conflicts: # tests/sqllab_tests.py

Blacking

d2f4ada

craig-rueda requested a review from dpgaspar November 19, 2019 21:44

mistercrunch approved these changes Nov 20, 2019

View reviewed changes

dpgaspar approved these changes Nov 20, 2019

View reviewed changes

dpgaspar merged commit e490414 into apache:master Nov 20, 2019

dpgaspar deleted the app_factory branch November 20, 2019 15:47

maxmelnick-deloitte mentioned this pull request Nov 20, 2019

Docker install not working #8620

Closed

robdiciuccio mentioned this pull request Nov 26, 2019

Async queries failing in celery workers #8651

Closed

3 tasks

mistercrunch added preset-io and removed preset labels Nov 26, 2019

craig-rueda mentioned this pull request Jan 2, 2020

Moving appbuilder.xxx out of view files and into app.py #8912

Merged

6 tasks

john-bodley mentioned this pull request Feb 6, 2020

[fix] tox running of specific tests #9097

Merged

12 tasks

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.36.0 labels Feb 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flask App factory PR #1 #8418

Flask App factory PR #1 #8418

craig-rueda commented Oct 21, 2019

john-bodley left a comment

john-bodley Oct 21, 2019

craig-rueda Oct 21, 2019

mistercrunch Oct 23, 2019

john-bodley Oct 21, 2019

john-bodley Oct 21, 2019

craig-rueda Oct 21, 2019

john-bodley Oct 21, 2019

john-bodley Oct 21, 2019

craig-rueda Oct 21, 2019

john-bodley Oct 21, 2019 •

edited

Loading

craig-rueda Oct 21, 2019

mistercrunch Oct 23, 2019

dpgaspar Oct 25, 2019 •

edited

Loading

mistercrunch commented Oct 23, 2019

john-bodley Nov 8, 2019

codecov-io commented Nov 15, 2019 •

edited

Loading

john-bodley left a comment

john-bodley Nov 18, 2019

craig-rueda Nov 18, 2019

john-bodley Nov 18, 2019

craig-rueda Nov 19, 2019

john-bodley Nov 18, 2019

dpgaspar Nov 19, 2019

craig-rueda Nov 19, 2019

mistercrunch Nov 20, 2019

dpgaspar Nov 19, 2019

craig-rueda Nov 19, 2019

mistercrunch Nov 20, 2019

mistercrunch left a comment •

edited

Loading

dpgaspar left a comment

john-bodley commented Dec 6, 2019


		APP_DIR = os.path.dirname(__file__)

		appbuilder = AppBuilder(update_perms=False)

Flask App factory PR #1 #8418

Flask App factory PR #1 #8418

Conversation

craig-rueda commented Oct 21, 2019

CATEGORY

SUMMARY

ADDITIONAL INFORMATION

REVIEWERS

john-bodley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

john-bodley Oct 21, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dpgaspar Oct 25, 2019 • edited Loading

Choose a reason for hiding this comment

mistercrunch commented Oct 23, 2019

Choose a reason for hiding this comment

codecov-io commented Nov 15, 2019 • edited Loading

Codecov Report

john-bodley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mistercrunch left a comment • edited Loading

Choose a reason for hiding this comment

dpgaspar left a comment

Choose a reason for hiding this comment

john-bodley commented Dec 6, 2019

john-bodley Oct 21, 2019 •

edited

Loading

dpgaspar Oct 25, 2019 •

edited

Loading

codecov-io commented Nov 15, 2019 •

edited

Loading

mistercrunch left a comment •

edited

Loading