Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Helm Chart does not work if you enable celery beat with extraConfigs #22105

Closed
3 tasks done
sker65 opened this issue Nov 11, 2022 · 1 comment · Fixed by #22586
Closed
3 tasks done

Helm Chart does not work if you enable celery beat with extraConfigs #22105

sker65 opened this issue Nov 11, 2022 · 1 comment · Fixed by #22586
Labels
#bug Bug report

Comments

@sker65
Copy link

sker65 commented Nov 11, 2022

If you deploy to k8s with the official helm chart and do enable celery beat pod for scheduled email reports the pod spec does not include extraConfigs if you configured any.

This leads to a crash loop as the celery beat pod refuses to start, if the flask app is somehow dependent on this extra config.

While is is included in the worker pods, it is not in the celery beat pod (kind of inconsistent as is basically the same flask app that needs to load).

How to reproduce the bug

  1. create an extra config that includes a file (in my example it is the client_secret.json, that the customSecurityManager want to load at startup time).
  2. Add a celery config like described in the example here: https://superset.apache.org/docs/installation/alerts-reports/
  3. Activate the celery beat like describe in doc over here: https://superset.apache.org/docs/installation/alerts-reports/
  4. Try to apply the helm chart
  5. The celery start command will crash as soon as it tries to load the app

see also https://github.com/apache/superset/blob/master/helm/superset/templates/deployment-beat.yaml (extraConfigs section missing in template)
vs https://github.com/apache/superset/blob/master/helm/superset/templates/deployment-worker.yaml#L109 (extraConfigs included if set)

Expected results

celery beat pod should start as normal superset or superset-workers do

Actual results

celery beat pod goes into crash loop / refuses to start

Checklist

Make sure to follow these steps before submitting your issue - thank you!

  • I have checked the superset logs for python stack traces and included it here as text if there are any.
  • I have reproduced the issue with at least the latest released version of superset.
  • I have checked the issue tracker for the same issue and I haven't found one similar.
@sker65 sker65 added the #bug Bug report label Nov 11, 2022
@sker65
Copy link
Author

sker65 commented Nov 11, 2022

This is the stacktrace that results when the client_screts.json not being found:

  File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 53, in convert
    return find_app(value)
  File "/usr/local/lib/python3.8/site-packages/celery/app/utils.py", line 384, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/usr/local/lib/python3.8/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/celery/utils/imports.py", line 100, in import_from_cwd
    return imp(module, package=package)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/superset/tasks/celery_app.py", line 31, in <module>
    flask_app = create_app()
  File "/app/superset/app.py", line 44, in create_app
    raise ex
  File "/app/superset/app.py", line 37, in create_app
    app_initializer.init_app()
  File "/app/superset/initialization/__init__.py", line 473, in init_app
    self.init_app_in_ctx()
  File "/app/superset/initialization/__init__.py", line 418, in init_app_in_ctx
    self.configure_fab()
  File "/app/superset/initialization/__init__.py", line 517, in configure_fab
    appbuilder.init_app(self.superset_app, db.session)
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/base.py", line 211, in init_app
    self.sm = self.security_manager_class(self)
  File "/app/uai/customSecurity.py", line 28, in __init__
    self.oid = OpenIDConnect(self.appbuilder.get_app)
  File "/usr/local/lib/python3.8/site-packages/flask_oidc/__init__.py", line 121, in __init__
    self.init_app(app)
  File "/usr/local/lib/python3.8/site-packages/flask_oidc/__init__.py", line 130, in init_app
    secrets = self.load_secrets(app)
  File "/usr/local/lib/python3.8/site-packages/flask_oidc/__init__.py", line 189, in load_secrets
    return _json_loads(open(app.config['OIDC_CLIENT_SECRETS'],
FileNotFoundError: [Errno 2] No such file or directory: '/app/configs/client_secret.json'

@sker65 sker65 changed the title Helm Chart does not work if you enable celebry beat with extraConfigs Helm Chart does not work if you enable celery beat with extraConfigs Nov 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
#bug Bug report
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant