-
Notifications
You must be signed in to change notification settings - Fork 253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
draft: add namespace scoped operator mode #1867
base: main
Are you sure you want to change the base?
Conversation
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/hold |
b3a3c6e
to
11dcd29
Compare
I chose to ignore the Kubebuilder markers for the time being, as it was easier to accomplish this with pure Kustomize when handling either If we will be doing a combination, such as restricting only secrets behind the |
Thanks for a demo today! Please update the PR to the latest, update the description per the discussions, and also add the identified obstacles and future improvements. It would also be helpful if you could make a test PR that would configure a single namespace (default config is fine) by default, and thus would enable testing this via dev-env and CI, it might uncover some edge cases elsewhere that break. |
11dcd29
to
97bf046
Compare
Sure @tuminoid, I updated now the PR to showcase the new method of using kustomize custom plugins to generate roles and rolebindings for multiple namespaces. However, this still requires cleaning especially in documentation and test-framework side. Good idea to open the test PR for dev-env, do we know into which namespace the BMH will go there? So I know what config to generate for the testing |
To answer myself the namespace used in dev-env for BMH is metal3, based on BMO logs from previous tests:
|
Yes, that is correct. |
0ee4d01
to
b219193
Compare
This adds automation and docs for restricting the operator scope from cluster wide to namespace restricted. Signed-off-by: NymanRobin <[email protected]>
b219193
to
11dbdad
Compare
/test metal3-dev-env-integration-test-ubuntu-main |
Just comment this here, since we will need to fix e2e anyway before merging this PR: I realized that without being able to reload the namespace names in-place, we face a dilemma situation in E2E, as in E2E each testcase runs on its own namespace. Perhaps the best solution would be to decide on the list of namespaces before the suite starts, and install BMO with that list. Otherwise we have to use fixed namespace(s) for the test (which is not nice for logging purposes), or not test the namespace scoped at all (not nice either). |
Hi @mquhuy, I replaced the random string with test as a first measure, meaning the amount of namespaces will still be the same but fixed. However, if your concern is that this might be problematic, could you please clarify the advantages of using random strings and generating namespaces in advance, rather than using fixed namespaces, I am interested in this 😄 Nevertheless, implementing this change should be relatively straightforward. It would involve rewriting the Currently, the biggest problem I see with e2e testing is that BMHs were expected to be deleted when their namespace was deleted. Since this isn’t happening, we’re encountering MAC address clashes between the BMHs, which is resulting in failures. |
There shouldn't be any needs for random names as of now, I guess, but for e.g. if we run a suite in which 1 test is ran twice, for some reason, then the logs from the second one would override the first one xD. Anyway having a random part in test ns is a common practice for CAPI-related projects, so I guess it's still nice to have.
Yes, what I was mentioning is related to e2e only, currently the test decides the ns name and create it right before a testcase, at which point BMO has already been installed and running. The solution might be just to decide and create all the ns before the suite starts, like I mentioned. No big issue on this, I'm just mentioning it here so that it won't be forgotten 😃
Sorry I'm not following, so with scoped BMO, the BMH will stay even if the ns containing it gets deleted? |
I mean we cannot delete any namespace under the watch config before tearing down the BMO, as otherwise the manager container will start to error out. If this is a design detail of the operator framework or if it comes from BMO I have not had time to figure out yet |
Throwing out some ideas regarding the unanswered questions here: I found a proposal that looks interesting in that regard that it could allow the operator to operate without all namespaces existing. Other than that it seems there is the possibility to dynamically change the watchNamespaces config at least based on this discussion in the operator framework mailing thread. The java-sdk also seems to have an API for this uncertain about the go one. I will ask also in the slack channel fro controller-runtime |
There is a really good hints in slack on the dynamic limitations and how to potentially circumvent them slack thread FYI @tuminoid @Sunnatillo |
🧪 Using this PR for a bit of CI testing, ignore some following test triggers. /retest |
OK, it works here too. Hmm. |
Signed-off-by: Sunnatillo <[email protected]>
@NymanRobin: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
/retest |
Note: This is still a very rough draft to promote some discussion on the topic, DO NOT MERGE!
This PR has currently generated namespaces for all tests, this is for demonstration purpose, this would of course not be the case when finalizing.
The current implementation in this PR gives BMO (Bare Metal Operator) access to resources within a one or more pre-defined namespace. There is a limitation that if for whatever reason the admin of BMO would like to add another namespace this is not possible without restarting the operator as the watchNamespaces cannot be dynamically changed to my knowledge.
""
and generate the correct config!kustomize
What this PR does / why we need it:
This PR adds automation and docs for restricting the operator scope from cluster wide to namespace restricted.
In case cluster wide scope is not needed it is better to restrict the permission through rbacs to one or more predefined namespaces.
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)
format, will close the issue(s) when PR gets merged):Fixes #1261