Skip to content
This repository has been archived by the owner on Aug 17, 2023. It is now read-only.

Commit

Permalink
Updated manager.py for reading secrets to check for the existence of …
Browse files Browse the repository at this point in the history
…a secret instead of listing

Issue:
When someone runs Kubeflow Faring from Kubeflow Pipeline, he/she will get below error:
IOError: [Errno 2] No such file or directory: ‘/etc/secrets/user-gcp-sa.json’

Reason:
When Kubeflow Fairing tries to launch the pod for model training like LightGBM distributed parallel training, it tries to check the presence of the secret by using list instead of get and when list fails due to lack of permission, it just does not attach any secrets to the pod. This causes the whole Kubeflow pipeline to fail.

Description of your changes:
The secret_exists function is incorrectly using list_namespaced_secret (https://github.com/kubeflow/fairing/blob/e0d44e870b467bbd773f836a8b1b648b79019dd1/kubeflow/fairing/kubernetes/manager.py#L220) to check for the existence of a secret instead of read_namespaced_secret. 
The default pipeline-runner SA used by Kubeflow Pipelines does not have the list permission for [secrets](https://github.com/kubeflow/manifests/blob/9e6ef8681003c195560fb7b3b75211830c19b7a0/pipeline/pipelines-runner/base/cluster-role.yaml#L9), so fairing fails to check that the secret exists and fails to attach it. kubeflow/pipelines#3742
This helped me run Kubeflow fairing from the Kubeflow pipeline successfully.
  • Loading branch information
raviranjan0309 authored May 19, 2020
1 parent b100779 commit 363af5b
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion kubeflow/fairing/kubernetes/manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ def secret_exists(self, name, namespace):
:returns: bool: True if the secret exists, otherwise return False.
"""
secrets = client.CoreV1Api().list_namespaced_secret(namespace)
secrets = client.CoreV1Api().read_namespaced_secret(namespace)
secret_names = [secret.metadata.name for secret in secrets.items]
return name in secret_names

Expand Down

0 comments on commit 363af5b

Please sign in to comment.