Application referencing project xyz which does not exist

[cmoulliard]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When an Application is installed on a cluster within a user's namespace and where its AppProject is also installed within a user's namespace, then we got as error within the Application resource status:

Application referencing project xyz which does not exist

even if the AppProject is well created under the user's namespace.

This issue has been discussed here: #8402 (reply in thread)

To Reproduce

• Create a cluster (kind, etc)
• Install Argocd
• Create a user's namespace: test
• Update the Argocd config map - argocd-cmd-params-cm to include as parameter: application.namespaces: test
• Rollout argocd server and application controller

kubectl rollout restart -n argocd deployment argocd-server
kubectl rollout restart -n argocd statefulset argocd-application-controller

• Deploy the following yaml definition

apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: cmoullia
  name: test
---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: guestbook
  namespace: test
spec:
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  destinations:
    - namespace: '*'
      server: '*'
  sourceRepos:
    - '*'
  sourceNamespaces:
    - '*'
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bootstrap-app
  namespace: test
spec:
  destination:
    name: in-cluster
    namespace: test
  project: guestbook
  source:
    path: apps
    repoURL: https://github.com/ch007m/my-argocd.git
    targetRevision: HEAD
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=false

Expected behavior

The Argocd Applications: bootstrap-app and guestbook should be installed and status been: sync and health

Screenshots

Version

v2.10.7

Logs

Status:
  Conditions:
    Last Transition Time:  2024-12-12T08:18:11Z
    Message:               Application referencing project guestbook which does not exist
    Type:                  InvalidSpecError
  Health:
    Status:  Unknown
  Sync:
    Status:  Unknown
Events:
  Type    Reason           Age   From                           Message
  ----    ------           ----  ----                           -------
  Normal  ResourceUpdated  25s   argocd-application-controller  Updated sync status:  -> Unknown
  Normal  ResourceUpdated  25s   argocd-application-controller  Updated health status:  -> Unknown

Remark

There is a workaround which is to create 2 AppProjects: one under argocd namespace and the other under the user's namespace.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: guestbook
  namespace: argocd
spec:
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  destinations:
    - namespace: '*'
      server: '*'
  sourceRepos:
    - '*'
  sourceNamespaces:
    - '*'
---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: guestbook
  namespace: test
spec:
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  destinations:
    - namespace: '*'
      server: '*'
  sourceRepos:
    - '*'
  sourceNamespaces:
    - '*'

[cmoulliard]

Another workaround is to create the AppProject under the namespace of the argocd control plane argocd, openshift-gitops but where we restrict the sourceNamespace to test namespace.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: guestbook
  namespace: argocd
spec:
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  destinations:
    - namespace: '*'
      server: '*'
  sourceRepos:
    - '*'
  sourceNamespaces:
    - 'test'

Remark: The issue that we have we this approach is that many users will not been able to create/delete an AppProject under the namespace of the control plane due to RBAC permission !!

[NoOverflow]

Hi, can confirm this issue also occurs on ArgoCD version v2.13.1+af54ef8 installed through the Red Hat openshift-gitops operator.

I'm running a setup with two ArgoCD instances for separation of concerns between infrastructure and applicative teams, each running in their own namespace (openshift-gitops, openshift-gitops-applicative).

The second instance (applicative) is configured for multitenancy with Application(+Set) in any namespace. Creating an AppProject in the openshift-gitops-applicative namespace in a declarative way works fine and I can see it show up in the UI.

However, when an Application is scoped to this AppProject, it fails while syncing:

Application referencing project app-test-sea-appproj which does not exist

Creating the same AppProj in the "default" ArgoCD namespace (in my case openshift-gitops) fixes the issue and syncing goes through.

It's as-if one of the ArgoCD API request does not take in account the control-plane namespace for its current ArgoCD instance when querying AppProjects, and instead defaults to the default one (argocd or openshift-gitops).

Reading the code I deduced this call stack:

[Controller]
AppController.go::newApplicationInformerAndLister ? https://github.com/argoproj/argo-cd/blob/af54ef8db5adfa77a08d4d05b1318a2198084c22/controller/appcontroller.go#L2232
  AppController.go::getAppProj (errors out)         https://github.com/argoproj/argo-cd/blob/af54ef8db5adfa77a08d4d05b1318a2198084c22/controller/appcontroller.go#L381
    AppController.go::projCache::GetAppProject      https://github.com/argoproj/argo-cd/blob/af54ef8db5adfa77a08d4d05b1318a2198084c22/controller/appcontroller.go#L366
      Argo.go::GetAppProjectByName                  https://github.com/argoproj/argo-cd/blob/af54ef8db5adfa77a08d4d05b1318a2198084c22/util/argo/argo.go#L661
        Lister::AppProject.go::Get (errors out)     https://github.com/argoproj/argo-cd/blob/af54ef8db5adfa77a08d4d05b1318a2198084c22/pkg/client/listers/application/v1alpha1/appproject.go#L75
  projectErrorToCondition                           https://github.com/argoproj/argo-cd/blob/af54ef8db5adfa77a08d4d05b1318a2198084c22/controller/appcontroller.go#L2349

I believe the issue comes from an incorrect namespace given to the AppProjectLister::Get method:

argo-cd/pkg/client/listers/application/v1alpha1/appproject.go

Lines 74 to 77 in af54ef8

	func (s appProjectNamespaceLister) Get(name string) (*v1alpha1.AppProject, error) {
	obj, exists, err := s.indexer.GetByKey(s.namespace + "/" + name)
	if err != nil {
	return nil, err

But I don't see how this could be the case since the `ns` variable should be equals to `Server.ns` which seems to be the control-plane namespace, so `openshift-gitops-applicative` in my case: https://github.com/argoproj/argo-cd/blob/af54ef8db5adfa77a08d4d05b1318a2198084c22/server/application/application.go#L1045

The namespace used to query AppProject objects is set through the constructor of the AppProjectNamespaceLister object here, reading the code it should be equal to the ApplicationController's namespace:

argo-cd/controller/appcontroller.go

Line 372 in af54ef8

proj, err := argo.GetAppProjectByName(projCache.name, applisters.NewAppProjectLister(projCache.ctrl.projInformer.GetIndexer()), projCache.ctrl.namespace, projCache.ctrl.settingsMgr, projCache.ctrl.db, ctx)

I will create a debug environment when I get the time and figure out what is going on.

@cmoulliard Can you tell me where in the documentation is mentioned the fact that you can create AppProjects objects outside of the ArgoCD namespace ? Because reading the code I don't see how this could work ...

[NoOverflow]

Well this is embarrassing, I found out why my setup wasn't working. Both ArgoCD instances had the same applicative namespace configured for the Application-in-any-namespace feature and were fighting for control over it.
Removing the namespace from the openshift-gitops instance fixed the issue... Ignore my previous comment 😄

However this confirms that AppProjects can only be located in the control-plane namespace, as the ApplicationController instance looks for them in the same namespace where he is running in itself. This makes sense since AppProjects are not meant for self-service and are restricted to ArgoCD administrators.

@cmoulliard What's your use-case for self-service AppProjects (i-e, outside of argocd control plane namespace) ?

[cmoulliard]

Can you tell me where in the documentation is mentioned the fact that you can create AppProjects objects outside of the ArgoCD namespace ?

The existing documentation don't mention at all that we should use an AppProject created under the Argocd control plane namespace (= gitops, openshift-gitops nalmespaces)

[cmoulliard]

What's your use-case for self-service AppProjects (i-e, outside of argocd control plane namespace) ?

My use is pretty simple. We would like to offer the possibility that users creates their applications within their own namespaces and that argocd reconcile them (= synced) without the need to touch a AppProject of the argocd control plane as users don't have access to such a namespace

[NoOverflow]

The existing documentation don't mention at all that we should use an AppProject created under the Argocd control plane namespace (= gitops, openshift-gitops nalmespaces)

True, I guess it is self-implied because if you allow users to create AppProjects, they effectively become ArgoCD administrators which doesn't make sense for most use-cases.

My use is pretty simple. We would like to offer the possibility that users creates their applications within their own namespaces and that argocd reconcile them (= synced) without the need to touch a AppProject of the argocd control plane as users don't have access to such a namespace

What you could do is use a Kyverno generate policy, that, on AppProject creation:

• Validate that the AppProject is restrained to the user's namespace(s) by checking /spec/destinations
• If valid, "mirror" this AppProject in ArgoCD's control-plane namespace

(See: https://kyverno.io/docs/writing-policies/generate/)

This obviously comes with some issues and edge cases but you seem to have a really specific use-case and that would be my approach with it.

But ideally you would have a process that automates the creation of an AppProject when creating the project for a user

[cmoulliard]

But ideally you would have a process that automates the creation of an AppProject when creating the project for a user

We have a project able to generate such an AppProject - https://github.com/quarkiverse/quarkus-argocd/blob/main/deployment/src/main/java/io/quarkiverse/argocd/deployment/ArgoCDProcessor.java#L80 so that should not be an issue for us until now to create by default the AppProject under the namespace of Argocd control plane (if of course we have the RBAC rights) ;-)

[cmoulliard]

True, I guess it is self-implied because if you allow users to create AppProjects, they effectively become ArgoCD administrators which doesn't make sense for most use-cases.

Documentation should be improved to clearly warn the user that the AppProject should always been created or updated under the Argocd control plane namespace

[NoOverflow]

Documentation should be improved to clearly warn the user that the AppProject should always been created or updated under the Argocd control plane namespace

Agreed 😄

[agaudreault]

However this confirms that AppProjects can only be located in the control-plane namespace, as the ApplicationController instance looks for them in the same namespace where he is running in itself. This makes sense since AppProjects are not meant for self-service and are restricted to ArgoCD administrators.

That is correct. All resources should be created in the argo control-plane namespace, unless specified otherwise. AppProject is the Security and tenancy control, allowing users to create them without validation would not make sense.

[agaudreault]

@cmoulliard Please open a PR on what you think the wording in the documentation should be to avoid confusion, or close the issue.