Approach to handle setting up license mapping in different resources

[b-per]

The current dbtcloud_license_map resource can only be set once for an entire dbt Cloud account

This is because the identifier of a resource map is unique for an account rather than each resource map having its own ID

This works fine when the resource is set only once (e.g. a centralised Terraform project for managing multiple dbt Cloud projects), but doesn't allow splitting the configuration of multiple projects into multiple Terraform projects/states.

There are a few different approaches to handle this use case but I would be keen to get feedback from the provider:

Approach 1: Keep the provider as-is and provide examples of how to manage having a state per project

My initial thinking is that I could update the repo https://github.com/b-per/dbt-avoid-chaos to show an example of how we could have:

• 1 folder/state per dbt Cloud project
• and 1 folder/state for the global configuration (e.g. license mapping)

Creating a new project might require creating a new folder and updating another one, but there is no change required on the Terraform side and we stick to the Terraform concept of unique identifier

Approach 2: Create a variation of the resource to allow "non-global" configuration of license mapping

e.g. something like

resource "dbtcloud_license_map" "test_license_map" {
  license_type = "developer"
  sso_license_mapping_groups = ["TEST-GROUP"]
  global_config = false
}

• we would check if some license mapping is already in place on the account
  • if it is not in place, we create a resource mapping and add the group "TEST-GROUP"
  • if it is in place, we add "TEST-GROUP" to the existing license type

The problems with that approach is that:

• If "TEST-GROUP" is removed from the config at some point, I don't know if there will be a way to remove it from the license type mapping (maybe using CustomizeDiff but I am not sure). If it is possible, the logic will be more complex than for other components.
• this shifts quite a bit away from the way the dbt Cloud API works

---

I am currently thinking of Approach 1 as the way to go but would be keen to get some feedback from the the current provider users.

[svenkerud]

Being that I brought this up I will provide som more information on our usecase, and how and why.

Usecase

We are a central Platform team for a data platform that has several projects running. As the number of projects increase it becomes increasingly difficult to orchestrate the upgrades of the different teams as we roll out changes to the DBT module we use to define a project. Thus we splitt out the state files for the separate teams into separate folders/state-files.

We currently run a singel dbt cloud terraform module that gets invoked by the team. The creation of a new team is copy paste of a folder, and setting of a new state name, and a new project name. The aim is for the creation of new projects to be as self scervice as possible, freeing us to work on other asspects of the platform not configuring new projects.

If we choose to go with Approach 1, we have two options. Either create a second dbt-cloud terraform module that only maintains these account level settings. In order to maintain our goal of self scervice, everyone who can create a new project, will need accesses to these account level settings. This is quite the security risk, and will need to be supported with a robust deployment pipeline to avoid accidental configuration changes. Such a pipeline would likely involve one of us, checking every PR that is run to see that noon e is changing account level stuff. Thus introducing time delay (form pr to one of us has time to check) and it limits self service. We would also have two DBT-cloud modules to maintain and keep up with changes, to avoid the setup breaking.

One solution to the two module issue, could be to implement the account level settings in the single dbt-cloud module and add some kind of logic to only run these setting in one "account level project" typically the central project (our current approach). This would however not solve the access issue, to the contrary, it would now mean that we needed to check that the account project settings weren't changed either. It would also introduce logic to the module that only is requiered for a single invocation of the module, making the project level code unnecessarily complex (to a small extent). And the time bottleneck would still remain. As I said we are currently trialing this, as the number of truly self service developers are low. and we are still required to do considerable amounts of configurations for them regardless.

Approach 2: would allow us to create a truly self contained DBT-cloud module, where all settings required for a new project could be handled by the team itself. We could be removed form the approval loop. Time to create teams goes down. accsess to otter code could be controlled with code ownership. It would increase the modularity approach to multi project terraform code. While I definitely see the challenges with the deletion aspect. And I wonder if it be worth taking this as an improvement request for the DBT-Cloud API as well as the DBT-Cloud provider. And as such I probbably se this more as a long term solution rather than a short term one.

[github-actions]

This issue has been marked as Stale because it has been open for 90 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[oameln]

Experiencing same issues as @svenkerud (forresten, hei Sigve).

+1 to approach 2.

[b-per]

I am trying approach 2 not for licenses yet, but for group permissions, via a new resource called dbtcloud_group_partial_permissions.

If anyone watching this issue has the same problem with group, please try the new resource from 0.3.3, add feedback here or in the other issue.

Based on the feedback I will think of creating partial resources for license mapping and for notifications as well.

[b-per]

I recently released dbtcloud_group_partial_permissions and dbtcloud_partial_notification to solve the same problem for other resource types.

I will now create a new dbtcloud_partial_license_map resource following the same logic and approach.

[b-per]

dbtcloud_partial_license_map is added to 0.3.6 which should be in the registry in a few minutes