azurerm_databricks_workspace - support for enhanced security compliance features

[ssouthcity]

Community Note

• Please vote on this PR by adding a 👍 reaction to the original PR to help the community and maintainers prioritize for review
• Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for PR followers and do not help prioritize for review

Description

Azure have recently introduced Enhanced Security Compliance features to Databricks Workspaces. This collection of features includes automatic updates of the workspace, security monitoring of activities within the workspace, as well as opting in to privacy and security standards. These features are now usable by everyone, as they have entered general availability. The only exception is automatic cluster updates, which is still in public preview. This PR addresses and adds these features to the Terraform provider schema.

After any of the Enhanced Security Compliance features have been activated on the workspace, the Azure API expects every subsequent PUT request to include the whole definition, including all of the features. My implementation solves this dilemma by always including the definition object with the create/update requests, using opt-out default values for all of the features. I would not consider this a breaking change, but users can now expect their POST/PUT requests to include this object.

PR Checklist

• I have followed the guidelines in our Contributing Documentation.
• I have checked to ensure there aren't other open Pull Requests for the same update/change.
• I have checked if my changes close any open issues. If so please include appropriate closing keywords below.
• I have updated/added Documentation as required written in a helpful and kind way to assist users that may be unfamiliar with the resource / data source.
• I have used a meaningful PR title to help maintainers and other users understand this change and help prevent duplicate work.
  For example: “resource_name_here - description of change e.g. adding property new_property_name_here”

Changes to existing Resource / Data Source

• I have added an explanation of what my changes do and why I'd like you to include them (This may be covered by linking to an issue above, but may benefit from additional explanation).
• I have written new tests for my resource or datasource changes & updated any relevent documentation.
• I have successfully run tests with my changes locally. If not, please provide details on testing challenges that prevented you running the tests.
• (For changes that include a state migration only). I have manually tested the migration path between relevant versions of the provider.

Testing

• My submission includes Test coverage as described in the Contribution Guide and the tests pass. (if this is not possible for any reason, please include details of why you did or could not add test coverage)

I was not able to run the acceptance test due to my company's strict Azure policies. Due to this it was also difficult to write tests, as I couldn't see if they ran green. As a compromise I did thorough experimentation with my feature, testing several different permutations of attribute values to ensure an invalid state is unreachable.

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

• azurerm_databricks_workspace - support for the enhanced_security_compliance property [GH-00000]

This is a (please select all that apply):

• Bug Fix
• New Feature (ie adding a service, resource, or data source)
• Enhancement
• Breaking Change

Related Issue(s)

Fixes #25700

Note

If this PR changes meaningfully during the course of review please update the title and description as required.

[gerrytan]

This PR needs work: it fails the acceptance test TestAccDatabricksWorkspace_basic at workspace creation with error:

HTTP/2.0 400 Bad Request

{
  "error": {
    "code": "InvalidSkuForEnhancedSecurityCompliance",
    "message": "The Enhanced Security and Compliance add-on is only available for Premium workspaces."
  }
}

(other response headers omitted for brevity)

workspace.Properties.EnhancedSecurityCompliance should be left nil if enhanced_security_compliance block not set in Terraform.

I will push a commit to fix.

---

Update 2024-10-30: fixed in 1b129ac0. It's still in my fork because I don't have permission to push into this branch yet. Will check with Hashicorp folks once I've completed review and testing.

[gerrytan]

@ssouthcity I have made some bugfix and added test coverage. Can you please cherry pick the additional commits I made in this branch: #27870 back into this PR as I don't have maintainer permission to this repo.

[gerrytan]

Thanks heaps @stephybun !

[manjinder-mckc]

our databricks workspace created by terraform, had security and compliance enabled manually outside of terraform, but an update on terraform side is resulting in this error:

Error: creating/updating Workspace (Subscription: "xxxxxxx-2270-40df-85a5-xxxxxxx"
2024-12-16 15:39:27 046f751d68df root[169] INFO │ Resource Group Name: "azureanalytics-xxxxxxx"
2024-12-16 15:39:27 046f751d68df root[169] INFO │ Workspace Name: "aza-databricks-xxxxxxx"): performing CreateOrUpdate: unexpected status 400 (400 Bad Request) with error: EnhancedSecurityCompliancePropertiesRequired: Existing Enhanced Security and Compliance properties must be specified in the PUT request.
2024-12-16 15:39:27 046f751d68df root[169] INFO │ 
2024-12-16 15:39:27 046f751d68df root[169] INFO │   with module.databricks_workspace.azurerm_databricks_workspace.databricks,
2024-12-16 15:39:27 046f751d68df root[169] INFO │   on databricks_workspace/main.tf line 17, in resource "azurerm_databricks_workspace" "databricks":
2024-12-16 15:39:27 046f751d68df root[169] INFO │   17: resource "azurerm_databricks_workspace" "databricks" {
2024-12-16 15:39:27 046f751d68df root[169] INFO │ 

terraform plan shows nothing though:

# module.databricks_workspace.azurerm_databricks_workspace.databricks will be updated in-place
2024-12-16 15:48:38 046f751d68df root[169] INFO   ~ resource "azurerm_databricks_workspace" "databricks" {
2024-12-16 15:48:38 046f751d68df root[169] INFO         id                                                  = "/subscriptions/xxxxxxx-2270-40df-85a5-xxxxxxxx/resourceGroups/azureanalytics-xxxxxxx/providers/Microsoft.Databricks/workspaces/aza-databricks-xxxxxxx"
2024-12-16 15:48:38 046f751d68df root[169] INFO         name                                                = "aza-databricks-xxxxxxx"
2024-12-16 15:48:38 046f751d68df root[169] INFO       ~ tags                                                = {
2024-12-16 15:48:38 046f751d68df root[169] INFO           ~ "deployment_version" = "2.9.0" -> "2.10.0"
2024-12-16 15:48:38 046f751d68df root[169] INFO             "instance_id"        = "xxxxxx-xxxxxxx-xxxxxxx-xxxxxxx"
2024-12-16 15:48:38 046f751d68df root[169] INFO             "lab_name"           = "azureanalytics"
2024-12-16 15:48:38 046f751d68df root[169] INFO             "lab_requester"      = ""
2024-12-16 15:48:38 046f751d68df root[169] INFO             "pbi_charge_code"    = "xxxxxx"
2024-12-16 15:48:38 046f751d68df root[169] INFO             "product_id"         = "xxxxx"
2024-12-16 15:48:38 046f751d68df root[169] INFO             "team_name"          = "senbotsandbox"
2024-12-16 15:48:38 046f751d68df root[169] INFO             "used_for"           = "prod"
2024-12-16 15:48:38 046f751d68df root[169] INFO         }
2024-12-16 15:48:38 046f751d68df root[169] INFO         # (14 unchanged attributes hidden)
2024-12-16 15:48:38 046f751d68df root[169] INFO 
2024-12-16 15:48:38 046f751d68df root[169] INFO         # (1 unchanged block hidden)
2024-12-16 15:48:38 046f751d68df root[169] INFO     }

[gerrytan]

Hi @manjinder-mckc , managing enhanced security and compliance feature outside of Terraform (while the rest of workspace config is in Terraform) is not supported. With this PR, enhanced security and compliance is now supported in Terraform.

You need to upgrade the azurerm provider version to 4.12.0 or higher and run terraform init -upgrade. The next terraform plan you run will show the delta between your local state and remote infrastructure. This will allow you to onboard the configuration.

Please also refer to the enhanced_security_compliance block documentation.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.