v10 Prerelease Update (#520)

Azure · Apr 5, 2024 · 6ee68fe · 6ee68fe
1 parent a200884
commit 6ee68fe
Show file tree

Hide file tree

Showing 209 changed files with 14,173 additions and 9,700 deletions.
diff --git a/Docs/azure-monitor-dine-siem.vsdx → Docs/Drawings/azure-monitor-dine-siem.vsdx b/Docs/azure-monitor-dine-siem.vsdx → Docs/Drawings/azure-monitor-dine-siem.vsdx
diff --git a/Docs/azure-security-controls-process.vsdx → ...ings/azure-security-controls-process.vsdx b/Docs/azure-security-controls-process.vsdx → ...ings/azure-security-controls-process.vsdx
diff --git a/Docs/epac-explanations.vsdx → Docs/Drawings/epac-explanations.vsdx b/Docs/epac-explanations.vsdx → Docs/Drawings/epac-explanations.vsdx
diff --git a/Docs/epac-github-flow.drawio → Docs/Drawings/epac-github-flow.drawio b/Docs/epac-github-flow.drawio → Docs/Drawings/epac-github-flow.drawio
diff --git a/Docs/EPAC-Presentation.pptx → .../Drawings/epac-overview-presentation.pptx b/Docs/EPAC-Presentation.pptx → .../Drawings/epac-overview-presentation.pptx
diff --git a/Docs/Drawings/epac-release-flow.drawio b/Docs/Drawings/epac-release-flow.drawio
diff --git a/Docs/Images/ci-cd-set-auto-complete.png b/Docs/Images/ci-cd-set-auto-complete.png
diff --git a/Docs/Images/epac-github-flow.png b/Docs/Images/epac-github-flow.png
diff --git a/Docs/Images/epac-release-flow.png b/Docs/Images/epac-release-flow.png
diff --git a/Docs/Images/shared-hierarchical.png b/Docs/Images/shared-hierarchical.png
diff --git a/Docs/changes.md b/Docs/changes.md
@@ -0,0 +1,159 @@
+# Changes in v10.0.0
+
+> [!CAUTION]
+> Read the [breaking changes](#breaking-changes-in-v1000) carefully and adjust your environment accordingly.
+
+## Breaking Changes in v10.0.0
+
+### Changes in `globalSettings.jsonc`
+
+> [!CAUTION]
+We heavily reworked the `globalSettings.jsonc` file. You will need to update the file.
+
+Deprecated top-level elements:
+
+- `globalNotScopes` is moved as an array into each `pacEnvironment`. If you used the `*` notation, copy the array into each `pacEnvironment`.
+- `managedIdentityLocations` is moved as a string into each `pacEnvironment`. If you used the `*` notation, copy the string into each `pacEnvironment`.
+
+Per `pacEnvironment`:
+
+- New required `managedIdentityLocation` string.
+- New optional `globalNotScopes` array.
+- New optional `deployedBy` string. We recommend against using it and let EPAC [generate the default value](#metadata-deployedby-and-assignedby).
+- `inheritedDefinitionsScopes` has been deprecated and removed. Please review the revised use case [Use Case 4:  Multiple Teams in a Hierarchical Organization](settings-desired-state.md#use-case-4-multiple-teams-in-a-hierarchical-organization).
+- `cloud` is now a required field. Previously, it was optional and defaulted to `AzureCloud`.
+- `desiredState` is now a required field.
+
+`desiredState` has newly required fields:
+
+- `strategy`: was optional and defaulted to `full`. We recommend setting it to `full`, except during a short transition period to EPAC. This was changed to require an explicit decision.
+- `keepDfcSecurityAssignments`: replaces `deleteDfcSecurityAssignments`  which defaulted to `true`. We highly recommend setting it to `false` and assigning any desired Initiative at management groups.
+
+`desiredState` fields `deleteExpiredExemptions` and `deleteOrphanedExemptions` are deprecated and removed. Exemptions with an ``unknownOwner` are only deleted when `strategy` is `full`. 
+
+The recommended `desiredState` settings are now as follows:
+
+```json
+"desiredState": {
+    "strategy": "full",
+    "keepDfcSecurityAssignments": false
+}
+```
+
+During a brief transition from a pre-EPAC to an EPAC usage, you can set `desiredState` to `ownedOnly` to keep existing Policy resources. This is not recommended for long-term use.
+
+```json
+"desiredState": {
+    "strategy": "ownedOnly",
+    "keepDfcSecurityAssignments": false
+}
+```
+
+### Desired State Handling for Policy Assignments
+
+Field `desiredState.includeResourceGroups` is deprecated/removed. This change removes all Policy Assignments in resource groups not defined in the Policy Assignment definition files. To keep the previous behavior, add a pattern `"/subscriptions/*/resourceGroups/*" to the `"excludedScopes"` array.
+
+Desired state handling for Policy Assignments related to Defender for Cloud (DfC) automatic Policy Assignments has been reworked. DfC creates two different types of Policy Assignments at the subscription level.
+
+- Security and Compliance Initiatives, such as, Microsoft cloud security benchmark, NIST SP 800-53, ... EPAC calls them DfC Security Policy Assignments. The PAC owner is listed as `managedByDfcSecurityPolicies`
+- Initiatives assigned by DfC when enrolling a subscription in a DfC workload protection plan. These assignments contain Policies required by DfC for finding vulnerabilities and threats. EPAC calls them DfC Defender Plan Policy Assignments. The PAC owner is listed as `managedByDfcDefenderPlans`.
+
+Previously, the `desiredState.deleteDfcSecurityAssignments` field (default `true`) and was used to control the deletion of DfC both types of auto-assigned Policy Assignments at the subscription level when the `desiredState.strategy` was `"full"`. The new field is `keepDfcSecurityAssignments`.
+
+- This behavior is now independent of the `desiredState.strategy` field. Therefore it will  delete DfC Security Policy Assignments at the subscription level, unless `desiredState.keepDfcSecurityAssignments` is set to `true`.
+- Assignments created by DfC when enrolling a subscription in a DfC workload protection plan are **never** deleted starting with v10.0.0
+
+### Build-PolicyDocumentation.ps1 ignores Policies with effect `Manual`
+
+- `Build-PolicyDocumentation.ps1` skips Policies with effect `Manual`. Using the switch parameter `-IncludeManualPolicies` overrides this behavior reverting to the previous behavior.
+
+### Deprecated Operational Scripts
+
+EPAC had multiple operational scripts which are not Policy as Code related. These scripts are now deprecated and will be removed in a future release. The scripts have been moved to a new folder `Scripts-Deprecated` and are not included in the PowerShell module. The scripts are:
+
+- `Get-AzMissingTags.ps1`
+- `Get-AzResourceTags.ps1`
+- `Get-AzStorageNetworkConfig.ps1`
+- `Get-AzUserRoleAssignments.ps1`
+
+We recommend that you use [Azure Governance Visualizer (AzGovViz)](https://github.com/JulianHayward/Azure-MG-Sub-Governance-Reporting) for these tasks.
+
+## Enhancements planned for v10.1.0
+
+- Script to update CSV effect/parameter files preserving extra columns: https://github.com/Azure/enterprise-azure-policy-as-code/issues/498.
+- Automatically disable deprecated Policies: https://github.com/Azure/enterprise-azure-policy-as-code/issues/516.
+- Cleanup/Improve `Export-PolicyResources` and `Build-PolicyDocumentation` scripts: https://github.com/Azure/enterprise-azure-policy-as-code/issues/517 and https://github.com/Azure/enterprise-azure-policy-as-code/issues/498.
+- Simplify exemption creation by allowing lists of scopes and Policy definitions: https://github.com/Azure/enterprise-azure-policy-as-code/issues/518.
+- Clarify SPNs, Least Privilege, and environments for CI/CD: https://github.com/Azure/enterprise-azure-policy-as-code/issues/519.
+
+## Enhancements in v10.0.0
+
+### Support for Cloud environments with limited Support for Resource Graph Queries
+
+- US Government Cloud handling of Role Assignments
+- China cloud (21v) handling for Role Assignments and Exemptions.
+
+### Cross-tenant (Lighthouse) support for Role Assignments.
+
+Cross-tenant Role Assignments are now supported. This is used if log collection is directed to a resource (Log Analytics, Event Hub. Storage) in a management tenant (e.g, Azure Lighthouse, and similar constructs) which requires you to use `additionalRoleAssignments` in the Policy Assignment file.
+
+### Simplified Exemption definitions
+
+Exemptions can be specified with a `policyDefinitionName` or `policyDefinitionId` instead of a `policyAssignmentId` and `policyDefinitionReferenceId`. EPAC creates as many Exemptions as needed to cover all Policy Assignments occurrences of the specified Policy
+- Support for Microsoft release flow in addition to GitHub flow (documentation and starter kit)
+- Schema updated to latest draft specification
+
+### Description field in Role Assignments
+
+The `description` field in Role Assignments is now populated with the Policy Assignment Id, reason and `deployedBy` value. This is useful for tracking the source of the Role Assignment.
+
+Reasons is one of:
+
+- `Role Assignment required by Policy` - Policy definition(s) specify the required Role Definition Ids.
+- `additional Role Assignment` - from filed "additionalRoleAssignments" in the Policy Assignment file.
+- `additional cross tenant Role Assignment` - from filed "additionalRoleAssignments" with `crossTenant` set to `$true` in the Policy Assignment file.
+
+### Metadata `deployedBy` and `assignedBy`
+
+`deployedBy` is a new field in the global settings per pacEnvironment. It is used to populate the `metadata` fields in the deployed resources.
+
+If not defined in global settings, EPAC generates it as `"epac/{{pacOwnerId}}/{{pacSelector}}"`. You can override this value in the Policy resource file by entering it directly to the respective `metadata` field. It is added to the deployed resources as follows:
+
+- Policy Definitions, Policy Set Definitions and Policy Exemptions - `metadata.deployedBy`.
+- Policy Assignments - `metadata.assignedBy` since Azure Portal displays it as 'Assigned by'.
+- Role Assignments - add the value to the [`description` field](#description-field-in-role-assignments).
+
+### Schema Updates
+
+Updating JSON schema to the latest [specification 2020-12](https://json-schema.org/specification).
+
+## Delayed to v10.1.0
+
+- Script to update CSV effect/parameter files preserving extra columns.
+- Automatically disable deprecated Policies.
+- `Get-AzExemptions` and `Export-AzPolicyResources` scripts to generate the new simplified Exemptions format.
+
+### Documentation Updates
+
+Reorganized the documentation to make it easier to find information. Added a new section on how to use the starter kit and how to use the Microsoft release flow.
+
+### Code Cleanup
+
+Ongoing cleanup of code: Removed unused code and improved code quality.
+
+### Performance
+
+Multiple lengthy sections of the code have been converted to parallel execution to improve performance. The change maybe ineffective if you limit the CI/CD agent to a single vCore or use the Azure DevOps provided CI/CD agents.
+
+The scripts `Build-DeploymentPlan`, `Deploy-PolicyPlan`, and `Build-PolicyDocumentation` have a new parameter `VirtualCores` to control the number of parallel threads and allowing you to optimize your performance. The code applies the following formula to adjust the `For-Each -Parallel` throttle limits (threads) based on the number of VirtualCores.
+
+- Threads = 1 x VirtualCores for pre-processing (pure compute) Policy and Policy Set parameters during Policy Assignment plan calculations
+- Threads = 2 x VirtualCores for Policy object deployment since it executes many REST calls to the Azure resource manager and therefore spends much of its time waiting on I/O. 
+- Threads = 4 (fixed) for reading and processing Policy resources; one each for
+  - Policy definitions
+  - Policy Set definitions
+  - Policy Assignments, Role Assignments, and Role Definitions
+  - Policy Exemptions
+
+Setting VirtualCores to zero (0) disables parallel processing. The default value is 4. EPAC also uses a minimum chunk size for deployments to avoid unnecessary overhead for small number of items.
+
diff --git a/Docs/ci-cd-ado-pipelines.md b/Docs/ci-cd-ado-pipelines.md
@@ -1,8 +1,8 @@
 # Azure DevOps Pipelines
 
-This page covers the specifics for Azure DevOps (ADO) pipelines. It si based on a simplified GitHub Flow as documented in [CI/CD Overview](./ci-cd-overview.md)
+This page covers the specifics for Azure DevOps (ADO) pipelines. It si based on a simplified GitHub Flow as documented in [CI/CD Overview](ci-cd-overview.md)
 
-Previously [setup App Registrations](./ci-cd-app-registrations.md) are a pre-requisite.
+Previously [setup App Registrations](ci-cd-app-registrations.md) are a pre-requisite.
 
 This repository contains starter pipelines
 
@@ -12,29 +12,29 @@ This repository contains starter pipelines
 
 ## Service connections for Azure DevOps CI/CD
 
-Create ADO service connections for each of the previously created [App Registrations](./ci-cd-app-registrations.md). You will need to retrieve the client id and create a client secret or authenticate with a X509 certificate configured for the SPN.
+Create ADO service connections for each of the previously created [App Registrations](ci-cd-app-registrations.md). You will need to retrieve the client id and create a client secret or authenticate with a X509 certificate configured for the SPN.
 
 When creating a Service Connection in Azure DevOps you can set up the service connections on a Subscription or a Management Group scope level. If you are using subscriptions to simulate a hierarchy during EPAC development, configure the service connection(s) scope level as **Subscription**. When creating a Service Connections for management groups (any EPAC environments) Deployment and EPAC Role Assignment the service connection scope level is **Management Group**.
 
 Subscription scope level | Management Group scope level
 :-----------:|:----------------:
-![image](./Images/azdoServiceConnectionSubConf.png) | ![image](./Images/azdoServiceConnectionMGConf.png)
+![image](Images/azdoServiceConnectionSubConf.png) | ![image](Images/azdoServiceConnectionMGConf.png)
 
 ## Single Tenant Pipeline
 
 ### Single Tenant Stages
 
 | Stage | Purpose | Trigger | Scripts |
 |-------|---------|---------|---------|
-| devStage | Feature branch DEV environment build, deploy and test | CI, Manual | Build-DeploymentPlans.ps1 <br> Deploy-PolicyPlan.ps1 <br/> Deploy-RolesPlan.ps1 |
-| tenantPlanFeatureStage | Feature branch based plan for prod deployment | CI, Manual | Build-DeploymentPlans.ps1 |
-| tenantPlanMainStage | Main branch based plan for prod deployment | PR Merged, Manual | Build-DeploymentPlans.ps1 |
-| tenantDeployStage | Deploy Policies defined by Main branch based plan | Prod stage approved | Deploy-PolicyPlan.ps1 |
-| tenantRolesStage | Assign roles defined by Main branch based plan | Role stage approved | Deploy-RolesPlan.ps1 |
+| devStage | Feature branch DEV environment build, deploy and test | CI, Manual | Build-DeploymentPlans <br> Deploy-PolicyPlan <br/> Deploy-RolesPlan |
+| tenantPlanFeatureStage | Feature branch based plan for prod deployment | CI, Manual | Build-DeploymentPlans |
+| tenantPlanMainStage | Main branch based plan for prod deployment | PR Merged, Manual | Build-DeploymentPlans |
+| tenantDeployStage | Deploy Policies defined by Main branch based plan | Prod stage approved | Deploy-PolicyPlan |
+| tenantRolesStage | Assign roles defined by Main branch based plan | Role stage approved | Deploy-RolesPlan |
 
 ### Single Tenant Service Connections and Roles
 
-Create Service Principals and associated service connections in Azure DevOps or the equivalent in your CI/CD tool. The SPNs require the following roles to adhere to the least privilege principle. If you have a single tenant, remove the last column and rows with connections ending in "-2".
+Create Service Principals and associated service connections in Azure DevOps or the equivalent in your CI/CD tool. The SPNs require the following roles to adhere to the least privilege principle. If you have a single tenant, remove the last column and rows with connections ending in "-2".  If a pacEnvironment in any of these stages is deploying to a lighthouse managed tenant and additionalRoleAssignemnts are to be used, ABAC assignments will need to be granted to the service principal at all remote scopes granting User Access Administrator for any roles that may need to be granted via additionalRoleAssignments.
 
 | Connection | Stages  | MG: epac-dev-mg | MG: Tenant Root |
 | :--- | :--- | :--- | :--- |
@@ -49,20 +49,20 @@ Create Service Principals and associated service connections in Azure DevOps or
 
 | Stage | Purpose | Trigger | Scripts |
 |-------|---------|---------|---------|
-| devStage | Feature branch EPAC DEV environment build, deploy and test | CI, Manual | Build-DeploymentPlans.ps1 <br> Deploy-PolicyPlan.ps1 <br/> Deploy-RolesPlan.ps1 |
-| tenantPlanFeatureStage-1 | Feature branch based plan for prod deployment (tenant 1) | CI, Manual | Build-DeploymentPlans.ps1 |
-| tenantPlanFeatureStage-2 | Feature branch based plan for prod deployment (tenant 2) | CI, Manual | Build-DeploymentPlans.ps1 |
+| devStage | Feature branch EPAC DEV environment build, deploy and test | CI, Manual | Build-DeploymentPlans <br> Deploy-PolicyPlan <br/> Deploy-RolesPlan |
+| tenantPlanFeatureStage-1 | Feature branch based plan for prod deployment (tenant 1) | CI, Manual | Build-DeploymentPlans |
+| tenantPlanFeatureStage-2 | Feature branch based plan for prod deployment (tenant 2) | CI, Manual | Build-DeploymentPlans |
 | completedFeature | Empty stage to complete feature branch | None | None |
-| tenantPlanMainStage-1 | Main branch based plan for prod deployment (tenant 1) | PR Merged, Manual | Build-DeploymentPlans.ps1 |
-| tenantDeployStage-1 | Deploy Policies defined by Main branch based plan (tenant 1) | Prod stage approved | Deploy-PolicyPlan.ps1 |
-| tenantRolesStage-1 | Assign roles defined by Main branch based plan (tenant 1) | Role stage approved | Deploy-RolesPlan.ps1 |
-| tenantPlanMainStage-2 | Main branch based plan for prod deployment (tenant 2) | PR Merged, Manual | Build-DeploymentPlans.ps1 |
-| tenantDeployStage-2 | Deploy Policies defined by Main branch based plan (tenant 2) | Prod stage approved | Deploy-PolicyPlan.ps1 |
-| tenantRolesStage-2 | Assign roles defined by Main branch based plan (tenant 2) | Role stage approved | Deploy-RolesPlan.ps1 |
+| tenantPlanMainStage-1 | Main branch based plan for prod deployment (tenant 1) | PR Merged, Manual | Build-DeploymentPlans |
+| tenantDeployStage-1 | Deploy Policies defined by Main branch based plan (tenant 1) | Prod stage approved | Deploy-PolicyPlan |
+| tenantRolesStage-1 | Assign roles defined by Main branch based plan (tenant 1) | Role stage approved | Deploy-RolesPlan |
+| tenantPlanMainStage-2 | Main branch based plan for prod deployment (tenant 2) | PR Merged, Manual | Build-DeploymentPlans |
+| tenantDeployStage-2 | Deploy Policies defined by Main branch based plan (tenant 2) | Prod stage approved | Deploy-PolicyPlan |
+| tenantRolesStage-2 | Assign roles defined by Main branch based plan (tenant 2) | Role stage approved | Deploy-RolesPlan |
 
 ### Multi Tenant Service Connections and Roles
 
-Create Service Principals and associated service connections in Azure DevOps or the equivalent in your CI/CD tool. The SPNs require the following roles to adhere to the least privilege principle. If you have a single tenant, remove the last column and rows with connections ending in "-2".
+Create Service Principals and associated service connections in Azure DevOps or the equivalent in your CI/CD tool. The SPNs require the following roles to adhere to the least privilege principle. If you have a single tenant, remove the last column and rows with connections ending in "-2".  If a pacEnvironment in any of these stages is deploying to a lighthouse managed tenant and additionalRoleAssignemnts are to be used, ABAC assignments will need to be granted to the service principal at all remote scopes granting User Access Administrator for any roles that may need to be granted via additionalRoleAssignments.
 
 | Connection | Stages  | MG: epac-dev-mg | MG: Tenant 1 Root | MG: Tenant 2 Root |
 | :--- | :--- | :--- | :--- | :--- |

diff --git a/Docs/ci-cd-app-registrations.md b/Docs/ci-cd-app-registrations.md
@@ -8,7 +8,7 @@ The following screenshot shows the Management Group hierarchy that used for the
 
 ## Custom `EPAC Resource Policy Reader Role`
 
-EPAC uses a set of Entra ID App Registrations (Service Principals). To build the deployment plan and adhere to the least-privilege-principle, a Resource Policy Reader role is required. This role is not built-in. EPAC contains script `New-AzPolicyReaderRole.ps1` to create this role or you can use the below JSON in Azure Portal.
+EPAC uses a set of Entra ID App Registrations (Service Principals). To build the deployment plan and adhere to the least-privilege-principle, a Resource Policy Reader role is required. This role is not built-in. EPAC contains script `New-AzPolicyReaderRole` to create this role or you can use the below JSON in Azure Portal.
 
 ```json
 {
-Original file line number
+Diff line change
@@ Expand Up @@
     ## Custom `EPAC Resource Policy Reader Role`
-    EPAC uses a set of Entra ID App Registrations (Service Principals). To build the deployment plan and adhere to the least-privilege-principle, a Resource Policy Reader role is required. This role is not built-in. EPAC contains script `New-AzPolicyReaderRole.ps1` to create this role or you can use the below JSON in Azure Portal.
+    EPAC uses a set of Entra ID App Registrations (Service Principals). To build the deployment plan and adhere to the least-privilege-principle, a Resource Policy Reader role is required. This role is not built-in. EPAC contains script `New-AzPolicyReaderRole` to create this role or you can use the below JSON in Azure Portal.
     ```json
     {
@@ Expand Down @@