Create the .NET Product Construction Service

[premun]

Context

As part of the Unified Build effort, we will need a new service that will be flowing code between VMR and the product repositories. The details are described here (Full Code Flow.md).

Architecture / integration with Maestro++

flowchart
  SubscriptionTrigger[1. External subscription trigger]
  EngCommon[Copy eng/common, update global.json..]
  VersionFiles[Update Versions.props, Version.Details.xml]

  SubscriptionTrigger--\nE.g. PR check finished, PR approved,\nuser calls trigger-subscription,\nbuild is added to a channel,\nweekly subscription is triggered... -->Maestro

  CallConductorService[3. Call Conductor service]
  GitHubPR[GitHub PR is opened/merged/updated/..]

  subgraph Maestro service
    Maestro{2. Maestro service processes the impuls}

    EngCommon-->VersionFiles
    Maestro--a. If source or target repo is VMR-->CallConductorService
    Maestro--b. If source repo is arcade-->EngCommon
    Maestro--c. Else-->VersionFiles
  end

  subgraph Conductor service
    Conductor{4. Conductor service prepares\nthe content in a branch\nof the target repository}
    Conductor--Maestro service is pinged back-->Maestro
  end

  CallConductorService-->Conductor

  VersionFiles-->GitHubPR
  Maestro--5. If is a ping from the Conductor service-->GitHubPR

  classDef New fill:#00DD00,stroke:#006600,stroke-width:1px,color:#006600
  class CallConductorService,Conductor,VmrChangesPushed New
  linkStyle 2,5,6,8 stroke-width:2px,fill:none,stroke:#00DD00,color:#00DD00

Loading

Requirements

• The service is called Product Construction Service.
• The service is an .NET 8.0 ASP.NET Core web service built using .NET Aspire.
• The service runs as an Azure Container App.
• The service is wired up with other infrastructure:
  • BAR database
  • Application Insights with support of custom metric collection (e.g. code flow info, disk use, git operations etc.)
  • KeyVaults
  • GitHub, AzDO accessibility
• Grafana dashboard is created for the service
• The code for the service lives in arcade-services and is deployed via a CI/CD pipeline. Blue/green deployment strategy should be used.
• There is a staging and production environment.
• The service keeps a local disk cache with clones of git repositories (VMR + product repos) and uses those without having to clone them frequently.
• The service can be run locally against either the staging or some local environment to exercise a sample flow.
• Local development story is a top priority
  • The service can be run locally in the same docker image as we push to the registry.
  • C# debugging must be available from Visual Studio.
  • Minimal setup required from devs (e.g. local BAR DB can be dockerized for instance

Milestones

• Bare service with no code-flow functionality but CI/CD story and resource connectivity is created
• Service gets an endpoint that will locally initialize/update/sync a VMR clone to test its basic git functionality

Full integration with Maestro and implementation of the real code flow will happen as a follow-up and not part of this.

Work Items

• Figure out if we want an Application Gateway in front of the VMR Backflow service #3159
• Switch the Service to West US 2 once the feature flag is available #3180
• Create a Production environment for the service and make deployments possible #3183
• Product Construction Service should be able to access BAR #3195
• Implement a workitem enqueuing functionality for the Product Construction Service #3201
• Implement the Worker part of the Product Construction Service #3202
• Give the .NET Product Construction Service ability to read Key Vault secrets #3163
• Create a staging environment for the Product Construction Service #3162
• Create a bare bones service for the VMR backflow, and make it deployable #3158
• Setup Application Insights for the new service #3173
• Add a volume to the PCS Container App, and initialize the VMR on it #3213
• Add support for custom metrics and a grafana dashboard for them #3214
• Add Authentication to the PCS, don't allow anonymous access #3215

[dkurepa]

This Epic has been completed. We created a follow up epic that will bring additional enchantments to the Product Construction Service: https://github.com/dotnet/arcade-services/milestone/39