Adding workloads to GKE. (#52)

* Adding workloads to GKE.

* Fixing GH workflow for workloads.

.github/workflows/generators_pr.yml → .github/workflows/workloads_pr.yml

@@ -1,8 +1,8 @@
-name: Generator tests
+name: Workload tests
 on:
   pull_request:
     paths:
-      - 'generators/**'
+      - 'workloads/**'
 jobs:
   lint:
     name: Lint files
@@ -19,4 +19,5 @@ jobs:
       - name: Lint with pylint
         run: |
           pip install pylint
-          pylint generators/.
+          pylint workloads/.
+          pylint workloads/**/.

README.md

@@ -58,15 +58,7 @@ gcloud config set project <PROJECT_ID>
 
 > **NOTE:** You can find your PROJECT_ID in [Cloud Console](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects).
 
-Then, run the the `scripts/configure_project.sh` helper script to enable the following APIs in your project:
-
-- compute.googleapis.com
-- spanner.googleapis.com
-- cloudbuild.googleapis.com
-- container.googleapis.com
-- artifactregistry
-
-Once the APIs are enabled, you can set up the Spanner infrastructure using either the gcloud command line or Terraform. Instructions for both are below.
+Then you can set up the Spanner infrastructure using either the gcloud command line or Terraform. Instructions for both are below.
 
 > **NOTE:** The Terraform scripts also create a GKE Autopilot cluster.
 
@@ -107,7 +99,7 @@ gcloud auth application-default login
 terraform apply
 ```
 
-### Setup schema
+### Schema management
 Schema is managed by [Wrench](https://github.com/cloudspannerecosystem/wrench).
 
 After installing wrench, migrate the schema by running the `./scripts/schema.sh` file (replace project/instance/database information with what was used in terraform file):
@@ -128,7 +120,7 @@ To deploy to GKE, follow the [instructions here](./docs/GKE.md).
 
 Otherwise, follow the local deployment instructions for player profile and tradepost.
 
-Once the services are deployed you can use the generators to [run workloads](./generators/README.md).
+Once the services are deployed you can use the generators to [run workloads](./docs/workloads.md).
 
 Then follow the README to clean up based on whether you deployed with gcloud or Terraform.
 
@@ -274,7 +266,7 @@ make test-all
 
 > **NOTE:** The tests rely on [testcontainers-go](https://github.com/testcontainers/testcontainers-go), so [Docker](https://www.docker.com/) must be installed.
 
-## How to clean up
+## Cleaning up
 
 ### GCloud command line
 
@@ -286,14 +278,14 @@ gcloud spanner instances delete sample-instance
 
 ### Terraform
 
-If the Spanner instance was created using terraform, then from the `infrastructure` directory you can destroy the infrastructure.
+If the infrastructure was created using terraform, then from the `infrastructure` directory you can destroy the infrastructure.
 
 ```
 cd infrastructure
 terraform destroy
 ```
 
-### Clean up build and tests
+### Clean up builds and tests
 The Makefile provides a `make clean` command that removes the binaries and docker containers that were created as part of building and testing the services.
 
 ```

docs/GKE.md

@@ -5,9 +5,9 @@ This repository provides support for running the backend applications on GKE.
 ## Setup
 
 ### Terraform
-The provided Terraform [gke.tf](../infrastructure/gke.tf) will provision a [GKE Autopilot](https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview) cluster. This method of provisioning GKE will automatically manage the nodes for the cluster as the backend applications are added.
+The provided Terraform [gke.tf](../infrastructure/backend_gke.tf) will provision a [GKE Autopilot](https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview) cluster. This method of provisioning GKE will automatically manage the nodes for the cluster as the backend applications are added.
 
-These terraform [services.tf](../infrastructure/services.tf) file enables the required cloud API services, such as `cloudbuild.googleapis.com` and `container.googleapis.com`.
+The terraform also enables the required cloud API services, such as `cloudbuild.googleapis.com` and `container.googleapis.com`.
 
 Additionally, there is a Cloud Spanner service account created for the backend services.
 
@@ -26,22 +26,29 @@ kubectl get namespaces
 If there are no issues with the kubectl commands, kubectl is properly authenticated.
 
 ### Create Cloud Build images
-Each service has a dockerfile that needs to be deployed to a container registry, such as [Google Container Registry](https://cloud.google.com/container-registry). The appropriate Cloud API services will have been enabled by the Terraform scripts found in the infrastructure folder.
+Each service and workload has a dockerfile that needs to be deployed to a container registry, such as [Google Container Registry](https://cloud.google.com/container-registry). The appropriate Cloud API services will have been enabled by the Terraform scripts found in the infrastructure folder.
 
-To deploy the services, run the [`scripts/build.sh`](../scripts/build.sh) script. This will submit all service images to Cloud Build.
+To build the service containers, run the [`scripts/services_build.sh`](../scripts/services_build.sh) script. This will submit all service images to Cloud Build.
 
 ```
 export PROJECT_ID=<YOUR_PROJECT_ID>
-./scripts/build.sh
+./scripts/services_build.sh
+```
+
+To build the workload containers, run the [`scripts/workloads_build.sh`](../scripts/workloads_build.sh) script. This will submit all service images to Cloud Build.
+
+```
+export PROJECT_ID=<YOUR_PROJECT_ID>
+./scripts/workloads_build.sh
 ```
 
 > **NOTE:** This will take some time time to complete all builds
 
 ### Spanner configuration and secrets
 
-Once the images have been built, it is time to deploy the [kubernetes manifests](../kubernetes-manifests). Each backend application provides a LoadBalance service and a deployment.
+Once the images have been built, it is time to deploy the [kubernetes manifests](../kubernetes-manifests). Each backend application and workload provides a LoadBalance service and a deployment.
 
-Create a config map for the Spanner instance
+Create a config map for the Spanner instance, replace the variables with your Spanner instance and database information:
 
 ```
 sed -e "s/PROJECT_ID/$PROJECT_ID/" \
@@ -51,16 +58,23 @@ sed -e "s/PROJECT_ID/$PROJECT_ID/" \
 
 kubectl apply -f spanner_config.yaml
 ```
-> Note: [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) manages credentials to ensure access to Cloud Spanner
+> **NOTE:** [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) manages credentials to ensure access to Cloud Spanner.
 
 ### Deploy the manifests
-Once you have the kubernetes secret and config map established to connec to Cloud Spanner, the only thing left is to deploy the manifests. A [`scripts/deploy.sh`](../scripts/deploy.sh) file has been created to assist with this process.
+Once you have the kubernetes secret and config map established to connec to Cloud Spanner, the only thing left is to deploy the manifests.
+
+To deploy the services, run the [`scripts/services_deploy.sh`](../scripts/services_deploy.sh).
 
 ```
 export PROJECT_ID=<google cloud project id>
-./scripts/deploy.sh
+./scripts/services_deploy.sh
 ```
 
-## What's next
+To deploy the workloads to GKE, run the [`scripts/workloads_deploy.sh`](../scripts/workloads_deploy.sh).
+
+```
+export PROJECT_ID=<google cloud project id>
+./scripts/workloads_deploy.sh
+```
 
-Once the services are deployed you can use the generators to [run workloads](../generators/README.md).
+For more information on running the workloads, follow [these instructions](./workloads.md).

docs/workloads.md

@@ -0,0 +1,91 @@
+# Workload
+
+You can run these workloads either locally or on GKE.
+
+Either way before the workloads can be used, first deploy the [services onto GKE](../docs/GKE.md).
+
+> **NOTE:** The GKE instructions also include deploying the workloads on GKE.
+
+The rest of this document explains how to run the workloads locally.
+
+### Workload dependencies
+
+There are several dependencies required to get the generators to work:
+
+- Python 3.7+
+- Locust
+
+Assuming python3.X is installed, install dependencies via [pip](https://pypi.org/project/pip/):
+
+```
+# if pip3 is symlinked to pip
+pip install -r requirements.txt
+
+# if pip3 is not symlinked to pip
+pip3 install -r requirements.txt
+```
+
+> **NOTE:** To avoid modifying existing pip libraries on your machine, consider a solution like [virtualenv](https://pypi.org/project/virtualenv/).
+
+
+## Using the workload generators
+The provided workload generators do the following:
+
+- _authentication\_server.py_: mimics player signup and player retrieval by UUID. Login is not handled currently due to the necessity to track password creation.
+
+Run on the CLI:
+```
+locust -H http://127.0.0.1:8080 -f ./workloads/profile/authentication_server.py --headless -u=2 -r=2 -t=10s
+```
+
+Run webUI on port 8090:
+```
+locust --web-port 8090 -f ./workloads/profile/authentication_server.py
+# Connect browser to http://localhost:8090
+```
+
+- _match\_server.py_: mimics game servers matching players together, and closing games out.
+
+Run on the CLI:
+```
+locust -H http://127.0.0.1:8081 -f ./workloads/matchmaking/match_server.py --headless -u=1 -r=1 -t=10s
+```
+
+Run on port 8091:
+```
+locust --web-port 8091 -f ./workloads/matchmaking/match_server.py
+# Connect browser to http://localhost:8091
+```
+
+- _item\_generator.py_: generates some random items for our game to use.
+
+Run on the CLI for 1 minute:
+```
+locust -H http://127.0.0.1:8082 -f ./workloads/item_generator.py --headless -u=1 -r=1 -t=60s
+```
+
+- _game\_server.py_: mimics adding loot and money to players during the course of a game
+
+Run on the CLI:
+```
+locust -H http://127.0.0.1:8082 -f ./workloads/game/game_server.py --headless -u=1 -r=1 -t=10s
+```
+
+Run on port 8092:
+```
+locust --web-port 8092 -f ./workloads/game/game_server.py
+# Connect browser to http://localhost:8092
+```
+
+- _trading\_server.py_: mimics posting and closing orders on trading post
+
+Run on the CLI:
+```
+locust -H http://127.0.0.1:8083 -f ./workloads/tradepost/trading_server.py --headless -u=1 -r=1 -t=10s
+```
+
+Run on port 8093:
+```
+locust --web-port 8093 -f ./workloads/tradepost/trading_server.py
+# Connect browser to http://localhost:8093
+```

infrastructure/main.tf

@@ -27,6 +27,16 @@ provider "google" {
   project = var.gcp_project
 }
 
+data "google_project" "project" {
+}
+
+resource "google_project_service" "project" {
+  for_each = toset(var.gcp_project_services)
+  service  = each.value
+
+  disable_on_destroy = false
+}
+
 data "google_client_config" "provider" {}
 
 data "google_container_cluster" "gke-provider" {

infrastructure/terraform.tfvars.sample

@@ -1,5 +1,26 @@
+# Copyright 2023 Google LLC All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 gcp_project = "PROJECT"
 
+gcp_project_services = [
+  "clouddeploy.googleapis.com",
+  "cloudbuild.googleapis.com",
+  "container.googleapis.com",
+  "artifactregistry.googleapis.com"
+]
+
 spanner_config = {
     instance_name     = "sample-instance"
     database_name     = "sample-game"

infrastructure/variables.tf

@@ -38,6 +38,12 @@ variable "spanner_config" {
   }
 }
 
+variable "gcp_project_services" {
+  type        = list(any)
+  description = "GCP Service APIs (<api>.googleapis.com) to enable for this project"
+  default     = []
+}
+
 variable "gke_config" {
   type = object({
     cluster_name = string