feat: generate plan yamls #2177

tedim52 · 2024-02-17T18:07:54Z

Description:

This change implements the generation of a yaml that represents the effect of a sequence of instructions on an enclave. The major changes are as follows:

Adds gRPC endpoints GetStarlarkPackge/ScriptPlanYaml to APIC and Enclave Manager for returning this yaml
Implements PlanYaml object and yaml generation logic in startosis_engine
Adds UpdatePlan(plan *PlanYaml) method to KurtosisInstruction interface so each instruction implements logic for updating the plan yaml
- Most of the knowledge needed to generate the yaml comes from the interpretation phase and is simply passed into yaml generation logic

Tests are in startosis_interpreter_plan_yaml_tests.go and demonstrate how the InstructionsPlan generates the yaml via the PlanYaml object.

eg. starlark script turned plan yaml:

def run(plan, hi_files_artifact):
	service = plan.add_service(
		name="db",
		config=ServiceConfig(
			image="postgres:latest",
			env_vars={
				"POSTGRES_DB": "kurtosis",
				"POSTGRES_USER": "kurtosis",
				"POSTGRES_PASSWORD": "kurtosis",
			},
			files = {
				"/root": hi_files_artifact,
			}
		)
	)
	execResult = plan.exec(
		service_name="db",
		recipe=ExecRecipe(
			command=["echo", service.ip_address + " " + service.hostname]
		),
		acceptable_codes=[0],
	)	
	runShResult = plan.run_sh(
		run="echo " + execResult["code"] + " " + execResult["output"],
	)
	plan.run_sh(
		run="echo " + runShResult.code + " " + runShResult.output,
	)

plan yaml:

packageId: DEFAULT_PACKAGE_ID_FOR_SCRIPT
services:
- uuid: "1"
  name: db
  image:
    name: postgres:latest
  envVars:
  - key: POSTGRES_DB
    value: kurtosis
  - key: POSTGRES_PASSWORD
    value: kurtosis
  - key: POSTGRES_USER
    value: kurtosis
  files:
  - mountPath: /root
    filesArtifacts:
    - uuid: "2"
      name: hi-file
filesArtifacts:
- uuid: "2"
  name: hi-file
tasks:
- uuid: "3"
  taskType: exec
  command:
  - echo
  - '{{ kurtosis.1.ip_address }} {{ kurtosis.1.hostname }}'
  serviceName: db
  acceptableCodes:
  - 0
- uuid: "4"
  taskType: sh
  command:
  - echo {{ kurtosis.3.code }} {{ kurtosis.3.output }}
  image: badouralix/curl-jq
- uuid: "5"
  taskType: sh
  command:
  - echo {{ kurtosis.4.code }} {{ kurtosis.4.output }}
  image: badouralix/curl-jq

Is this change user facing?

NO

References:

The Enclave Manager uses this plan yaml to render packages in the Enclave Builder:
#2250

gitguardian · 2024-02-17T18:07:59Z

⚠️ GitGuardian has uncovered 39 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
9966505	Triggered	Generic Password	`ec3cf48`	core/server/api_container/server/startosis_engine/startosis_interpreter_plan_yaml_test.go	View secret
9966505	Triggered	Generic Password	`ec3cf48`	core/server/api_container/server/startosis_engine/startosis_interpreter_plan_yaml_test.go	View secret
9966505	Triggered	Generic Password	`f69701f`	core/server/api_container/server/startosis_engine/startosis_interpreter_plan_yaml_test.go	View secret
9966505	Triggered	Generic Password	`b254a49`	core/server/api_container/server/startosis_engine/startosis_interpreter_plan_yaml_test.go	View secret
9966505	Triggered	Generic Password	`b254a49`	core/server/api_container/server/startosis_engine/startosis_interpreter_plan_yaml_test.go	View secret
9583343	Triggered	Generic Password	`3998b57`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`8c7a419`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`323ac97`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`80ace33`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`908ad7d`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`ccc256b`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`458fd75`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`ccc256b`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`21e4ead`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`13a5cba`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`0544d1b`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`3998b57`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`9109cac`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`21e4ead`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`8c7a419`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`13a5cba`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`679ed73`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`d190067`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`a4e7031`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`13a5cba`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`aef291e`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`13a5cba`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`aef291e`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`ae9f7ce`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`80ace33`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`21e4ead`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`a4e7031`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`9109cac`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`21e4ead`	core/server/api_container/server/startosis_engine/plan_yaml_generator_test.go	View secret
9583343	Triggered	Generic Password	`13a5cba`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`323ac97`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`d190067`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`03a7535`	core/server/api_container/server/startosis_engine/plan.yml	View secret
9583343	Triggered	Generic Password	`ec3cf48`	core/server/api_container/server/startosis_engine/startosis_interpreter_plan_yaml_test.go	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}

tedim52 · 2024-03-11T22:23:03Z

The GitGuardian secrets are only picking up the PASSWORD env vars that I set when writing tests, no secrets were actually used in this PR.

tedim52 · 2024-03-11T22:25:38Z

I considered renaming PlanYaml to EnclaveState for a while as the yaml represents state of an enclave post an instructions plan containing services and files artifacts, but the fact that tasks are in it throws me off a bit as tasks aren't really state, but maybe we consider them so. Open to suggestions on naming.

h4ck3rk3y

spent some time reviewing this - overall happy as long as it works!

some of the plan_yaml_generator stuff can be cleaned up but we can get to it in our own time.

api/protobuf/core/api_container_service.proto

core/server/api_container/server/startosis_engine/plan_yaml/plan_yaml.go

enclave-manager/server/server.go

core/server/api_container/server/startosis_engine/kurtosis_instruction/request/request.go

...erver/api_container/server/startosis_engine/kurtosis_instruction/add_service/add_services.go

core/server/api_container/server/startosis_engine/plan_yaml/plan_yaml_generator.go

@tedim52

## Description: This PR adds package nodes to the enclave builder UI. It's based on the backend work added by @tedim52 in #2177. ### Demo This demo has two paths - the first one shows how a package can be added to the enclave builder, and used by a service defined in the builder as a dependency. The second one shows how two packages can be combined in the same enclave - allowing connectivity between them at runtime. https://github.com/kurtosis-tech/kurtosis/assets/4419574/64ed2b91-1565-49ed-841b-c655dd6b28f9 ## Is this change user facing? YES ## References (if applicable): - Discussion on slack. --------- Co-authored-by: Tedi Mitiku <[email protected]>

🤖 I have created a release *beep* *boop* --- ## [0.88.6](0.88.5...0.88.6) (2024-03-13) ### Features * define `build_args` in `ImageBuildSpec` ([#2257](#2257)) ([9812375](9812375)) * generate plan yamls ([#2177](#2177)) ([2292010](2292010)) * port guessing based on port number for docker transpiler ([#2281](#2281)) ([5c98ce2](5c98ce2)) * support package nodes in enclave builder ui ([#2283](#2283)) ([7fa27ed](7fa27ed)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: kurtosisbot <[email protected]>

## Description: This change implements the generation of a yaml that represents the effect of a sequence of instructions on an enclave. The major changes are as follows: - Adds gRPC endpoints `GetStarlarkPackge/ScriptPlanYaml` to APIC and Enclave Manager for returning this yaml - Implements `PlanYaml` object and yaml generation logic in `startosis_engine` - Adds `UpdatePlan(plan *PlanYaml)` method to `KurtosisInstruction` interface so each instruction implements logic for updating the plan yaml - Most of the knowledge needed to generate the yaml comes from the interpretation phase and is simply passed into yaml generation logic Tests are in `startosis_interpreter_plan_yaml_tests.go` and demonstrate how the `InstructionsPlan` generates the yaml via the `PlanYaml` object. eg. starlark script turned plan yaml: ``` def run(plan, hi_files_artifact): service = plan.add_service( name="db", config=ServiceConfig( image="postgres:latest", env_vars={ "POSTGRES_DB": "kurtosis", "POSTGRES_USER": "kurtosis", "POSTGRES_PASSWORD": "kurtosis", }, files = { "/root": hi_files_artifact, } ) ) execResult = plan.exec( service_name="db", recipe=ExecRecipe( command=["echo", service.ip_address + " " + service.hostname] ), acceptable_codes=[0], ) runShResult = plan.run_sh( run="echo " + execResult["code"] + " " + execResult["output"], ) plan.run_sh( run="echo " + runShResult.code + " " + runShResult.output, ) ``` plan yaml: ``` packageId: DEFAULT_PACKAGE_ID_FOR_SCRIPT services: - uuid: "1" name: db image: name: postgres:latest envVars: - key: POSTGRES_DB value: kurtosis - key: POSTGRES_PASSWORD value: kurtosis - key: POSTGRES_USER value: kurtosis files: - mountPath: /root filesArtifacts: - uuid: "2" name: hi-file filesArtifacts: - uuid: "2" name: hi-file tasks: - uuid: "3" taskType: exec command: - echo - '{{ kurtosis.1.ip_address }} {{ kurtosis.1.hostname }}' serviceName: db acceptableCodes: - 0 - uuid: "4" taskType: sh command: - echo {{ kurtosis.3.code }} {{ kurtosis.3.output }} image: badouralix/curl-jq - uuid: "5" taskType: sh command: - echo {{ kurtosis.4.code }} {{ kurtosis.4.output }} image: badouralix/curl-jq ``` ## Is this change user facing? NO ## References: The Enclave Manager uses this plan yaml to render packages in the Enclave Builder: #2250 --------- Co-authored-by: Ben Gazzard <[email protected]>

🤖 I have created a release *beep* *boop* --- ## [0.88.6](0.88.5...0.88.6) (2024-03-13) ### Features * define `build_args` in `ImageBuildSpec` ([#2257](#2257)) ([9812375](9812375)) * generate plan yamls ([#2177](#2177)) ([2292010](806a13e)) * port guessing based on port number for docker transpiler ([#2281](#2281)) ([5c98ce2](e20fb32)) * support package nodes in enclave builder ui ([#2283](#2283)) ([7fa27ed](3c25691)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: kurtosisbot <[email protected]>

## Description: This change implements the generation of a yaml that represents the effect of a sequence of instructions on an enclave. The major changes are as follows: - Adds gRPC endpoints `GetStarlarkPackge/ScriptPlanYaml` to APIC and Enclave Manager for returning this yaml - Implements `PlanYaml` object and yaml generation logic in `startosis_engine` - Adds `UpdatePlan(plan *PlanYaml)` method to `KurtosisInstruction` interface so each instruction implements logic for updating the plan yaml - Most of the knowledge needed to generate the yaml comes from the interpretation phase and is simply passed into yaml generation logic Tests are in `startosis_interpreter_plan_yaml_tests.go` and demonstrate how the `InstructionsPlan` generates the yaml via the `PlanYaml` object. eg. starlark script turned plan yaml: ``` def run(plan, hi_files_artifact): service = plan.add_service( name="db", config=ServiceConfig( image="postgres:latest", env_vars={ "POSTGRES_DB": "kurtosis", "POSTGRES_USER": "kurtosis", "POSTGRES_PASSWORD": "kurtosis", }, files = { "/root": hi_files_artifact, } ) ) execResult = plan.exec( service_name="db", recipe=ExecRecipe( command=["echo", service.ip_address + " " + service.hostname] ), acceptable_codes=[0], ) runShResult = plan.run_sh( run="echo " + execResult["code"] + " " + execResult["output"], ) plan.run_sh( run="echo " + runShResult.code + " " + runShResult.output, ) ``` plan yaml: ``` packageId: DEFAULT_PACKAGE_ID_FOR_SCRIPT services: - uuid: "1" name: db image: name: postgres:latest envVars: - key: POSTGRES_DB value: kurtosis - key: POSTGRES_PASSWORD value: kurtosis - key: POSTGRES_USER value: kurtosis files: - mountPath: /root filesArtifacts: - uuid: "2" name: hi-file filesArtifacts: - uuid: "2" name: hi-file tasks: - uuid: "3" taskType: exec command: - echo - '{{ kurtosis.1.ip_address }} {{ kurtosis.1.hostname }}' serviceName: db acceptableCodes: - 0 - uuid: "4" taskType: sh command: - echo {{ kurtosis.3.code }} {{ kurtosis.3.output }} image: badouralix/curl-jq - uuid: "5" taskType: sh command: - echo {{ kurtosis.4.code }} {{ kurtosis.4.output }} image: badouralix/curl-jq ``` ## Is this change user facing? NO ## References: The Enclave Manager uses this plan yaml to render packages in the Enclave Builder: #2250 --------- Co-authored-by: Ben Gazzard <[email protected]>

🤖 I have created a release *beep* *boop* --- ## [0.88.6](0.88.5...0.88.6) (2024-03-13) ### Features * define `build_args` in `ImageBuildSpec` ([#2257](#2257)) ([9812375](9812375)) * generate plan yamls ([#2177](#2177)) ([2292010](e157686)) * port guessing based on port number for docker transpiler ([#2281](#2281)) ([5c98ce2](3a32230)) * support package nodes in enclave builder ui ([#2283](#2283)) ([7fa27ed](0b2395d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: kurtosisbot <[email protected]>

tedim52 added 10 commits February 15, 2024 15:43

setup plan yaml generator

400f08b

checkpoint

ffdf6cc

add scaffolding

d22b7d2

add get arguments to kurtosis instruction interface

46fae9d

adjust plan yaml generator to get service config

1b437e4

add postgres package test

f4b3152

start render templates impl

908ad7d

finish render templates impl

a4e7031

fix render templates impl

03a7535

impl upload files

aef291e

tedim52 marked this pull request as draft February 17, 2024 18:08

tedim52 added 18 commits February 17, 2024 13:45

start impl run sh

458fd75

adjust task yaml def

9f5f442

impl run sh checkpoint

52f003c

finish run sh impl

eff877f

impl run python

32b53b1

commento out tests for now

ee35dda

impl store service

f8c6939

change order of name and uuid

5ecf11a

catch service config err

6146fc1

add plan yaml endpoint to protobuf

9daaabc

regenerate proto bindings

29b3f2c

implement plan yaml retrieval

27acec6

impl plan yaml endpoint in enclave mngr

247f6cc

fix protobuf

316f182

impl endpoints in gateway

4ab95ea

Merge branch 'main' into tedi/dryrun

fd8ac0c

fmt

8a67c71

tidy

1972b43

touch up

321ad2f

tedim52 changed the title ~~feat: generate plan yamls from dry run~~ feat: generate plan yamls Mar 11, 2024

tedim52 requested a review from laurentluce March 12, 2024 13:09

tedim52 added 3 commits March 12, 2024 10:01

Merge branch 'main' into tedi/dryrun

97ddc8f

fix duplicate field

a6a13b3

Merge branch 'main' into tedi/dryrun

4cad9bb

h4ck3rk3y approved these changes Mar 12, 2024

View reviewed changes

tedim52 added 2 commits March 13, 2024 10:04

address comments and fix merge conflicts

5a8a6f2

Merge branch 'main' into tedi/dryrun

3ca4954

tedim52 enabled auto-merge March 13, 2024 14:29

tedim52 added this pull request to the merge queue Mar 13, 2024

tedim52 removed this pull request from the merge queue due to a manual request Mar 13, 2024

tedim52 added 2 commits March 13, 2024 14:16

refactor files artifacts expansion to file mount logic

6cfc873

Merge branch 'main' into tedi/dryrun

2f40aee

tedim52 enabled auto-merge March 13, 2024 18:18

tedim52 added this pull request to the merge queue Mar 13, 2024

Merged via the queue into main with commit 2292010 Mar 13, 2024
51 of 52 checks passed

tedim52 deleted the tedi/dryrun branch March 13, 2024 18:41

kurtosisbot mentioned this pull request Mar 13, 2024

chore(main): release 0.88.6 #2278

Merged

laurentluce removed their request for review March 13, 2024 18:46

kurtosisbot mentioned this pull request Nov 1, 2024

chore(main): release kurtosis 2.0.0 #2589

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: generate plan yamls #2177

feat: generate plan yamls #2177

tedim52 commented Feb 17, 2024 •

edited

Loading

gitguardian bot commented Feb 17, 2024 •

edited

Loading

tedim52 commented Mar 11, 2024

tedim52 commented Mar 11, 2024

h4ck3rk3y left a comment

feat: generate plan yamls #2177

feat: generate plan yamls #2177

Conversation

tedim52 commented Feb 17, 2024 • edited Loading

Description:

Is this change user facing?

References:

gitguardian bot commented Feb 17, 2024 • edited Loading

⚠️ GitGuardian has uncovered 39 secrets following the scan of your pull request.

tedim52 commented Mar 11, 2024

tedim52 commented Mar 11, 2024

h4ck3rk3y left a comment

Choose a reason for hiding this comment

tedim52 commented Feb 17, 2024 •

edited

Loading

gitguardian bot commented Feb 17, 2024 •

edited

Loading