feat: Added Smart Mode that Automatically detects changed Environment… #62

fritzduchardt · 2023-08-24T15:52:02Z

…s and Applications

kbudde · 2023-08-24T17:34:55Z

At a first look, this looks very good and powerful 😎.

…s and Applications

Zebradil

Wow, that's solid!

But could we get a description of this mode? To better understand edge cases and application areas. For example, what if there are several base branches (main←development←feature)? What if I want to apply smart mode locally for all commits on my feature branch (like it's done for CI)?

We will then be able to use it for help and documentation.

Zebradil · 2023-08-25T20:59:52Z

cmd/root.go

+			if targetEnvironments == nil && targetApplications == nil {
+				log.Warn().Err(err).Msg("Smart Mode does not identify changes. Rendering everything.")
+			}


Would it make sense to skip processing if nothing changed? To process forcefully, we can add a flag or use positional arguments.

myks sync --force # or myks sync ALL

Was thinking about that..let's do it.

Zebradil · 2023-08-25T21:19:36Z

internal/myks/git.go

+func extractChangedFilePathsWithStatus(cfs []ChangedFile, status string) []string {
+	var paths []string
+	for _, cf := range cfs {
+		if status == "" || cf.status == status {
+			paths = append(paths, cf.path)
+		}
+	}
+	return paths
+}
+
+func extractChangedFilePathsWithoutStatus(cfs []ChangedFile, status string) []string {
+	var paths []string
+	for _, cf := range cfs {
+		if status == "" || cf.status != status {
+			paths = append(paths, cf.path)
+		}
+	}
+	return paths
+}


Checking for status can be moved out of the loop.

I think checking for status should be done outside the function.

I'd use a more generic function, which accepts a test callback:

func filter[T any](ss []T, test func(T) bool) (ret []T) { for _, s := range ss { if test(s) { ret = append(ret, s) } } return }

(borrowed from here)

But I'm not blocking you with this suggestion, we can refactor this later.

Like it, functions are first class citizens in go after all

Zebradil · 2023-08-25T21:37:32Z

internal/myks/globe.go

-					g.environments[path] = env
-				} else {
-					log.Warn().Str("path", path).Msg("Unable to collect environment, skipping")
+			if path != g.EnvironmentBaseDir {


Is it to remove the warning from logs?

yes. Otherwise we always get a warning on the root level env-data.ytt.yaml

I think we should decrease the log level instead. Otherwise, we'll still have this annoying message for nested groups.

Sure, lets set it to debug

Zebradil · 2023-08-25T21:44:25Z

internal/myks/globe.go

+		log.Warn().Msg("Unable to find data schema file, creating one")
+		if err := os.MkdirAll(filepath.Dir(dataSchemaFileName), 0o750); err != nil {
+			log.Fatal().Err(err).Msg("Unable to create data schema file directory")
+		}
 		if err := os.WriteFile(dataSchemaFileName, dataSchema, 0o600); err != nil {


I think thins won't be necessary after merging from release-2.0.

internal/myks/globe.go

…s and Applications

# Conflicts: # internal/myks/globe.go

elyesbenamor · 2023-09-04T13:39:21Z

@fritzduchardt for auto mode we have also some issue rendering and detecting rendered files
Bug:

render an application
remove the rendered folder from the environment
try to render again to see if it will detect the removed directory
1:31PM INF Setting log level to: info 1:31PM INF [global] Smart mode detected changes in environments: [], applications: [] 1:31PM WRN Smart Mode did not find any changes. Existing.
all in all it will not detect the removed directory and it wont render anything

fritzduchardt · 2023-09-04T15:42:08Z

@fritzduchardt for auto mode we have also some issue rendering and detecting rendered files Bug:

render an application

remove the rendered folder from the environment

try to render again to see if it will detect the removed directory
1:31PM INF Setting log level to: info 1:31PM INF [global] Smart mode detected changes in environments: [], applications: [] 1:31PM WRN Smart Mode did not find any changes. Existing.
all in all it will not detect the removed directory and it wont render anything

Fair point - will look into it

kbudde · 2023-09-05T17:52:16Z

@fritzduchardt for auto mode we have also some issue rendering and detecting rendered files Bug:

render an application

remove the rendered folder from the environment

try to render again to see if it will detect the removed directory
1:31PM INF Setting log level to: info 1:31PM INF [global] Smart mode detected changes in environments: [], applications: [] 1:31PM WRN Smart Mode did not find any changes. Existing.
all in all it will not detect the removed directory and it wont render anything

Fair point - will look into it

Let's not over complicate the smart mode.
I think this can be solved in #63

fritzduchardt · 2023-09-06T15:40:33Z

@fritzduchardt for auto mode we have also some issue rendering and detecting rendered files Bug:

render an application

remove the rendered folder from the environment

try to render again to see if it will detect the removed directory
1:31PM INF Setting log level to: info 1:31PM INF [global] Smart mode detected changes in environments: [], applications: [] 1:31PM WRN Smart Mode did not find any changes. Existing.
all in all it will not detect the removed directory and it wont render anything

Fair point - will look into it

Let's not over complicate the smart mode. I think this can be solved in #63

Agreed - a stab at coding this reveals that the complexity increase would be substantial, especially, because the env structure in the rendered folder is flattened. So, once @Zebradil gives me his well, I am happy to merge this.

Zebradil

🚀

# [1.3.0](v1.2.0...v1.3.0) (2023-09-19) ### Bug Fixes * Add documentation the myks sync step ([#38](#38)) ([e61a10c](e61a10c)), closes [#37](#37) * apply smart mode logic only to supported commands ([#83](#83)) ([2bc754f](2bc754f)) * argocd source plugin config type in schema ([520156d](520156d)) * cleanup vendir folder ([#90](#90)) ([a20df1a](a20df1a)) * consistent behavior on rendering ALL applications ([#79](#79)) ([2aab516](2aab516)) * correct sources for the global-ytt rendering step ([#50](#50)) ([5a0e4d7](5a0e4d7)) * create myks data schema file on init and on every run ([#84](#84)) ([976291e](976291e)) * data values of prototype of argocd app ([b5d7ff9](b5d7ff9)) * do not fail on absent rendered directory ([eaf1202](eaf1202)) * do not fail without vendir configs ([2f73cda](2f73cda)) * do not override ArgoCD defaults set by user ([#74](#74)) ([f2cf4ce](f2cf4ce)), closes [#70](#70) * **docker:** do not build arm64, it is not supported ([3971ae7](3971ae7)) * **docker:** specify full image tag ([f3222e5](f3222e5)) * formatting ([fd65f05](fd65f05)) * generate ArgoCD secret only if enabled ([4b3ed11](4b3ed11)) * helm value file merge ([#33](#33)) ([3c9c0ea](3c9c0ea)), closes [#32](#32) * init Globe core attributes earlier ([#85](#85)) ([20c48fd](20c48fd)) * log errors during vendir sync ([5dc1b5e](5dc1b5e)) * make render errors appear in the log with full error message ([c325da2](c325da2)) * process map keys instead of values ([3b86a03](3b86a03)) * reduce usage of pointers to cope with race conditions ([#88](#88)) ([d734933](d734933)) * search in the default envs directory ([ef4a75e](ef4a75e)) * skip helm rendering ([80a8eb5](80a8eb5)) * **smart-mode:** detect changes when myks root is in subdirectory ([0522b67](0522b67)) * update data-schema.ytt.yaml according to the latest Myks changes ([5ef9d34](5ef9d34)) * use ArgoCD application path relatively to git root ([92f0617](92f0617)) ### Features * add a finalizer to ArgoCD project CR ([acf67fd](acf67fd)) * add argocd-apps prototype ([6772744](6772744)) * add arm binaries ([a74d63e](a74d63e)) * add common overlays example to assets ([39965c5](39965c5)) * add example environment configs ([8edba12](8edba12)) * add flag to control parallelism ([#40](#40)) ([144f5fd](144f5fd)) * add git branch detection and refactor data schema ([05e41d4](05e41d4)) * add init command and a data schema file ([c11c27a](c11c27a)) * add prototypes in the init command ([f31471e](f31471e)) * add step for rendering ytt packages ([#36](#36)) ([d1078c6](d1078c6)) * Add vendir authentication via environment ([b0c50c2](b0c50c2)) * Add vendir sync caching ([24ff41c](24ff41c)) * Added docker image ([ae8988d](ae8988d)) * Added Smart Mode that Automatically detects changed Environment… ([#62](#62)) ([e404b6b](e404b6b)) * always write data-schema file ([fa83bee](fa83bee)) * ArgoCD support ([#41](#41)) ([e45d585](e45d585)) * configure ArgoCD Application finalizers and source.plugin ([#56](#56)) ([80940aa](80940aa)) * create initial .myks.yaml and print configs ([#87](#87)) ([215ccd3](215ccd3)) * detect additional and missing applications ([#89](#89)) ([2c7e101](2c7e101)) * do not convert git URL protocol ([2823eb6](2823eb6)) * dump configuration as ytt values ([af65436](af65436)) * fail on non existing apps ([#52](#52)) ([87aafa3](87aafa3)), closes [#3](#3) * fine-grained ArgoCD project destination ([04d3b78](04d3b78)) * get git repo URL ([c9b726a](c9b726a)) * **helm:** add support for helm capabilities ([#48](#48)) ([1a13ee1](1a13ee1)), closes [#31](#31) * **init:** allow overwriting of data ([#49](#49)) ([f3f5983](f3f5983)) * provide argocd-specific configuration with prototypes ([06e5e5c](06e5e5c)) * provide example default values for all environments ([0f9cab6](0f9cab6)) * Push images to docker hub and ghcr ([#65](#65)) ([10bdc63](10bdc63)) * Refactoring to make log output more intelligible. ([#39](#39)) ([71cd34c](71cd34c)) * **smart-mode:** configuration option for smart-mode base revision ([#95](#95)) ([4400184](4400184)) * **smart-mode:** precisely select envs and apps for processing ([#96](#96)) ([ffb47ad](ffb47ad)) * support multiple content items in vendir configs ([#92](#92)) ([fc50be0](fc50be0)) * tweak prefix logic on argo cr to allow for project names like "… ([#72](#72)) ([af01180](af01180)) * validate root directory ([f81b719](f81b719)) * vendir sync caching ([7279cc7](7279cc7)) ### Performance Improvements * **docker:** ignore not needed files ([3479804](3479804))

# [2.0.0](v1.2.0...v2.0.0) (2023-09-19) ### Bug Fixes * Add documentation the myks sync step ([#38](#38)) ([e61a10c](e61a10c)), closes [#37](#37) * apply smart mode logic only to supported commands ([#83](#83)) ([2bc754f](2bc754f)) * argocd source plugin config type in schema ([520156d](520156d)) * cleanup vendir folder ([#90](#90)) ([a20df1a](a20df1a)) * consistent behavior on rendering ALL applications ([#79](#79)) ([2aab516](2aab516)) * correct sources for the global-ytt rendering step ([#50](#50)) ([5a0e4d7](5a0e4d7)) * create myks data schema file on init and on every run ([#84](#84)) ([976291e](976291e)) * data values of prototype of argocd app ([b5d7ff9](b5d7ff9)) * do not fail on absent rendered directory ([eaf1202](eaf1202)) * do not fail without vendir configs ([2f73cda](2f73cda)) * do not override ArgoCD defaults set by user ([#74](#74)) ([f2cf4ce](f2cf4ce)), closes [#70](#70) * **docker:** do not build arm64, it is not supported ([3971ae7](3971ae7)) * **docker:** specify full image tag ([f3222e5](f3222e5)) * formatting ([fd65f05](fd65f05)) * generate ArgoCD secret only if enabled ([4b3ed11](4b3ed11)) * helm value file merge ([#33](#33)) ([3c9c0ea](3c9c0ea)), closes [#32](#32) * init Globe core attributes earlier ([#85](#85)) ([20c48fd](20c48fd)) * log errors during vendir sync ([5dc1b5e](5dc1b5e)) * make render errors appear in the log with full error message ([c325da2](c325da2)) * process map keys instead of values ([3b86a03](3b86a03)) * reduce usage of pointers to cope with race conditions ([#88](#88)) ([d734933](d734933)) * search in the default envs directory ([ef4a75e](ef4a75e)) * skip helm rendering ([80a8eb5](80a8eb5)) * **smart-mode:** detect changes when myks root is in subdirectory ([0522b67](0522b67)) * update data-schema.ytt.yaml according to the latest Myks changes ([5ef9d34](5ef9d34)) * use ArgoCD application path relatively to git root ([92f0617](92f0617)) ### Features * add a finalizer to ArgoCD project CR ([acf67fd](acf67fd)) * add argocd-apps prototype ([6772744](6772744)) * add arm binaries ([a74d63e](a74d63e)) * add common overlays example to assets ([39965c5](39965c5)) * add example environment configs ([8edba12](8edba12)) * add flag to control parallelism ([#40](#40)) ([144f5fd](144f5fd)) * add git branch detection and refactor data schema ([05e41d4](05e41d4)) * add init command and a data schema file ([c11c27a](c11c27a)) * add prototypes in the init command ([f31471e](f31471e)) * add step for rendering ytt packages ([#36](#36)) ([d1078c6](d1078c6)) * Add vendir authentication via environment ([b0c50c2](b0c50c2)) * Add vendir sync caching ([24ff41c](24ff41c)) * Added docker image ([ae8988d](ae8988d)) * Added Smart Mode that Automatically detects changed Environment… ([#62](#62)) ([e404b6b](e404b6b)) * always write data-schema file ([fa83bee](fa83bee)) * ArgoCD support ([#41](#41)) ([e45d585](e45d585)) * configure ArgoCD Application finalizers and source.plugin ([#56](#56)) ([80940aa](80940aa)) * create initial .myks.yaml and print configs ([#87](#87)) ([215ccd3](215ccd3)) * detect additional and missing applications ([#89](#89)) ([2c7e101](2c7e101)) * do not convert git URL protocol ([2823eb6](2823eb6)) * dump configuration as ytt values ([af65436](af65436)) * fail on non existing apps ([#52](#52)) ([87aafa3](87aafa3)), closes [#3](#3) * fine-grained ArgoCD project destination ([04d3b78](04d3b78)) * get git repo URL ([c9b726a](c9b726a)) * **helm:** add support for helm capabilities ([#48](#48)) ([1a13ee1](1a13ee1)), closes [#31](#31) * **init:** allow overwriting of data ([#49](#49)) ([f3f5983](f3f5983)) * provide argocd-specific configuration with prototypes ([06e5e5c](06e5e5c)) * provide example default values for all environments ([0f9cab6](0f9cab6)) * Push images to docker hub and ghcr ([#65](#65)) ([10bdc63](10bdc63)) * Refactoring to make log output more intelligible. ([#39](#39)) ([71cd34c](71cd34c)) * release 2.0 ([b7b486d](b7b486d)) * **smart-mode:** configuration option for smart-mode base revision ([#95](#95)) ([4400184](4400184)) * **smart-mode:** precisely select envs and apps for processing ([#96](#96)) ([ffb47ad](ffb47ad)) * support multiple content items in vendir configs ([#92](#92)) ([fc50be0](fc50be0)) * tweak prefix logic on argo cr to allow for project names like "… ([#72](#72)) ([af01180](af01180)) * validate root directory ([f81b719](f81b719)) * vendir sync caching ([7279cc7](7279cc7)) ### Performance Improvements * **docker:** ignore not needed files ([3479804](3479804)) ### BREAKING CHANGES * release 2.0 This is an empty commit to trigger a major release.

fritzduchardt requested a review from Zebradil August 24, 2023 15:52

feat: Added Smart Mode that Automatically detects changed Environment…

6fe1ca6

…s and Applications

fritzduchardt force-pushed the smart-mode branch from 9fac770 to 6fe1ca6 Compare August 25, 2023 12:03

Zebradil reviewed Aug 25, 2023

View reviewed changes

Fritz Durchardt added 3 commits August 28, 2023 09:52

feat: Added Smart Mode that Automatically detects changed Environment…

3bf945f

…s and Applications

Merge remote-tracking branch 'origin/smart-mode' into smart-mode

1b3f1e9

# Conflicts: # internal/myks/globe.go

Added documentation for smart mode

7868abb

fritzduchardt requested a review from Zebradil September 1, 2023 13:40

Fritz Durchardt added 2 commits September 3, 2023 11:20

Fix Formatting

c6e58c3

Fix Linting

b87486a

Simplification of code with slices package

61dc3ea

Zebradil approved these changes Sep 7, 2023

View reviewed changes

Zebradil merged commit e404b6b into release-2.0 Sep 7, 2023

Zebradil deleted the smart-mode branch September 7, 2023 05:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Added Smart Mode that Automatically detects changed Environment… #62

feat: Added Smart Mode that Automatically detects changed Environment… #62

fritzduchardt commented Aug 24, 2023

kbudde commented Aug 24, 2023

Zebradil left a comment

Zebradil Aug 25, 2023

fritzduchardt Aug 28, 2023

Zebradil Aug 25, 2023

Zebradil Aug 25, 2023

fritzduchardt Aug 28, 2023

Zebradil Aug 25, 2023

fritzduchardt Aug 28, 2023

Zebradil Aug 28, 2023

fritzduchardt Sep 1, 2023

Zebradil Aug 25, 2023

elyesbenamor commented Sep 4, 2023

fritzduchardt commented Sep 4, 2023

kbudde commented Sep 5, 2023

fritzduchardt commented Sep 6, 2023

Zebradil left a comment

feat: Added Smart Mode that Automatically detects changed Environment… #62

feat: Added Smart Mode that Automatically detects changed Environment… #62

Conversation

fritzduchardt commented Aug 24, 2023

kbudde commented Aug 24, 2023

Zebradil left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elyesbenamor commented Sep 4, 2023

fritzduchardt commented Sep 4, 2023

kbudde commented Sep 5, 2023

fritzduchardt commented Sep 6, 2023

Zebradil left a comment

Choose a reason for hiding this comment