Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Added feature to run rendering in synchronous mode. #40

Merged
merged 12 commits into from
Aug 18, 2023

Conversation

fritzduchardt
Copy link
Collaborator

This makes debugging easier and takes load of slow machines. It has proven useful in the past.

@fritzduchardt fritzduchardt requested a review from Zebradil August 7, 2023 14:25
…debugging easier and takes load of slow machines.
@fritzduchardt fritzduchardt force-pushed the add_synchronous_processing branch from cc2b81d to 3cbd585 Compare August 7, 2023 14:33
Copy link
Member

@Zebradil Zebradil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about a flag to control the level of parallelism instead of just switching it on and off?

  • --async=0 to disable
  • --async=1 to enable
  • --async=8 to run maximum of 8 parallel threads

For now, we can skip implementing the last one, but I would like to settle with the flag name.

internal/myks/util_test.go Outdated Show resolved Hide resolved
@fritzduchardt fritzduchardt requested a review from Zebradil August 11, 2023 06:26

// Create the destination file
dstFile, err := os.Create(destination)
if err != nil {
return err
}
defer dstFile.Close()
defer saveClose(dstFile)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error is logged but not returned.
Named result values are your friend :) https://blog.learngoprogramming.com/5-gotchas-of-defer-in-go-golang-part-iii-36a1ab3d6ef1

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, using them now, but got to say, they are NOT my friends looking at the complexity of the resulting code:

saveClose := func(srcFile fs.File) {
		closeErr := srcFile.Close()
		if closeErr != nil {
			if err == nil {
				err = closeErr
			} else {
				err = errors.Join(err, closeErr)
			}
		}
	}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, what about:

saveClose := func(srcFile fs.File) {
	closeErr := srcFile.Close()
	err = errors.Join(err, closeErr)
}

errors.Join is ignoring nil errors. If all errors are nil it'll return nil.
Downside you always have to unwrap the error.

This commit adds support for asynchronous processing in the `process` function. It allows tasks to be executed concurrently based on the specified `asyncLevel`. Additionally, test cases have been added to ensure the correct execution of the function.
@fritzduchardt fritzduchardt requested a review from kbudde August 12, 2023 09:00
cmd/root.go Outdated
@@ -68,6 +69,7 @@ var rootCmd = &cobra.Command{
func init() {
cobra.OnInitialize(initConfig)
rootCmd.PersistentFlags().StringP("log-level", "l", "info", "Set the logging level")
rootCmd.PersistentFlags().IntVarP(&asyncLevel, "async", "a", 1, "Run asynchronously. Activated by default. Set to 0 to deactivate.")
Copy link
Member

@kbudde kbudde Aug 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now as async controls the number of parallel routines, I would propose to set some better default.
1 is the same as 0 just using different code.
What about 4 or 8?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about inverting the logic a little bit and doing the following:

  • async level is N means that maximum N applications are being processed in parallel,
  • async level is 0 means the limit is not set, any number of applications can be processed in parallel.

In this case, the default might be 0. I see it hard to set a viable default value. Moreover, for different operations (sync, render) the "ideal" number would differ.

// run async
if asyncLevel > 0 {
var eg errgroup.Group
semaphore := make(chan struct{}, asyncLevel)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be replaced by eg.SetLimit(asyncLevel). No semaphore needed in this case :)
https://pkg.go.dev/golang.org/x/sync/errgroup#Group.SetLimit

@kbudde
Copy link
Member

kbudde commented Aug 17, 2023

@fritzduchardt , @Zebradil I added one commit. I think it can be merged now.

@Zebradil Zebradil merged commit 144f5fd into release-2.0 Aug 18, 2023
@Zebradil Zebradil deleted the add_synchronous_processing branch August 18, 2023 22:30
@Zebradil Zebradil mentioned this pull request Aug 19, 2023
mykso-bot added a commit that referenced this pull request Sep 19, 2023
# [1.3.0](v1.2.0...v1.3.0) (2023-09-19)

### Bug Fixes

* Add documentation the myks sync step ([#38](#38)) ([e61a10c](e61a10c)), closes [#37](#37)
* apply smart mode logic only to supported commands ([#83](#83)) ([2bc754f](2bc754f))
* argocd source plugin config type in schema ([520156d](520156d))
* cleanup vendir folder ([#90](#90)) ([a20df1a](a20df1a))
* consistent behavior on rendering ALL applications ([#79](#79)) ([2aab516](2aab516))
* correct sources for the global-ytt rendering step ([#50](#50)) ([5a0e4d7](5a0e4d7))
* create myks data schema file on init and on every run ([#84](#84)) ([976291e](976291e))
* data values of prototype of argocd app ([b5d7ff9](b5d7ff9))
* do not fail on absent rendered directory ([eaf1202](eaf1202))
* do not fail without vendir configs ([2f73cda](2f73cda))
* do not override ArgoCD defaults set by user ([#74](#74)) ([f2cf4ce](f2cf4ce)), closes [#70](#70)
* **docker:** do not build arm64, it is not supported ([3971ae7](3971ae7))
* **docker:** specify full image tag ([f3222e5](f3222e5))
* formatting ([fd65f05](fd65f05))
* generate ArgoCD secret only if enabled ([4b3ed11](4b3ed11))
* helm value file merge ([#33](#33)) ([3c9c0ea](3c9c0ea)), closes [#32](#32)
* init Globe core attributes earlier ([#85](#85)) ([20c48fd](20c48fd))
* log errors during vendir sync ([5dc1b5e](5dc1b5e))
* make render errors appear in the log with full error message ([c325da2](c325da2))
* process map keys instead of values ([3b86a03](3b86a03))
* reduce usage of pointers to cope with race conditions ([#88](#88)) ([d734933](d734933))
* search in the default envs directory ([ef4a75e](ef4a75e))
* skip helm rendering ([80a8eb5](80a8eb5))
* **smart-mode:** detect changes when myks root is in subdirectory ([0522b67](0522b67))
* update data-schema.ytt.yaml according to the latest Myks changes ([5ef9d34](5ef9d34))
* use ArgoCD application path relatively to git root ([92f0617](92f0617))

### Features

* add a finalizer to ArgoCD project CR ([acf67fd](acf67fd))
* add argocd-apps prototype ([6772744](6772744))
* add arm binaries ([a74d63e](a74d63e))
* add common overlays example to assets ([39965c5](39965c5))
* add example environment configs ([8edba12](8edba12))
* add flag to control parallelism ([#40](#40)) ([144f5fd](144f5fd))
* add git branch detection and refactor data schema ([05e41d4](05e41d4))
* add init command and a data schema file ([c11c27a](c11c27a))
* add prototypes in the init command ([f31471e](f31471e))
* add step for rendering ytt packages ([#36](#36)) ([d1078c6](d1078c6))
* Add vendir authentication via environment ([b0c50c2](b0c50c2))
* Add vendir sync caching ([24ff41c](24ff41c))
* Added docker image ([ae8988d](ae8988d))
* Added Smart Mode that Automatically detects changed Environment… ([#62](#62)) ([e404b6b](e404b6b))
* always write data-schema file ([fa83bee](fa83bee))
* ArgoCD support ([#41](#41)) ([e45d585](e45d585))
* configure ArgoCD Application finalizers and source.plugin ([#56](#56)) ([80940aa](80940aa))
* create initial .myks.yaml and print configs ([#87](#87)) ([215ccd3](215ccd3))
* detect additional and missing applications ([#89](#89)) ([2c7e101](2c7e101))
* do not convert git URL protocol ([2823eb6](2823eb6))
* dump configuration as ytt values ([af65436](af65436))
* fail on non existing apps ([#52](#52)) ([87aafa3](87aafa3)), closes [#3](#3)
* fine-grained ArgoCD project destination ([04d3b78](04d3b78))
* get git repo URL ([c9b726a](c9b726a))
* **helm:** add support for helm capabilities ([#48](#48)) ([1a13ee1](1a13ee1)), closes [#31](#31)
* **init:** allow overwriting of data ([#49](#49)) ([f3f5983](f3f5983))
* provide argocd-specific configuration with prototypes ([06e5e5c](06e5e5c))
* provide example default values for all environments ([0f9cab6](0f9cab6))
* Push images to docker hub and ghcr ([#65](#65)) ([10bdc63](10bdc63))
* Refactoring to make log output more intelligible. ([#39](#39)) ([71cd34c](71cd34c))
* **smart-mode:** configuration option for smart-mode base revision ([#95](#95)) ([4400184](4400184))
* **smart-mode:** precisely select envs and apps for processing ([#96](#96)) ([ffb47ad](ffb47ad))
* support multiple content items in vendir configs ([#92](#92)) ([fc50be0](fc50be0))
* tweak prefix logic on argo cr to allow for project names like "… ([#72](#72)) ([af01180](af01180))
* validate root directory ([f81b719](f81b719))
* vendir sync caching ([7279cc7](7279cc7))

### Performance Improvements

* **docker:** ignore not needed files ([3479804](3479804))
mykso-bot added a commit that referenced this pull request Sep 19, 2023
# [2.0.0](v1.2.0...v2.0.0) (2023-09-19)

### Bug Fixes

* Add documentation the myks sync step ([#38](#38)) ([e61a10c](e61a10c)), closes [#37](#37)
* apply smart mode logic only to supported commands ([#83](#83)) ([2bc754f](2bc754f))
* argocd source plugin config type in schema ([520156d](520156d))
* cleanup vendir folder ([#90](#90)) ([a20df1a](a20df1a))
* consistent behavior on rendering ALL applications ([#79](#79)) ([2aab516](2aab516))
* correct sources for the global-ytt rendering step ([#50](#50)) ([5a0e4d7](5a0e4d7))
* create myks data schema file on init and on every run ([#84](#84)) ([976291e](976291e))
* data values of prototype of argocd app ([b5d7ff9](b5d7ff9))
* do not fail on absent rendered directory ([eaf1202](eaf1202))
* do not fail without vendir configs ([2f73cda](2f73cda))
* do not override ArgoCD defaults set by user ([#74](#74)) ([f2cf4ce](f2cf4ce)), closes [#70](#70)
* **docker:** do not build arm64, it is not supported ([3971ae7](3971ae7))
* **docker:** specify full image tag ([f3222e5](f3222e5))
* formatting ([fd65f05](fd65f05))
* generate ArgoCD secret only if enabled ([4b3ed11](4b3ed11))
* helm value file merge ([#33](#33)) ([3c9c0ea](3c9c0ea)), closes [#32](#32)
* init Globe core attributes earlier ([#85](#85)) ([20c48fd](20c48fd))
* log errors during vendir sync ([5dc1b5e](5dc1b5e))
* make render errors appear in the log with full error message ([c325da2](c325da2))
* process map keys instead of values ([3b86a03](3b86a03))
* reduce usage of pointers to cope with race conditions ([#88](#88)) ([d734933](d734933))
* search in the default envs directory ([ef4a75e](ef4a75e))
* skip helm rendering ([80a8eb5](80a8eb5))
* **smart-mode:** detect changes when myks root is in subdirectory ([0522b67](0522b67))
* update data-schema.ytt.yaml according to the latest Myks changes ([5ef9d34](5ef9d34))
* use ArgoCD application path relatively to git root ([92f0617](92f0617))

### Features

* add a finalizer to ArgoCD project CR ([acf67fd](acf67fd))
* add argocd-apps prototype ([6772744](6772744))
* add arm binaries ([a74d63e](a74d63e))
* add common overlays example to assets ([39965c5](39965c5))
* add example environment configs ([8edba12](8edba12))
* add flag to control parallelism ([#40](#40)) ([144f5fd](144f5fd))
* add git branch detection and refactor data schema ([05e41d4](05e41d4))
* add init command and a data schema file ([c11c27a](c11c27a))
* add prototypes in the init command ([f31471e](f31471e))
* add step for rendering ytt packages ([#36](#36)) ([d1078c6](d1078c6))
* Add vendir authentication via environment ([b0c50c2](b0c50c2))
* Add vendir sync caching ([24ff41c](24ff41c))
* Added docker image ([ae8988d](ae8988d))
* Added Smart Mode that Automatically detects changed Environment… ([#62](#62)) ([e404b6b](e404b6b))
* always write data-schema file ([fa83bee](fa83bee))
* ArgoCD support ([#41](#41)) ([e45d585](e45d585))
* configure ArgoCD Application finalizers and source.plugin ([#56](#56)) ([80940aa](80940aa))
* create initial .myks.yaml and print configs ([#87](#87)) ([215ccd3](215ccd3))
* detect additional and missing applications ([#89](#89)) ([2c7e101](2c7e101))
* do not convert git URL protocol ([2823eb6](2823eb6))
* dump configuration as ytt values ([af65436](af65436))
* fail on non existing apps ([#52](#52)) ([87aafa3](87aafa3)), closes [#3](#3)
* fine-grained ArgoCD project destination ([04d3b78](04d3b78))
* get git repo URL ([c9b726a](c9b726a))
* **helm:** add support for helm capabilities ([#48](#48)) ([1a13ee1](1a13ee1)), closes [#31](#31)
* **init:** allow overwriting of data ([#49](#49)) ([f3f5983](f3f5983))
* provide argocd-specific configuration with prototypes ([06e5e5c](06e5e5c))
* provide example default values for all environments ([0f9cab6](0f9cab6))
* Push images to docker hub and ghcr ([#65](#65)) ([10bdc63](10bdc63))
* Refactoring to make log output more intelligible. ([#39](#39)) ([71cd34c](71cd34c))
* release 2.0 ([b7b486d](b7b486d))
* **smart-mode:** configuration option for smart-mode base revision ([#95](#95)) ([4400184](4400184))
* **smart-mode:** precisely select envs and apps for processing ([#96](#96)) ([ffb47ad](ffb47ad))
* support multiple content items in vendir configs ([#92](#92)) ([fc50be0](fc50be0))
* tweak prefix logic on argo cr to allow for project names like "… ([#72](#72)) ([af01180](af01180))
* validate root directory ([f81b719](f81b719))
* vendir sync caching ([7279cc7](7279cc7))

### Performance Improvements

* **docker:** ignore not needed files ([3479804](3479804))

### BREAKING CHANGES

* release 2.0
This is an empty commit to trigger a major release.
mykso-bot added a commit that referenced this pull request Sep 19, 2023
# [2.0.0](v1.2.0...v2.0.0) (2023-09-19)

### Bug Fixes

* Add documentation the myks sync step ([#38](#38)) ([e61a10c](e61a10c)), closes [#37](#37)
* apply smart mode logic only to supported commands ([#83](#83)) ([2bc754f](2bc754f))
* argocd source plugin config type in schema ([520156d](520156d))
* cleanup vendir folder ([#90](#90)) ([a20df1a](a20df1a))
* consistent behavior on rendering ALL applications ([#79](#79)) ([2aab516](2aab516))
* correct sources for the global-ytt rendering step ([#50](#50)) ([5a0e4d7](5a0e4d7))
* create myks data schema file on init and on every run ([#84](#84)) ([976291e](976291e))
* data values of prototype of argocd app ([b5d7ff9](b5d7ff9))
* do not fail on absent rendered directory ([eaf1202](eaf1202))
* do not fail without vendir configs ([2f73cda](2f73cda))
* do not override ArgoCD defaults set by user ([#74](#74)) ([f2cf4ce](f2cf4ce)), closes [#70](#70)
* **docker:** do not build arm64, it is not supported ([3971ae7](3971ae7))
* **docker:** specify full image tag ([f3222e5](f3222e5))
* formatting ([fd65f05](fd65f05))
* generate ArgoCD secret only if enabled ([4b3ed11](4b3ed11))
* helm value file merge ([#33](#33)) ([3c9c0ea](3c9c0ea)), closes [#32](#32)
* init Globe core attributes earlier ([#85](#85)) ([20c48fd](20c48fd))
* log errors during vendir sync ([5dc1b5e](5dc1b5e))
* make render errors appear in the log with full error message ([c325da2](c325da2))
* process map keys instead of values ([3b86a03](3b86a03))
* reduce usage of pointers to cope with race conditions ([#88](#88)) ([d734933](d734933))
* search in the default envs directory ([ef4a75e](ef4a75e))
* skip helm rendering ([80a8eb5](80a8eb5))
* **smart-mode:** detect changes when myks root is in subdirectory ([0522b67](0522b67))
* update data-schema.ytt.yaml according to the latest Myks changes ([5ef9d34](5ef9d34))
* use ArgoCD application path relatively to git root ([92f0617](92f0617))

### Features

* add a finalizer to ArgoCD project CR ([acf67fd](acf67fd))
* add argocd-apps prototype ([6772744](6772744))
* add arm binaries ([a74d63e](a74d63e))
* add common overlays example to assets ([39965c5](39965c5))
* add example environment configs ([8edba12](8edba12))
* add flag to control parallelism ([#40](#40)) ([144f5fd](144f5fd))
* add git branch detection and refactor data schema ([05e41d4](05e41d4))
* add init command and a data schema file ([c11c27a](c11c27a))
* add prototypes in the init command ([f31471e](f31471e))
* add step for rendering ytt packages ([#36](#36)) ([d1078c6](d1078c6))
* Add vendir authentication via environment ([b0c50c2](b0c50c2))
* Add vendir sync caching ([24ff41c](24ff41c))
* Added docker image ([ae8988d](ae8988d))
* Added Smart Mode that Automatically detects changed Environment… ([#62](#62)) ([e404b6b](e404b6b))
* always write data-schema file ([fa83bee](fa83bee))
* ArgoCD support ([#41](#41)) ([e45d585](e45d585))
* configure ArgoCD Application finalizers and source.plugin ([#56](#56)) ([80940aa](80940aa))
* create initial .myks.yaml and print configs ([#87](#87)) ([215ccd3](215ccd3))
* detect additional and missing applications ([#89](#89)) ([2c7e101](2c7e101))
* do not convert git URL protocol ([2823eb6](2823eb6))
* dump configuration as ytt values ([af65436](af65436))
* fail on non existing apps ([#52](#52)) ([87aafa3](87aafa3)), closes [#3](#3)
* fine-grained ArgoCD project destination ([04d3b78](04d3b78))
* get git repo URL ([c9b726a](c9b726a))
* **helm:** add support for helm capabilities ([#48](#48)) ([1a13ee1](1a13ee1)), closes [#31](#31)
* **init:** allow overwriting of data ([#49](#49)) ([f3f5983](f3f5983))
* provide argocd-specific configuration with prototypes ([06e5e5c](06e5e5c))
* provide example default values for all environments ([0f9cab6](0f9cab6))
* Push images to docker hub and ghcr ([#65](#65)) ([10bdc63](10bdc63))
* Refactoring to make log output more intelligible. ([#39](#39)) ([71cd34c](71cd34c))
* release 2.0 ([b7b486d](b7b486d))
* **smart-mode:** configuration option for smart-mode base revision ([#95](#95)) ([4400184](4400184))
* **smart-mode:** precisely select envs and apps for processing ([#96](#96)) ([ffb47ad](ffb47ad))
* support multiple content items in vendir configs ([#92](#92)) ([fc50be0](fc50be0))
* tweak prefix logic on argo cr to allow for project names like "… ([#72](#72)) ([af01180](af01180))
* validate root directory ([f81b719](f81b719))
* vendir sync caching ([7279cc7](7279cc7))

### Performance Improvements

* **docker:** ignore not needed files ([3479804](3479804))

### BREAKING CHANGES

* release 2.0
This is an empty commit to trigger a major release.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants