Skip to content
This repository has been archived by the owner on Oct 9, 2023. It is now read-only.

Add rate limiter configs for Presto plugin #75

Merged
merged 8 commits into from
Apr 3, 2020
Merged

Add rate limiter configs for Presto plugin #75

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4525b4a
Add rate limiter configs
Apr 3, 2020
19260a0
add extern storage location
Apr 3, 2020
89c6e4c
lint
Apr 3, 2020
ca41639
undo
Apr 3, 2020
c73a25e
undo2
Apr 3, 2020
3d36ebd
update field description
Apr 3, 2020
e003060
Update go/tasks/plugins/presto/config/config.go
lu4nm3 Apr 3, 2020
b362458
add comments in field description for rate limiter
Apr 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions go/tasks/plugins/presto/client/prestostatus_enumer.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 11 additions & 1 deletion go/tasks/plugins/presto/config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,11 @@ type RefreshCacheConfig struct {
LruCacheSize int `json:"lruCacheSize" pflag:",Size of the cache"`
}

type RateLimiterConfig struct {
Rate int64 `json:"rate" pflag:",Allowed rate of calls per second."`
Burst int `json:"burst" pflag:",Allowed burst rate of calls per second."`
}

var (
defaultConfig = Config{
Environment: URLMustParse(""),
Expand All @@ -52,6 +57,10 @@ var (
Workers: 15,
LruCacheSize: 10000,
},
RateLimiterConfig: RateLimiterConfig{
Rate: 15,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What will this be used for? Will it gate all calls to Mozart? Or just Presto? Will it be for only creation calls? Or both creation calls and all the status checking calls in the auto-refresh cache as well? Can you add some comments about how best to calculate a good number for this config?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be used for all calls going to Mozart only for Presto. Sure I can add some comments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In terms of the values themselves, I sort of took them from the awsbatch rate limiter. I'm sure we can adjust them later based on how the Presto cluster behaves

Burst: 20,
},
}

prestoConfigSection = pluginsConfig.MustRegisterSubSection(prestoConfigSectionKey, &defaultConfig)
Expand All @@ -63,7 +72,8 @@ type Config struct {
DefaultRoutingGroup string `json:"defaultRoutingGroup" pflag:",Default Presto routing group"`
DefaultUser string `json:"defaultUser" pflag:",Default Presto user"`
RoutingGroupConfigs []RoutingGroupConfig `json:"routingGroupConfigs" pflag:"-,A list of cluster configs. Each of the configs corresponds to a service cluster"`
RefreshCacheConfig RefreshCacheConfig `json:"refreshCacheConfig" pflag:"Rate limiter config"`
RefreshCacheConfig RefreshCacheConfig `json:"refreshCacheConfig" pflag:"Refresh cache config"`
RateLimiterConfig RateLimiterConfig `json:"rateLimiterConfig" pflag:"Rate limiter config for ALL requests going to Presto"`
}

// Retrieves the current config value or default.
Expand Down