Skip to content

Commit

Permalink
Introduce profiles and the promscale profile
Browse files Browse the repository at this point in the history
A profile is an optional "mode" to put the tuner in that tailors the recommendations for a particular workload.
If no profile is specified, the tuner is run with a default profile which provides exactly the same behavior as
before and corresponds to the typical timescaledb workload.

The only other valid profile defined at the moment is "promscale".

The profile is specified using either the `--profile` argument or the `TSTUNE_PROFILE` environment variable. The environment variable overrides the command line argument if the command line argument is set to blank. In this way, the environment variable can be passed to `docker run`, and we therefore do not need different docker images per profile.

The promscale profile needs to make recommendations for a couple of postgres settings that are measured in units of time. This PR adds functionality to timescaledb-tune to be able to interpret and handle settings measured in time. This was not as straightforward as was hoped because how postgres interprets a setting's value depends upon:

1. whether the setting is a real or integer
2. the default units
3. whether units were specified in the input and whether those units are larger, smaller, or equal to the defaults
4. whether the input was a fractional value

With the addition of profiles, it is possible for one profile to provide recommendations for settings which a different profile simply ignores. This PR had to provide a way for a Recommender to signal that it was providing "no recommendation" for a given setting.

A SettingsGroup now returns an appropriate Recommender based upon the selected profile. A NullRecommender can be provided in the case that no recommendations are provided for the selected profile and SettingsGroup. A NullRecommender provides no recommendation for all settings passed to it.

SettingsGroups can now have a mixture of settings that are measured in time, bytes, or unit-less. The FloatParser implementations and logic had to change to accommodate this.
  • Loading branch information
jgpruitt committed Aug 29, 2022
1 parent 0d781e1 commit 03b980c
Show file tree
Hide file tree
Showing 25 changed files with 1,607 additions and 200 deletions.
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,14 @@ that you accepted from the prompts.

#### Other invocations

By default, timescaledb-tune provides recommendations for a typical timescaledb workload. The `--profile` flag can be
used to tailor the recommendations for other workload types. Currently, the only non-default profile is "promscale".
The `TSTUNE_PROFILE` environment variable can also be used to affect this behavior.

```bash
$ timescaledb-tune --profile promscale
```

If you want recommendations for a specific amount of memory and/or CPUs:
```bash
$ timescaledb-tune --memory="4GB" --cpus=2
Expand Down
7 changes: 7 additions & 0 deletions cmd/timescaledb-tune/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,16 @@ func init() {
flag.BoolVar(&f.UseColor, "color", true, "Use color in output (works best on dark terminals)")
flag.BoolVar(&f.DryRun, "dry-run", false, "Whether to just show the changes without overwriting the configuration file")
flag.BoolVar(&f.Restore, "restore", false, "Whether to restore a previously made conf file backup")
flag.StringVar(&f.Profile, "profile", "", "a specific \"mode\" for tailoring recommendations to a special workload type. If blank or unspecified, a default is used unless the TSTUNE_PROFILE environment variable is set. Valid values: \"promscale\"")

flag.BoolVar(&showVersion, "version", false, "Show the version of this tool")
flag.Parse()

// the TSTUNE_PROFILE environment variable overrides the --profile flag if the --profile is blank or unset
// this is designed for docker usage
if val := os.Getenv("TSTUNE_PROFILE"); val != "" && f.Profile == "" {
f.Profile = val
}
}

func main() {
Expand Down
212 changes: 200 additions & 12 deletions internal/parse/parse.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ import (
"math"
"regexp"
"strconv"
"strings"
"time"
)

// Byte equivalents (using 1024) of common byte measurements
Expand All @@ -19,23 +21,93 @@ const (

// Suffixes for byte measurements that are valid to PostgreSQL
const (
TB = "TB" // terabyte
GB = "GB" // gigabyte
MB = "MB" // megabyte
KB = "kB" // kilobyte
B = "" // no unit, therefore: bytes
TB = "TB" // terabyte
GB = "GB" // gigabyte
MB = "MB" // megabyte
KB = "kB" // kilobyte
B = "" // no unit, therefore: bytes
)

// TimeUnit represents valid suffixes for time measurements used by PostgreSQL settings
// https://www.postgresql.org/docs/current/config-setting.html#20.1.1.%20Parameter%20Names%20and%20Values
type TimeUnit int64

const (
Microseconds TimeUnit = iota
Milliseconds
Seconds
Minutes
Hours
Days
)

func (tu TimeUnit) String() string {
switch tu {
case Microseconds:
return "us"
case Milliseconds:
return "ms"
case Seconds:
return "s"
case Minutes:
return "min"
case Hours:
return "h"
case Days:
return "d"
default:
return "unrecognized"
}
}

func ParseTimeUnit(val string) (TimeUnit, error) {
switch strings.ToLower(val) {
case "us":
return Microseconds, nil
case "ms":
return Milliseconds, nil
case "s":
return Seconds, nil
case "min":
return Minutes, nil
case "h":
return Hours, nil
case "d":
return Days, nil
default:
return TimeUnit(-1), fmt.Errorf(errUnrecognizedTimeUnitsFmt, val)
}
}

// VarType represents the values from the vartype column of the pg_settings table
type VarType int64

const (
VarTypeReal VarType = iota
VarTypeInteger
)

func (v VarType) String() string {
switch v {
case VarTypeReal:
return "real"
case VarTypeInteger:
return "integer"
default:
return "unrecognized"
}
}

const (
errIncorrectFormatFmt = "incorrect PostgreSQL bytes format: '%s'"
errCouldNotParseBytesFmt = "could not parse bytes number: %v"
errCouldNotParseVersionFmt = "unable to parse PG version string: %s"
errUnknownMajorVersionFmt = "unknown major PG version: %s"
errIncorrectBytesFormatFmt = "incorrect PostgreSQL bytes format: '%s'"
errIncorrectTimeFormatFmt = "incorrect PostgreSQL time format: '%s'"
errCouldNotParseBytesFmt = "could not parse bytes number: %v"
errUnrecognizedTimeUnitsFmt = "unrecognized time units: %s"
)

var (
pgBytesRegex = regexp.MustCompile("^([0-9]+)((?:k|M|G|T)B)?$")
pgVersionRegex = regexp.MustCompile("^PostgreSQL ([0-9]+?).([0-9]+?).*")
pgBytesRegex = regexp.MustCompile("^(?:')?([0-9]+)((?:k|M|G|T)B)?(?:')?$")
pgTimeRegex = regexp.MustCompile(`^(?:')?([0-9]+(\.[0-9]+)?)(?:\s*)(us|ms|s|min|h|d)?(?:')?$`)
)

func parseIntToFloatUnits(bytes uint64) (float64, string) {
Expand Down Expand Up @@ -98,7 +170,7 @@ func BytesToPGFormat(bytes uint64) string {
func PGFormatToBytes(val string) (uint64, error) {
res := pgBytesRegex.FindStringSubmatch(val)
if len(res) != 3 {
return 0.0, fmt.Errorf(errIncorrectFormatFmt, val)
return 0.0, fmt.Errorf(errIncorrectBytesFormatFmt, val)
}
num, err := strconv.ParseInt(res[1], 10, 64)
if err != nil {
Expand All @@ -121,3 +193,119 @@ func PGFormatToBytes(val string) (uint64, error) {
}
return ret, nil
}

func NextSmallerTimeUnits(units TimeUnit) TimeUnit {
switch units {
case Microseconds:
return Microseconds
case Milliseconds:
return Microseconds
case Seconds:
return Milliseconds
case Minutes:
return Seconds
case Hours:
return Minutes
default: // Days
return Hours
}
}

func UnitsToDuration(units TimeUnit) time.Duration {
switch units {
case Microseconds:
return time.Microsecond
case Milliseconds:
return time.Millisecond
case Seconds:
return time.Second
case Minutes:
return time.Minute
case Hours:
return time.Hour
case Days:
return 24 * time.Hour
default:
return time.Nanosecond
}
}

func TimeConversion(fromUnits, toUnits TimeUnit) (float64, error) {
return float64(UnitsToDuration(fromUnits)) / float64(UnitsToDuration(toUnits)), nil
}

func PGFormatToTime(val string, defaultUnits TimeUnit, vt VarType) (float64, TimeUnit, error) {
// the default units, whether units were specified, and the variable type ALL impact how the value is interpreted
// https://www.postgresql.org/docs/current/config-setting.html#20.1.1.%20Parameter%20Names%20and%20Values

// select unit, vartype, array_agg(name) from pg_settings where unit in ('us', 'ms', 's', 'm', 'h', 'd') group by 1, 2;

// parse it
res := pgTimeRegex.FindStringSubmatch(val)
if res == nil || len(res) < 2 {
return -1.0, TimeUnit(-1), fmt.Errorf(errIncorrectTimeFormatFmt, val)
}

// extract the numeric portion
v, err := strconv.ParseFloat(res[1], 64)
if err != nil {
return -1.0, TimeUnit(-1), fmt.Errorf(errIncorrectTimeFormatFmt, val)
}

// extract the units or use the default
unitsWereUnspecified := true
units := defaultUnits
if len(res) >= 4 && res[3] != "" {
unitsWereUnspecified = false
units, err = ParseTimeUnit(res[3])
if err != nil {
return -1.0, TimeUnit(-1), err
}
}

convert := func(v float64, units TimeUnit, vt VarType) (float64, TimeUnit, error) {
if _, fract := math.Modf(v); fract < math.Nextafter(0.0, 1.0) {
// not distinguishable as a fractional value
switch vt {
case VarTypeInteger:
return math.Trunc(v), units, nil
case VarTypeReal:
return v, units, nil
}
} else {
// IS a fractional value. it had a decimal component
toUnits := NextSmallerTimeUnits(units)
if err != nil {
return -1.0, TimeUnit(-1), err
}
conv, err := TimeConversion(units, toUnits)
if err != nil {
return -1.0, TimeUnit(-1), err
}
return math.Round(v * conv), toUnits, nil
}
return -1.0, TimeUnit(-1), fmt.Errorf(errIncorrectTimeFormatFmt, val)
}

if unitsWereUnspecified {
switch vt {
case VarTypeInteger:
return math.Round(v), units, nil
case VarTypeReal:
return convert(v, units, vt)
}
} else /* units WERE specified */ {
if units == defaultUnits {
switch vt {
case VarTypeInteger:
return math.Round(v), units, nil
case VarTypeReal:
return convert(v, units, vt)
}
} else /* specified units are different from the default units */ {
return convert(v, units, vt)
}
}
// should never get here!
return -1.0, TimeUnit(-1), fmt.Errorf(errIncorrectTimeFormatFmt, val)
}
Loading

0 comments on commit 03b980c

Please sign in to comment.