Skip to content

Commit

Permalink
Add SPLUNK_MEMORY_TOTAL_MIB (open-telemetry#48)
Browse files Browse the repository at this point in the history 
Today, SPLUNK_BALLAST_SIZE_MIB needs to be computed and manually
configured by a user. Most users do not know or care about the ballast
and under normal circumstances they should not need to. Users will
understand the total amount of memory allocated to the Collector. As
such, introduce a SPLUNK_MEMORY_TOTAL_MIB env var that can be used
instead of or in addition to SPLUNK_BALLAST_SIZE_MIB. This has the added
bonus that now memory spike and limit env vars are no longer required
for non-linux systems. This change is backwards compatible.

A few other minor enhancements:

- Memory spike/limit env vars are now applied even if SPLUNK_CONFIG is
  passed
- Add logging for custom behavior so configuration is clear
- Add accessible endpoints to README
  • Loading branch information
@flands
flands authored Feb 2, 2021
1 parent b8f9aa8 commit adc00a6
Show file tree
Hide file tree
Showing 3 changed files with 281 additions and 29 deletions.
46 changes: 31 additions & 15 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,19 +41,35 @@ which require the following environment variables:

- `SPLUNK_REALM` (no default): Which realm to send the data to (for example: `us0`)
- `SPLUNK_ACCESS_TOKEN` (no default): Access token to authenticate requests
- `SPLUNK_BALLAST_SIZE_MIB` (no default): How much memory to allocate to the ballast. This should be set to 1/3 to 1/2 of configured memory.
- `SPLUNK_MEMORY_TOTAL_MIB` (no default): Total memory allocated to the Collector.

In addition, the following environment variables are optional:
The following environment variables are optional:

- `SPLUNK_CONFIG` (default = `/etc/otel/collector/splunk-config_linux.yaml`): Which configuration to load.
- `SPLUNK_MEMORY_LIMIT_PERCENTAGE` (default = `90`): Maximum total memory to be allocated by the process heap.
- `SPLUNK_MEMORY_SPIKE_PERCENTAGE` (default = `20`): Maximum spike between the measurements of memory usage.

When running on a non-linux system, the following environment variables are required:

- `SPLUNK_CONFIG` (default = `/etc/otel/collector/splunk-config_non_linux.yaml`): Configuration to load.
- `SPLUNK_MEMORY_LIMIT_MIB` (no default): Maximum total memory to be allocated by the process heap.
- `SPLUNK_MEMORY_SPIKE_MIB` (no default): Maximum spike between the measurements of memory usage.
- `SPLUNK_BALLAST_SIZE_MIB` (no default): How much memory to allocate to the ballast.
- For Linux systems:
- `SPLUNK_MEMORY_LIMIT_PERCENTAGE` (default = `90`): Maximum total memory to be allocated by the process heap.
- `SPLUNK_MEMORY_SPIKE_PERCENTAGE` (default = `20`): Maximum spike between the measurements of memory usage.
- For non-Linux systems:
- `SPLUNK_MEMORY_LIMIT_MIB` (no default): Maximum total memory to be allocated by the process heap.
- `SPLUNK_MEMORY_SPIKE_MIB` (no default): Maximum spike between the measurements of memory usage.

> `SPLUNK_MEMORY_TOTAL_MIB` automatically configures the ballast, memory limit,
> and memory spike. If the optional environment variables are defined, they
> will override the value calculated from `SPLUNK_MEMORY_TOTAL_MIB`.
With the Collector configured, the following endpoints are accessible:

- `http(s)://<collectorFQDN>:13133/` Health endpoint useful for load balancer monitoring
- `http(s)://<collectorFQDN>:[14250|14268]` Jaeger [gRPC|Thrift HTTP] receiver
- `http(s)://<collectorFQDN>:55678` OpenCensus gRPC and HTTP receiver
- `http(s)://localhost:55679/debug/[tracez|pipelinez]` zPages monitoring
- `http(s)://<collectorFQDN>:55680` OpenTelemetry gRPC receiver
- `http(s)://<collectorFQDN>:6060` HTTP Forwarder used to receive Smart Agent `apiUrl` data
- `http(s)://<collectorFQDN>:7276` SignalFx Infrastructure Monitoring gRPC receiver
- `http(s)://localhost:8888/metrics` Prometheus metrics for the Collector
- `http(s)://<collectorFQDN>:9411/api/[v1|v2]/spans` Zipkin JSON (can be set to proto)receiver
- `http(s)://<collectorFQDN>:9943/v2/trace` SignalFx APM receiver

The following sections describe how to deploy the Collector in supported environments.

Expand All @@ -62,7 +78,7 @@ The following sections describe how to deploy the Collector in supported environ
Deploy from a Docker container. Replace `0.1.0` with the latest stable version number:

```bash
$ docker run --rm -e SPLUNK_ACCESS_TOKEN=12345 -e SPLUNK_BALLAST_SIZE_MIB=683 \
$ docker run --rm -e SPLUNK_ACCESS_TOKEN=12345 -e SPLUNK_MEMORY_TOTAL_MIB=1024 \
-e SPLUNK_REALM=us0 -p 13133:13133 -p 14250:14250 -p 14268:14268 -p 55678-55680:55678-55680 \
-p 6060:6060 -p 7276:7276 -p 8888:8888 -p 9411:9411 -p 9943:9943 \
--name otelcol quay.io/signalfx/splunk-otel-collector:0.1.0
Expand All @@ -80,7 +96,7 @@ file on GitHub.

```bash
$ make otelcol
$ SPLUNK_REALM=us0 SPLUNK_ACCESS_TOKEN=12345 SPLUNK_BALLAST_SIZE_MIB=683 \
$ SPLUNK_REALM=us0 SPLUNK_ACCESS_TOKEN=12345 SPLUNK_MEMORY_TOTAL_MIB=1024 \
./bin/otelcol
```

Expand Down Expand Up @@ -197,7 +213,7 @@ least a CPU core per Collector. Multiple Collectors can deployed behind a
simple round-robin load balancer. Each Collector runs independently, so scale
increases linearly with the number of Collectors you deploy.

The Collector does not persist data to disk so no disk space is required.
> The Collector does not persist data to disk so no disk space is required.
## Advanced Configuration

Expand All @@ -209,7 +225,7 @@ specified. Command line arguments take priority over environment variables.
For example in Docker:

```bash
$ docker run --rm -e SPLUNK_ACCESS_TOKEN=12345 -e SPLUNK_BALLAST_SIZE_MIB=683 \
$ docker run --rm -e SPLUNK_ACCESS_TOKEN=12345 -e SPLUNK_MEMORY_TOTAL_MIB=1024 \
-e SPLUNK_REALM=us0 -p 13133:13133 -p 14250:14250 -p 14268:14268 -p 55678-55680:55678-55680 \
-p 6060:6060 -p 7276:7276 -p 8888:8888 -p 9411:9411 -p 9943:9943 \
--name otelcol quay.io/signalfx/splunk-otel-collector:0.1.0 \
Expand All @@ -230,7 +246,7 @@ the `--config` command line argument to provide a custom configuration.
For example in Docker:

```bash
$ docker run --rm -e SPLUNK_ACCESS_TOKEN=12345 -e SPLUNK_BALLAST_SIZE_MIB=683 \
$ docker run --rm -e SPLUNK_ACCESS_TOKEN=12345 -e SPLUNK_MEMORY_TOTAL_MIB=1024 \
-e SPLUNK_REALM=us0 -e SPLUNK_CONFIG=/etc/collector.yaml -p 13133:13133 -p 14250:14250 \
-p 14268:14268 -p 55678-55680:55678-55680 -p 6060:6060 -p 7276:7276 -p 8888:8888 \
-p 9411:9411 -p 9943:9943 -v collector.yaml:/etc/collector.yaml:ro \
Expand Down
108 changes: 94 additions & 14 deletions cmd/otelcol/main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ const (
memLimitMiBEnvVarName = "SPLUNK_MEMORY_LIMIT_MIB"
memSpikeEnvVarName = "SPLUNK_MEMORY_SPIKE_PERCENTAGE"
memSpikeMiBEnvVarName = "SPLUNK_MEMORY_SPIKE_MIB"
memTotalEnvVarName = "SPLUNK_MEMORY_TOTAL_MIB"
realmEnvVarName = "SPLUNK_REALM"
tokenEnvVarName = "SPLUNK_ACCESS_TOKEN"

Expand All @@ -50,22 +51,50 @@ const (
defaultLocalSAPMNonLinuxConfig = "cmd/otelcol/config/collector/splunk_config_non_linux.yaml"
defaultLocalOTLPLinuxConfig = "cmd/otelcol/config/collector/otlp_config_linux.yaml"
defaultLocalOTLPNonLinuxConfig = "cmd/otelcol/config/collector/otlp_config_non_linux.yaml"
defaultMemoryBallastPercentage = 50
defaultMemoryLimitPercentage = 90
defaultMemoryLimitMaxMiB = 2048
defaultMemorySpikePercentage = 25
defaultMemorySpikeMaxMiB = 2048
)

func main() {
// TODO: Use same format as the collector
log.SetFlags(log.Ldate | log.Ltime | log.Lshortfile)

// Check if the total memory is specified via the env var.
memTotalEnvVarVal := os.Getenv(memTotalEnvVarName)
memTotalSizeMiB := 0
if memTotalEnvVarVal != "" {
// Check if it is a numeric value.
val, err := strconv.Atoi(memTotalEnvVarVal)
if err != nil {
log.Fatalf("Expected a number in %s env variable but got %s", memTotalEnvVarName, memTotalEnvVarVal)
}
if 10 > val {
log.Fatalf("Expected a number greater than 10 for %s env variable but got %s", memTotalEnvVarName, memTotalEnvVarVal)
}
memTotalSizeMiB = val
}

// Check runtime parameters
// Runtime parameters take priority over environment variables
// Runtime parameters are not validated
args := os.Args[1:]
if !contains(args, "--mem-ballast-size-mib") {
useBallastSizeFromEnvVar()
useMemorySizeFromEnvVar(memTotalSizeMiB)
} else {
log.Printf("Ballast CLI argument found, ignoring %s if set", ballastEnvVarName)
}
if !contains(args, "--config") {
useConfigFromEnvVar()
} else {
log.Printf("Config CLI argument found, please ensure memory_limiter settings are correct")
}
if runtime.GOOS == "linux" {
useMemorySettingsPercentageFromEnvVar()
} else {
useMemorySettingsMiBFromEnvVar(memTotalSizeMiB)
}

factories, err := components.Get()
Expand Down Expand Up @@ -98,18 +127,27 @@ func contains(arr []string, str string) bool {
return false
}

func useBallastSizeFromEnvVar() {
func useMemorySizeFromEnvVar(memTotalSizeMiB int) {
// Check if the ballast is specified via the env var.
ballastSize := os.Getenv(ballastEnvVarName)
if ballastSize != "" {
// Check if it is a numeric value.
_, err := strconv.Atoi(ballastSize)
val, err := strconv.Atoi(ballastSize)
if err != nil {
log.Fatalf("Expected a number in %s env variable but got %s", ballastEnvVarName, ballastSize)
}
if 0 > val {
log.Fatalf("Expected a number greater than 0 for %s env variable but got %s", ballastEnvVarName, ballastSize)
}

// Inject the command line flag that controls the ballast size.
os.Args = append(os.Args, "--mem-ballast-size-mib="+ballastSize)
} else if memTotalSizeMiB > 0 {
halfMem := strconv.Itoa(memTotalSizeMiB * defaultMemoryBallastPercentage / 100)
log.Printf("Set ballast to %s MiB", halfMem)
// Inject the command line flag that controls the ballast size.
os.Args = append(os.Args, "--mem-ballast-size-mib="+halfMem)
os.Setenv(ballastEnvVarName, halfMem)
}
}

Expand All @@ -130,7 +168,6 @@ func useConfigFromEnvVar() {
if config == "" {
log.Fatalf("Unable to find the default configuration file, ensure %s environment variable is set properly", configEnvVarName)
}
useMemorySettingsPercentageFromEnvVar()
} else {
_, err := os.Stat(defaultDockerSAPMNonLinuxConfig)
if err == nil {
Expand All @@ -143,7 +180,6 @@ func useConfigFromEnvVar() {
if config == "" {
log.Fatalf("Unable to find the default configuration file, ensure %s environment variable is set properly", configEnvVarName)
}
useMemorySettingsMiBFromEnvVar()
}
} else {
// Check if file exists.
Expand All @@ -165,25 +201,31 @@ func useConfigFromEnvVar() {
defaultLocalOTLPNonLinuxConfig:
// The following environment variables are required.
// If any are missing stop here.
requiredEnvVars := []string{ballastEnvVarName, realmEnvVarName, tokenEnvVarName}
requiredEnvVars := []string{realmEnvVarName, tokenEnvVarName}
for _, v := range requiredEnvVars {
if len(os.Getenv(v)) == 0 {
log.Printf("Usage: %s=12345 %s=us0 %s=684 %s", tokenEnvVarName, realmEnvVarName, ballastEnvVarName, os.Args[0])
log.Printf("Usage: %s=12345 %s=us0 %s=1024 %s", tokenEnvVarName, realmEnvVarName, memTotalEnvVarName, os.Args[0])
log.Fatalf("ERROR: Missing environment variable %s", v)
}
}
// Needed for backwards compatibility
if len(os.Getenv(memTotalEnvVarName)) == 0 && len(os.Getenv(ballastEnvVarName)) == 0 {
log.Printf("Usage: %s=12345 %s=us0 %s=1024 %s", tokenEnvVarName, realmEnvVarName, memTotalEnvVarName, os.Args[0])
log.Fatalf("ERROR: Missing environment variable %s", memTotalEnvVarName)
}
}

// Inject the command line flag that controls the configuration.
os.Args = append(os.Args, "--config="+config)
}

func checkMemorySettingsMiBFromEnvVar(envVar string) int {
func checkMemorySettingsMiBFromEnvVar(envVar string, memTotalSizeMiB int) int {
// Check if the memory limit is specified via the env var
// Ensure memory limit is valid
var envVarResult int
var envVarResult int = 0
envVarVal := os.Getenv(envVar)
if envVarVal != "" {
switch {
case envVarVal != "":
// Check if it is a numeric value.
val, err := strconv.Atoi(envVarVal)
if err != nil {
Expand All @@ -193,16 +235,54 @@ func checkMemorySettingsMiBFromEnvVar(envVar string) int {
log.Fatalf("Expected a number greater than 0 for %s env variable but got %s", envVar, envVarVal)
}
envVarResult = val
} else {
case memTotalSizeMiB > 0:
break
default:
log.Printf("Usage: %s=12345 %s=us0 %s=684 %s=1024 %s=256 %s", tokenEnvVarName, realmEnvVarName, ballastEnvVarName, memLimitMiBEnvVarName, memSpikeMiBEnvVarName, os.Args[0])
log.Fatalf("ERROR: Missing environment variable %s", envVar)
}
return envVarResult
}

func useMemorySettingsMiBFromEnvVar() {
memLimit := checkMemorySettingsMiBFromEnvVar(memLimitMiBEnvVarName)
memSpike := checkMemorySettingsMiBFromEnvVar(memSpikeMiBEnvVarName)
func useMemorySettingsMiBFromEnvVar(memTotalSizeMiB int) {
// Check if memory limit is specified via environment variable
memLimit := checkMemorySettingsMiBFromEnvVar(memLimitMiBEnvVarName, memTotalSizeMiB)
// Use if set, otherwise memory total size must be specified
if memLimit == 0 {
if memTotalSizeMiB == 0 {
panic("PANIC: Both memory limit MiB and memory total size are set to zero. This should never happen.")
}
// If not set, compute based on memory total size specified
// and default memory limit percentage const
memLimitMiB := memTotalSizeMiB * defaultMemoryLimitPercentage / 100
// The memory limit should be set to defaultMemoryLimitPercentage of total memory
// while reserving a maximum of defaultMemoryLimitMaxMiB of memory.
if (memTotalSizeMiB - memLimitMiB) < defaultMemoryLimitMaxMiB {
memLimit = memLimitMiB
} else {
memLimit = (memTotalSizeMiB - defaultMemoryLimitMaxMiB)
}
log.Printf("Set memory limit to %d MiB", memLimit)
}
// Check if memory spike is specified via environment variable
memSpike := checkMemorySettingsMiBFromEnvVar(memSpikeMiBEnvVarName, memTotalSizeMiB)
// Use if set, otherwise memory total size must be specified
if memSpike == 0 {
if memTotalSizeMiB == 0 {
panic("PANIC: Both memory limit MiB and memory total size are set to zero. This should never happen.")
}
// If not set, compute based on memory total size specified
// and default memory spike percentage const
memSpikeMiB := memTotalSizeMiB * defaultMemorySpikePercentage / 100
// The memory spike should be set to defaultMemorySpikePercentage of total memory
// while specifying a maximum of defaultMemorySpikeMaxMiB of memory.
if memSpikeMiB < defaultMemorySpikeMaxMiB {
memSpike = memSpikeMiB
} else {
memSpike = defaultMemorySpikeMaxMiB
}
log.Printf("Set memory spike limit to %d MiB", memSpike)
}
setMemorySettingsToEnvVar(memLimit, memLimitMiBEnvVarName, memSpike, memSpikeMiBEnvVarName)
}

Expand Down
Loading

0 comments on commit adc00a6

Please sign in to comment.