Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding S3 Compatible Storage Support and Changing "timeout" to "frequency" #2

Merged
merged 10 commits into from
Sep 21, 2020
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ If your configuration is right and Vault is running on the same host as the agen

`retain` The number of backups to retain.

`timeout` How often to run the snapshot agent. Examples: `30s`, `1h`. See https://golang.org/pkg/time/#ParseDuration for a full list of valid time units.
`frequency` How often to run the snapshot agent. Examples: `30s`, `1h`. See https://golang.org/pkg/time/#ParseDuration for a full list of valid time units.

`role_id` Specifies the role_id used to call the Vault API. See the authentication steps below.

Expand All @@ -71,7 +71,7 @@ Note that if you specify more than one storage option, *all* options will be wri

`local_storage` - Object for writing to a file on disk.

`aws_storage` - Object for writing to an S3 bucket.
`aws_storage` - Object for writing to an S3 bucket (Support AWS S3 but also S3 Compatible Storage).

`google_storage` - Object for writing to GCS.

Expand All @@ -87,6 +87,10 @@ Note that if you specify more than one storage option, *all* options will be wri

`secret_access_key` - Recommended to use the standard `SECRET_ACCESS_KEY` env var, but its possible to specify this in the config

`s3_endpoint` - S3 compatible storage endpoint (ex: http://127.0.0.1:9000)

`s3_force_path_style` - Needed if your S3 Compatible storage support only path-style or you would like to use S3's FIPS Endpoint.

`s3_region` - S3 region as is required for programmatic interaction with AWS

`s3_bucket` - bucket to store snapshots in (required for AWS writes to work)
Expand Down
20 changes: 11 additions & 9 deletions config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ import (

// Configuration is the overall config object
type Configuration struct {
Address string `json:"addr"`
Retain int64 `json:"retain"`
Timeout string `json:"freq"`
AWS S3Config `json:"aws_storage"`
Local LocalConfig `json:"local_storage"`
GCP GCPConfig `json:"google_storage"`
Azure AzureConfig `json:"azure_storage"`
RoleID string `json:"role_id"`
SecretID string `json:"secret_id"`
Address string `json:"addr"`
Retain int64 `json:"retain"`
Frequency string `json:"frequency"`
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the formatter needs to be run here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be good

AWS S3Config `json:"aws_storage"`
Local LocalConfig `json:"local_storage"`
GCP GCPConfig `json:"google_storage"`
Azure AzureConfig `json:"azure_storage"`
RoleID string `json:"role_id"`
SecretID string `json:"secret_id"`
}

// AzureConfig is the configuration for Azure blob snapshots
Expand All @@ -44,11 +44,13 @@ type S3Config struct {
Uploader *s3manager.Uploader
AccessKeyID string `json:"access_key_id"`
SecretAccessKey string `json:"secret_access_key"`
Endpoint string `json:"s3_endpoint"`
Region string `json:"s3_region"`
Bucket string `json:"s3_bucket"`
KeyPrefix string `json:"s3_key_prefix"`
SSE bool `json:"s3_server_side_encryption"`
StaticSnapshotName string `json:"s3_static_snapshot_name"`
S3ForcePathStyle bool `json:"s3_force_path_style"`
}

// ReadConfig reads the configuration file
Expand Down
6 changes: 3 additions & 3 deletions main.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,10 @@ func main() {
}

snapshotter, err := snapshot_agent.NewSnapshotter(c)
timeout, err := time.ParseDuration(c.Timeout)
frequency, err := time.ParseDuration(c.Frequency)

if err != nil {
timeout = time.Hour
frequency = time.Hour
}

currentIP, err := getInstanceIP()
Expand Down Expand Up @@ -90,7 +90,7 @@ func main() {
}
}
select {
case <-time.After(timeout):
case <-time.After(frequency):
continue
case <-done:
os.Exit(1)
Expand Down
8 changes: 8 additions & 0 deletions snapshot_agent/agent.go
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,14 @@ func (s *Snapshotter) ConfigureS3(config *config.Configuration) error {
awsConfig.Credentials = credentials.NewStaticCredentials(config.AWS.AccessKeyID, config.AWS.SecretAccessKey, "")
}

if config.AWS.Endpoint != "" {
awsConfig.Endpoint = aws.String(config.AWS.Endpoint)
}

if config.AWS.S3ForcePathStyle != false {
awsConfig.S3ForcePathStyle = aws.Bool(config.AWS.S3ForcePathStyle)
}

sess := session.Must(session.NewSession(awsConfig))
s.S3Client = s3.New(sess)
s.Uploader = s3manager.NewUploader(sess)
Expand Down