-
Notifications
You must be signed in to change notification settings - Fork 249
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added two new construct patterns: aws-events-rule-kinesisstream and aws-events-rule-kinesisfirehose-s3 #92
Changes from 9 commits
fe2cdbf
e854aa6
55fce09
902c6d2
5bf05cc
39381d1
ed925a7
714ba6b
b2472a1
e544a87
d374ed1
4d30b8c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
lib/*.js | ||
test/*.js | ||
*.d.ts | ||
coverage | ||
test/lambda/index.js |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
lib/*.js | ||
test/*.js | ||
!test/lambda/* | ||
*.js.map | ||
*.d.ts | ||
node_modules | ||
*.generated.ts | ||
dist | ||
.jsii | ||
|
||
.LAST_BUILD | ||
.nyc_output | ||
coverage | ||
.nycrc | ||
.LAST_PACKAGE | ||
*.snk |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
# Exclude typescript source and config | ||
*.ts | ||
tsconfig.json | ||
coverage | ||
.nyc_output | ||
*.tgz | ||
*.snk | ||
*.tsbuildinfo | ||
|
||
# Include javascript files and typescript declarations | ||
!*.js | ||
!*.d.ts | ||
|
||
# Exclude jsii outdir | ||
dist | ||
|
||
# Include .jsii | ||
!.jsii | ||
|
||
# Include .jsii | ||
!.jsii |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,97 @@ | ||
# aws-events-rule-kinesisfirehose-s3 module | ||
<!--BEGIN STABILITY BANNER--> | ||
|
||
--- | ||
|
||
![Stability: Experimental](https://img.shields.io/badge/stability-Experimental-important.svg?style=for-the-badge) | ||
|
||
> All classes are under active development and subject to non-backward compatible changes or removal in any | ||
> future version. These are not subject to the [Semantic Versioning](https://semver.org/) model. | ||
> This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package. | ||
|
||
--- | ||
<!--END STABILITY BANNER--> | ||
|
||
| **Reference Documentation**:| <span style="font-weight: normal">https://docs.aws.amazon.com/solutions/latest/constructs/</span>| | ||
|:-------------|:-------------| | ||
<div style="height:8px"></div> | ||
|
||
| **Language** | **Package** | | ||
|:-------------|-----------------| | ||
|![Python Logo](https://docs.aws.amazon.com/cdk/api/latest/img/python32.png) Python|`aws_solutions_constructs.aws_events_rule_kinesisfirehose_s3`| | ||
|![Typescript Logo](https://docs.aws.amazon.com/cdk/api/latest/img/typescript32.png) Typescript|`@aws-solutions-constructs/aws-events-rule-kinesisfirehose-s3`| | ||
|![Java Logo](https://docs.aws.amazon.com/cdk/api/latest/img/java32.png) Java|`software.amazon.awsconstructs.services.eventsrulekinesisfirehoses3`| | ||
|
||
This AWS Solutions Construct implements an Amazon CloudWatch Events rule to send data to an Amazon Kinesis Data Firehose delivery stream connected to an Amazon S3 bucket. | ||
|
||
Here is a minimal deployable pattern definition in Typescript: | ||
|
||
``` javascript | ||
import { EventsRuleToKinesisFirehoseToS3, EventsRuleToKinesisFirehoseToS3Props } from '@aws-solutions-constructs/aws-events-rule-kinesisfirehose-s3'; | ||
|
||
const props: EventsRuleToKinesisFirehoseToS3Props = { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When this code is copied into the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
eventRuleProps: { | ||
schedule: events.Schedule.rate(cdk.Duration.minutes(5)) | ||
}; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Missing closing bracket
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
|
||
new EventsRuleToKinesisFirehoseToS3(this, 'test-events-rule-firehose-s3', props); | ||
|
||
``` | ||
|
||
## Initializer | ||
|
||
``` text | ||
new EventsRuleToKinesisFirehoseToS3(scope: Construct, id: string, props: EventsRuleToKinesisFirehoseToS3Props); | ||
``` | ||
|
||
_Parameters_ | ||
|
||
* scope [`Construct`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_core.Construct.html) | ||
* id `string` | ||
* props [`EventsRuleToKinesisFirehoseToS3Props`](#pattern-construct-props) | ||
|
||
## Pattern Construct Props | ||
|
||
| **Name** | **Type** | **Description** | | ||
|:-------------|:----------------|-----------------| | ||
|eventRuleProps|[`events.RuleProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-events.RuleProps.html)|User provided eventRuleProps to override the defaults.| | ||
|kinesisFirehoseProps?|[`kinesisfirehose.CfnDeliveryStreamProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesisfirehose.CfnDeliveryStreamProps.html)|Optional user provided props to override the default props for Kinesis Firehose Delivery Stream| | ||
|existingBucketObj?|[`s3.IBucket`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.IBucket.html)|Existing instance of S3 Bucket object, if this is set then the bucketProps is ignored.| | ||
|bucketProps?|[`s3.BucketProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.BucketProps.html)|User provided props to override the default props for the S3 Bucket.| | ||
|
||
## Pattern Properties | ||
|
||
| **Name** | **Type** | **Description** | | ||
|:-------------|:----------------|-----------------| | ||
|eventsRule|[`events.Rule`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-events.Rule.html)|Returns an instance of events.Rule created by the construct.| | ||
|kinesisFirehose|[`kinesisfirehose.CfnDeliveryStream`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesisfirehose.CfnDeliveryStream.html)|Returns an instance of kinesisfirehose.CfnDeliveryStream created by the construct| | ||
|s3Bucket?|[`s3.Bucket`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.Bucket.html)|Returns an instance of s3.Bucket created by the construct| | ||
|s3LoggingBucket?|[`s3.Bucket`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.Bucket.html)|Returns an instance of s3.Bucket created by the construct as the logging bucket for the primary bucket.| | ||
|eventsRole|[`iam.Role`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-iam.Role.html)|Returns an instance of the iam.Role created by the construct for Events Rule| | ||
|kinesisFirehoseRole|[`iam.Role`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-iam.Role.html)|Returns an instance of the iam.Role created by the construct for Kinesis Data Firehose delivery stream| | ||
|kinesisFirehoseLogGroup|[`logs.LogGroup`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-logs.LogGroup.html)|Returns an instance of the LogGroup created by the construct for Kinesis Data Firehose delivery stream| | ||
|
||
## Default settings | ||
|
||
Out of the box implementation of the Construct without any override will set the following defaults: | ||
|
||
### Amazon CloudWatch Events Rule | ||
* Configure least privilege access IAM role for Events Rule to publish to the Kinesis Firehose Delivery Stream. | ||
|
||
### Amazon Kinesis Firehose | ||
* Enable CloudWatch logging for Kinesis Firehose | ||
* Configure least privilege access IAM role for Amazon Kinesis Firehose | ||
|
||
### Amazon S3 Bucket | ||
* Configure Access logging for S3 Bucket | ||
* Enable server-side encryption for S3 Bucket using AWS managed KMS Key | ||
* Turn on the versioning for S3 Bucket | ||
* Don't allow public access for S3 Bucket | ||
* Retain the S3 Bucket when deleting the CloudFormation stack | ||
* Applies Lifecycle rule to move noncurrent object versions to Glacier storage after 90 days | ||
|
||
## Architecture | ||
![Architecture Diagram](architecture.png) | ||
|
||
*** | ||
© Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,122 @@ | ||
/** | ||
* Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance | ||
* with the License. A copy of the License is located at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES | ||
* OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions | ||
* and limitations under the License. | ||
*/ | ||
|
||
import * as events from '@aws-cdk/aws-events'; | ||
import * as kinesisfirehose from '@aws-cdk/aws-kinesisfirehose'; | ||
import * as defaults from '@aws-solutions-constructs/core'; | ||
import * as iam from '@aws-cdk/aws-iam'; | ||
import * as s3 from '@aws-cdk/aws-s3'; | ||
import * as logs from '@aws-cdk/aws-logs'; | ||
import { Construct } from '@aws-cdk/core'; | ||
import { overrideProps } from '@aws-solutions-constructs/core'; | ||
import { KinesisFirehoseToS3 } from '@aws-solutions-constructs/aws-kinesisfirehose-s3'; | ||
|
||
/** | ||
* @summary The properties for the EventsRuleToKinesisFirehoseToS3 Construct | ||
*/ | ||
export interface EventsRuleToKinesisFirehoseToS3Props { | ||
/** | ||
* User provided eventRuleProps to override the defaults | ||
* | ||
* @default - None | ||
*/ | ||
readonly eventRuleProps: events.RuleProps | ||
/** | ||
* User provided props to override the default props for the Kinesis Firehose. | ||
* | ||
* @default - Default props are used | ||
*/ | ||
readonly kinesisFirehoseProps?: kinesisfirehose.CfnDeliveryStreamProps | any | ||
/** | ||
* Existing instance of S3 Bucket object, if this is set then the bucketProps is ignored. | ||
* | ||
* @default - None | ||
*/ | ||
readonly existingBucketObj?: s3.IBucket, | ||
/** | ||
* User provided props to override the default props for the S3 Bucket. | ||
* | ||
* @default - Default props are used | ||
*/ | ||
readonly bucketProps?: s3.BucketProps | ||
} | ||
|
||
export class EventsRuleToKinesisFirehoseToS3 extends Construct { | ||
public readonly eventsRule: events.Rule; | ||
public readonly eventsRole: iam.Role; | ||
public readonly kinesisFirehose: kinesisfirehose.CfnDeliveryStream; | ||
public readonly kinesisFirehoseLogGroup: logs.LogGroup; | ||
public readonly kinesisFirehoseRole: iam.Role; | ||
public readonly s3Bucket?: s3.Bucket; | ||
public readonly s3LoggingBucket?: s3.Bucket; | ||
|
||
/** | ||
* @summary Constructs a new instance of the EventsRuleToKinesisFirehoseToS3 class. | ||
* @param {cdk.App} scope - represents the scope for all the resources. | ||
* @param {string} id - this is a a scope-unique id. | ||
* @param {EventsRuleToKinesisFirehoseToS3Props} props - user provided props for the construct | ||
* @since 0.8.0 | ||
* @access public | ||
*/ | ||
constructor(scope: Construct, id: string, props: EventsRuleToKinesisFirehoseToS3Props) { | ||
super(scope, id); | ||
|
||
// Set up the Kinesis Firehose using KinesisFirehoseToS3 construct | ||
const firehoseToS3 = new KinesisFirehoseToS3(this, 'KinesisFirehoseToS3', { | ||
kinesisFirehoseProps: props.kinesisFirehoseProps, | ||
existingBucketObj: props.existingBucketObj, | ||
bucketProps: props.bucketProps | ||
}); | ||
this.kinesisFirehose = firehoseToS3.kinesisFirehose; | ||
this.s3Bucket = firehoseToS3.s3Bucket; | ||
this.kinesisFirehoseRole = firehoseToS3.kinesisFirehoseRole; | ||
this.s3LoggingBucket = firehoseToS3.s3LoggingBucket; | ||
this.kinesisFirehoseLogGroup = firehoseToS3.kinesisFirehoseLogGroup; | ||
|
||
// Create an events service role | ||
this.eventsRole = new iam.Role(this, 'EventsRuleInvokeKinesisFirehoseRole', { | ||
assumedBy: new iam.ServicePrincipal('events.amazonaws.com'), | ||
description: 'Events Rule To Kinesis Firehose Role', | ||
}); | ||
|
||
// Setup the IAM policy that grants events rule the permission to send cw events data to kinesis firehose | ||
const eventsPolicy = new iam.Policy(this, 'EventsRuleInvokeKinesisFirehosePolicy', { | ||
statements: [new iam.PolicyStatement({ | ||
actions: [ | ||
'firehose:PutRecord', | ||
'firehose:PutRecordBatch' | ||
], | ||
resources: [this.kinesisFirehose.attrArn] | ||
}) | ||
]}); | ||
|
||
// Attach policy to role | ||
eventsPolicy.attachToRole(this.eventsRole); | ||
|
||
// Set up the Kinesis Firehose as the target for event rule | ||
const KinesisFirehoseEventTarget: events.IRuleTarget = { | ||
bind: () => ({ | ||
id: '', | ||
arn: this.kinesisFirehose.attrArn, | ||
role: this.eventsRole | ||
}) | ||
}; | ||
|
||
// Set up the events rule props | ||
const defaultEventsRuleProps = defaults.DefaultEventsRuleProps([KinesisFirehoseEventTarget]); | ||
const eventsRuleProps = overrideProps(defaultEventsRuleProps, props.eventRuleProps, true); | ||
|
||
this.eventsRule = new events.Rule(this, 'EventsRule', eventsRuleProps); | ||
|
||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,87 @@ | ||
{ | ||
"name": "@aws-solutions-constructs/aws-events-rule-kinesisfirehose-s3", | ||
"version": "1.67.0", | ||
"description": "CDK Constructs for Amazon CloudWatch Events Rule to Amazon Kinesis Firehose to Amazon S3 integration.", | ||
"main": "lib/index.js", | ||
"types": "lib/index.d.ts", | ||
"repository": { | ||
"type": "git", | ||
"url": "https://github.com/awslabs/aws-solutions-constructs.git", | ||
"directory": "source/patterns/@aws-solutions-constructs/aws-events-rule-kinesisfirehose-s3" | ||
}, | ||
"author": { | ||
"name": "Amazon Web Services", | ||
"url": "https://aws.amazon.com", | ||
"organization": true | ||
}, | ||
"license": "Apache-2.0", | ||
"scripts": { | ||
"build": "tsc -b .", | ||
"lint": "eslint -c ../eslintrc.yml --ext=.js,.ts . && tslint --project .", | ||
"lint-fix": "eslint -c ../eslintrc.yml --ext=.js,.ts --fix .", | ||
"test": "jest --coverage", | ||
"clean": "tsc -b --clean", | ||
"watch": "tsc -b -w", | ||
"integ": "cdk-integ", | ||
"integ-assert": "cdk-integ-assert", | ||
"integ-no-clean": "cdk-integ --no-clean", | ||
"jsii": "jsii", | ||
"jsii-pacmak": "jsii-pacmak", | ||
"build+lint+test": "npm run jsii && npm run lint && npm test && npm run integ-assert", | ||
"snapshot-update": "npm run jsii && npm test -- -u && npm run integ-assert" | ||
}, | ||
"jsii": { | ||
"outdir": "dist", | ||
"targets": { | ||
"java": { | ||
"package": "software.amazon.awsconstructs.services.eventsrulekinesisfirehoses3", | ||
"maven": { | ||
"groupId": "software.amazon.awsconstructs", | ||
"artifactId": "eventsrulekinesisfirehoses3" | ||
} | ||
}, | ||
"dotnet": { | ||
"namespace": "Amazon.Constructs.AWS.EventsRuleKinesisFirehoseS3", | ||
"packageId": "Amazon.Constructs.AWS.EventsRuleKinesisFirehoseS3", | ||
"signAssembly": true, | ||
"iconUrl": "https://raw.githubusercontent.com/aws/aws-cdk/master/logo/default-256-dark.png" | ||
}, | ||
"python": { | ||
"distName": "aws-solutions-constructs.aws-events-rule-kinesis-firehose-s3", | ||
"module": "aws_solutions_constructs.aws_events_rule_kinesis_firehose_s3" | ||
} | ||
} | ||
}, | ||
"dependencies": { | ||
"@aws-cdk/aws-iam": "~1.67.0", | ||
"@aws-cdk/aws-kinesisfirehose": "~1.67.0", | ||
"@aws-cdk/aws-events": "~1.67.0", | ||
"@aws-cdk/core": "~1.67.0", | ||
"@aws-cdk/aws-s3": "~1.67.0", | ||
"@aws-cdk/aws-logs": "~1.67.0", | ||
"@aws-solutions-constructs/core": "~1.67.0", | ||
"@aws-solutions-constructs/aws-kinesisfirehose-s3": "~1.67.0", | ||
"constructs": "^3.0.4" | ||
}, | ||
"devDependencies": { | ||
"@aws-cdk/assert": "~1.67.0", | ||
"@types/jest": "^24.0.23", | ||
"@types/node": "^10.3.0" | ||
}, | ||
"jest": { | ||
"moduleFileExtensions": [ | ||
"js" | ||
] | ||
}, | ||
"peerDependencies": { | ||
"@aws-cdk/aws-iam": "~1.67.0", | ||
"@aws-cdk/aws-kinesisfirehose": "~1.67.0", | ||
"@aws-cdk/aws-events": "~1.67.0", | ||
"@aws-cdk/aws-s3": "~1.67.0", | ||
"@aws-cdk/core": "~1.67.0", | ||
"@aws-solutions-constructs/core": "~1.67.0", | ||
"@aws-solutions-constructs/aws-kinesisfirehose-s3": "~1.67.0", | ||
"constructs": "^3.0.4", | ||
"@aws-cdk/aws-logs": "~1.67.0" | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing import statement for
cdk.Duration
, please addimport * as cdk from '@aws-cdk/core';
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done