Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added two new construct patterns: aws-events-rule-kinesisstream and aws-events-rule-kinesisfirehose-s3 #92

Merged
merged 12 commits into from
Oct 26, 2020
Merged
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
lib/*.js
test/*.js
*.d.ts
coverage
test/lambda/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
lib/*.js
test/*.js
!test/lambda/*
*.js.map
*.d.ts
node_modules
*.generated.ts
dist
.jsii

.LAST_BUILD
.nyc_output
coverage
.nycrc
.LAST_PACKAGE
*.snk
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Exclude typescript source and config
*.ts
tsconfig.json
coverage
.nyc_output
*.tgz
*.snk
*.tsbuildinfo

# Include javascript files and typescript declarations
!*.js
!*.d.ts

# Exclude jsii outdir
dist

# Include .jsii
!.jsii

# Include .jsii
!.jsii
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# aws-events-rule-kinesisfirehose-s3 module
<!--BEGIN STABILITY BANNER-->

---

![Stability: Experimental](https://img.shields.io/badge/stability-Experimental-important.svg?style=for-the-badge)

> All classes are under active development and subject to non-backward compatible changes or removal in any
> future version. These are not subject to the [Semantic Versioning](https://semver.org/) model.
> This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.

---
<!--END STABILITY BANNER-->

| **Reference Documentation**:| <span style="font-weight: normal">https://docs.aws.amazon.com/solutions/latest/constructs/</span>|
|:-------------|:-------------|
<div style="height:8px"></div>

| **Language** | **Package** |
|:-------------|-----------------|
|![Python Logo](https://docs.aws.amazon.com/cdk/api/latest/img/python32.png) Python|`aws_solutions_constructs.aws_events_rule_kinesisfirehose_s3`|
|![Typescript Logo](https://docs.aws.amazon.com/cdk/api/latest/img/typescript32.png) Typescript|`@aws-solutions-constructs/aws-events-rule-kinesisfirehose-s3`|
|![Java Logo](https://docs.aws.amazon.com/cdk/api/latest/img/java32.png) Java|`software.amazon.awsconstructs.services.eventsrulekinesisfirehoses3`|

This AWS Solutions Construct implements an Amazon CloudWatch Events rule to send data to an Amazon Kinesis Data Firehose delivery stream connected to an Amazon S3 bucket.

Here is a minimal deployable pattern definition in Typescript:

``` javascript
import { EventsRuleToKinesisFirehoseToS3, EventsRuleToKinesisFirehoseToS3Props } from '@aws-solutions-constructs/aws-events-rule-kinesisfirehose-s3';

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing import statement for cdk.Duration, please add import * as cdk from '@aws-cdk/core';

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

const props: EventsRuleToKinesisFirehoseToS3Props = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When this code is copied into the cdk init app the props variable collides with the existing one, please change the props variable name to eventsRuleToKinesisFirehoseToS3Props here and on line 37

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

eventRuleProps: {
schedule: events.Schedule.rate(cdk.Duration.minutes(5))
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing closing bracket }

    const props: EventsRuleToKinesisFirehoseToS3Props = {
      eventRuleProps: {
      schedule: events.Schedule.rate(cdk.Duration.minutes(5))
      }
    };

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


new EventsRuleToKinesisFirehoseToS3(this, 'test-events-rule-firehose-s3', props);

```

## Initializer

``` text
new EventsRuleToKinesisFirehoseToS3(scope: Construct, id: string, props: EventsRuleToKinesisFirehoseToS3Props);
```

_Parameters_

* scope [`Construct`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_core.Construct.html)
* id `string`
* props [`EventsRuleToKinesisFirehoseToS3Props`](#pattern-construct-props)

## Pattern Construct Props

| **Name** | **Type** | **Description** |
|:-------------|:----------------|-----------------|
|eventRuleProps|[`events.RuleProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-events.RuleProps.html)|User provided eventRuleProps to override the defaults.|
|kinesisFirehoseProps?|[`kinesisfirehose.CfnDeliveryStreamProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesisfirehose.CfnDeliveryStreamProps.html)|Optional user provided props to override the default props for Kinesis Firehose Delivery Stream|
|existingBucketObj?|[`s3.IBucket`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.IBucket.html)|Existing instance of S3 Bucket object, if this is set then the bucketProps is ignored.|
|bucketProps?|[`s3.BucketProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.BucketProps.html)|User provided props to override the default props for the S3 Bucket.|

## Pattern Properties

| **Name** | **Type** | **Description** |
|:-------------|:----------------|-----------------|
|eventsRule|[`events.Rule`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-events.Rule.html)|Returns an instance of events.Rule created by the construct.|
|kinesisFirehose|[`kinesisfirehose.CfnDeliveryStream`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesisfirehose.CfnDeliveryStream.html)|Returns an instance of kinesisfirehose.CfnDeliveryStream created by the construct|
|s3Bucket?|[`s3.Bucket`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.Bucket.html)|Returns an instance of s3.Bucket created by the construct|
|s3LoggingBucket?|[`s3.Bucket`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.Bucket.html)|Returns an instance of s3.Bucket created by the construct as the logging bucket for the primary bucket.|
|eventsRole|[`iam.Role`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-iam.Role.html)|Returns an instance of the iam.Role created by the construct for Events Rule|
|kinesisFirehoseRole|[`iam.Role`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-iam.Role.html)|Returns an instance of the iam.Role created by the construct for Kinesis Data Firehose delivery stream|
|kinesisFirehoseLogGroup|[`logs.LogGroup`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-logs.LogGroup.html)|Returns an instance of the LogGroup created by the construct for Kinesis Data Firehose delivery stream|

## Default settings

Out of the box implementation of the Construct without any override will set the following defaults:

### Amazon CloudWatch Events Rule
* Configure least privilege access IAM role for Events Rule to publish to the Kinesis Firehose Delivery Stream.

### Amazon Kinesis Firehose
* Enable CloudWatch logging for Kinesis Firehose
* Configure least privilege access IAM role for Amazon Kinesis Firehose

### Amazon S3 Bucket
* Configure Access logging for S3 Bucket
* Enable server-side encryption for S3 Bucket using AWS managed KMS Key
* Turn on the versioning for S3 Bucket
* Don't allow public access for S3 Bucket
* Retain the S3 Bucket when deleting the CloudFormation stack
* Applies Lifecycle rule to move noncurrent object versions to Glacier storage after 90 days

## Architecture
![Architecture Diagram](architecture.png)

***
&copy; Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
/**
* Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
* with the License. A copy of the License is located at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES
* OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions
* and limitations under the License.
*/

import * as events from '@aws-cdk/aws-events';
import * as kinesisfirehose from '@aws-cdk/aws-kinesisfirehose';
import * as defaults from '@aws-solutions-constructs/core';
import * as iam from '@aws-cdk/aws-iam';
import * as s3 from '@aws-cdk/aws-s3';
import * as logs from '@aws-cdk/aws-logs';
import { Construct } from '@aws-cdk/core';
import { overrideProps } from '@aws-solutions-constructs/core';
import { KinesisFirehoseToS3 } from '@aws-solutions-constructs/aws-kinesisfirehose-s3';

/**
* @summary The properties for the EventsRuleToKinesisFirehoseToS3 Construct
*/
export interface EventsRuleToKinesisFirehoseToS3Props {
/**
* User provided eventRuleProps to override the defaults
*
* @default - None
*/
readonly eventRuleProps: events.RuleProps
/**
* User provided props to override the default props for the Kinesis Firehose.
*
* @default - Default props are used
*/
readonly kinesisFirehoseProps?: kinesisfirehose.CfnDeliveryStreamProps | any
/**
* Existing instance of S3 Bucket object, if this is set then the bucketProps is ignored.
*
* @default - None
*/
readonly existingBucketObj?: s3.IBucket,
/**
* User provided props to override the default props for the S3 Bucket.
*
* @default - Default props are used
*/
readonly bucketProps?: s3.BucketProps
}

export class EventsRuleToKinesisFirehoseToS3 extends Construct {
public readonly eventsRule: events.Rule;
public readonly eventsRole: iam.Role;
public readonly kinesisFirehose: kinesisfirehose.CfnDeliveryStream;
public readonly kinesisFirehoseLogGroup: logs.LogGroup;
public readonly kinesisFirehoseRole: iam.Role;
public readonly s3Bucket?: s3.Bucket;
public readonly s3LoggingBucket?: s3.Bucket;

/**
* @summary Constructs a new instance of the EventsRuleToKinesisFirehoseToS3 class.
* @param {cdk.App} scope - represents the scope for all the resources.
* @param {string} id - this is a a scope-unique id.
* @param {EventsRuleToKinesisFirehoseToS3Props} props - user provided props for the construct
* @since 0.8.0
* @access public
*/
constructor(scope: Construct, id: string, props: EventsRuleToKinesisFirehoseToS3Props) {
super(scope, id);

// Set up the Kinesis Firehose using KinesisFirehoseToS3 construct
const firehoseToS3 = new KinesisFirehoseToS3(this, 'KinesisFirehoseToS3', {
kinesisFirehoseProps: props.kinesisFirehoseProps,
existingBucketObj: props.existingBucketObj,
bucketProps: props.bucketProps
});
this.kinesisFirehose = firehoseToS3.kinesisFirehose;
this.s3Bucket = firehoseToS3.s3Bucket;
this.kinesisFirehoseRole = firehoseToS3.kinesisFirehoseRole;
this.s3LoggingBucket = firehoseToS3.s3LoggingBucket;
this.kinesisFirehoseLogGroup = firehoseToS3.kinesisFirehoseLogGroup;

// Create an events service role
this.eventsRole = new iam.Role(this, 'EventsRuleInvokeKinesisFirehoseRole', {
assumedBy: new iam.ServicePrincipal('events.amazonaws.com'),
description: 'Events Rule To Kinesis Firehose Role',
});

// Setup the IAM policy that grants events rule the permission to send cw events data to kinesis firehose
const eventsPolicy = new iam.Policy(this, 'EventsRuleInvokeKinesisFirehosePolicy', {
statements: [new iam.PolicyStatement({
actions: [
'firehose:PutRecord',
'firehose:PutRecordBatch'
],
resources: [this.kinesisFirehose.attrArn]
})
]});

// Attach policy to role
eventsPolicy.attachToRole(this.eventsRole);

// Set up the Kinesis Firehose as the target for event rule
const KinesisFirehoseEventTarget: events.IRuleTarget = {
bind: () => ({
id: '',
arn: this.kinesisFirehose.attrArn,
role: this.eventsRole
})
};

// Set up the events rule props
const defaultEventsRuleProps = defaults.DefaultEventsRuleProps([KinesisFirehoseEventTarget]);
const eventsRuleProps = overrideProps(defaultEventsRuleProps, props.eventRuleProps, true);

this.eventsRule = new events.Rule(this, 'EventsRule', eventsRuleProps);

}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
{
"name": "@aws-solutions-constructs/aws-events-rule-kinesisfirehose-s3",
"version": "1.67.0",
"description": "CDK Constructs for Amazon CloudWatch Events Rule to Amazon Kinesis Firehose to Amazon S3 integration.",
"main": "lib/index.js",
"types": "lib/index.d.ts",
"repository": {
"type": "git",
"url": "https://github.com/awslabs/aws-solutions-constructs.git",
"directory": "source/patterns/@aws-solutions-constructs/aws-events-rule-kinesisfirehose-s3"
},
"author": {
"name": "Amazon Web Services",
"url": "https://aws.amazon.com",
"organization": true
},
"license": "Apache-2.0",
"scripts": {
"build": "tsc -b .",
"lint": "eslint -c ../eslintrc.yml --ext=.js,.ts . && tslint --project .",
"lint-fix": "eslint -c ../eslintrc.yml --ext=.js,.ts --fix .",
"test": "jest --coverage",
"clean": "tsc -b --clean",
"watch": "tsc -b -w",
"integ": "cdk-integ",
"integ-assert": "cdk-integ-assert",
"integ-no-clean": "cdk-integ --no-clean",
"jsii": "jsii",
"jsii-pacmak": "jsii-pacmak",
"build+lint+test": "npm run jsii && npm run lint && npm test && npm run integ-assert",
"snapshot-update": "npm run jsii && npm test -- -u && npm run integ-assert"
},
"jsii": {
"outdir": "dist",
"targets": {
"java": {
"package": "software.amazon.awsconstructs.services.eventsrulekinesisfirehoses3",
"maven": {
"groupId": "software.amazon.awsconstructs",
"artifactId": "eventsrulekinesisfirehoses3"
}
},
"dotnet": {
"namespace": "Amazon.Constructs.AWS.EventsRuleKinesisFirehoseS3",
"packageId": "Amazon.Constructs.AWS.EventsRuleKinesisFirehoseS3",
"signAssembly": true,
"iconUrl": "https://raw.githubusercontent.com/aws/aws-cdk/master/logo/default-256-dark.png"
},
"python": {
"distName": "aws-solutions-constructs.aws-events-rule-kinesis-firehose-s3",
"module": "aws_solutions_constructs.aws_events_rule_kinesis_firehose_s3"
}
}
},
"dependencies": {
"@aws-cdk/aws-iam": "~1.67.0",
"@aws-cdk/aws-kinesisfirehose": "~1.67.0",
"@aws-cdk/aws-events": "~1.67.0",
"@aws-cdk/core": "~1.67.0",
"@aws-cdk/aws-s3": "~1.67.0",
"@aws-cdk/aws-logs": "~1.67.0",
"@aws-solutions-constructs/core": "~1.67.0",
"@aws-solutions-constructs/aws-kinesisfirehose-s3": "~1.67.0",
"constructs": "^3.0.4"
},
"devDependencies": {
"@aws-cdk/assert": "~1.67.0",
"@types/jest": "^24.0.23",
"@types/node": "^10.3.0"
},
"jest": {
"moduleFileExtensions": [
"js"
]
},
"peerDependencies": {
"@aws-cdk/aws-iam": "~1.67.0",
"@aws-cdk/aws-kinesisfirehose": "~1.67.0",
"@aws-cdk/aws-events": "~1.67.0",
"@aws-cdk/aws-s3": "~1.67.0",
"@aws-cdk/core": "~1.67.0",
"@aws-solutions-constructs/core": "~1.67.0",
"@aws-solutions-constructs/aws-kinesisfirehose-s3": "~1.67.0",
"constructs": "^3.0.4",
"@aws-cdk/aws-logs": "~1.67.0"
}
}
Loading