Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(aws-iot-sqs): initial implementation #267

Merged
merged 12 commits into from
Aug 12, 2021

Conversation

ericquinones
Copy link
Contributor

@ericquinones ericquinones commented Jul 16, 2021

Initial PR for new pattern: aws-iot-sqs: #266

Description of changes:

  • README and architecture diagram have been approved
  • Added implementation of new pattern: aws-iot-sqs
  • Added unit and integration tests

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@aws-solutions-constructs-team
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
  • Commit ID: 945c6dd
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-solutions-constructs-team
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
  • Commit ID: 3e3c973
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Copy link
Contributor

@biffgaut biffgaut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need more info on enableEncryptionWithCustomerManagedKey. It varies from the Design Guidelines - since consistency is very important for us changing the SQS interface here introduces backlog items to change it everywhere else.

| **Name** | **Type** | **Description** |
|:-------------|:----------------|-----------------|
|encryptionKey?|[`kms.Key`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kms.Key.html)|Returns an instance of `kms.Key` used for the SQS queue.|
|iotRole|[`iam.Role`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-iam.Role.html)|Returns an instance of `iam.Role` created by the construct, which allows IoT to publish messages to the SQS Queue|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be iotActionsRole

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. I will rename it to iotActionsRole

|deadLetterQueueProps?|[`sqs.QueueProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-sqs.QueueProps.html)|Optional user provided properties for the dead letter queue.|
|deployDeadLetterQueue?|`boolean`|Whether to deploy a secondary queue to be used as a dead letter queue. Default `true`.|
|maxReceiveCount?|`number`|The number of times a message can be unsuccessfully dequeued before being moved to the dead-letter queue. Required field if `deployDeadLetterQueue`=`true`.|
|enableEncryptionWithCustomerManagedKey?|`boolean`|Use a KMS Key, either managed by this CDK app, or imported. If importing an encryption key, it must be specified in the `encryptionKey` property for this construct.|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This varies from the design guidelines for SQS and it's not clear what it accomplishes. encryptionKey already allows a client to specify an existing key to encrypt the queue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When creating the queue, I was planning to use the buildQueue SQS helper provided by Solutions Constructs. In this snippet it seems like the logic is that if enableEncryptionWithCustomerManagedKey is false, the queue will not have encryption enabled. If true and the user provides a key in encryptionKey, that one will be used. Otherwise a new key will be built.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was also using the aws-sns-sqs pattern as a guideline, which does something similar.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upon further review, I'm inclined to agree with you. I believe it's the design guidelines that need to be updated.

|deployDeadLetterQueue?|`boolean`|Whether to deploy a secondary queue to be used as a dead letter queue. Default `true`.|
|maxReceiveCount?|`number`|The number of times a message can be unsuccessfully dequeued before being moved to the dead-letter queue. Required field if `deployDeadLetterQueue`=`true`.|
|enableEncryptionWithCustomerManagedKey?|`boolean`|Use a KMS Key, either managed by this CDK app, or imported. If importing an encryption key, it must be specified in the `encryptionKey` property for this construct.|
|encryptionKey?|[`kms.Key`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kms.Key.html)|An optional, imported encryption key to encrypt the SQS queue, and SNS Topic.|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cut paste error - SNS Topic?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that was it. I'll remove the reference to SNS. Thanks!

@aws-solutions-constructs-team
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
  • Commit ID: f11ed75
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Copy link
Contributor

@biffgaut biffgaut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can go ahead and start implementing - this looks good. Thanks!

|deadLetterQueueProps?|[`sqs.QueueProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-sqs.QueueProps.html)|Optional user provided properties for the dead letter queue.|
|deployDeadLetterQueue?|`boolean`|Whether to deploy a secondary queue to be used as a dead letter queue. Default `true`.|
|maxReceiveCount?|`number`|The number of times a message can be unsuccessfully dequeued before being moved to the dead-letter queue. Required field if `deployDeadLetterQueue`=`true`.|
|enableEncryptionWithCustomerManagedKey?|`boolean`|Use a KMS Key, either managed by this CDK app, or imported. If importing an encryption key, it must be specified in the `encryptionKey` property for this construct.|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upon further review, I'm inclined to agree with you. I believe it's the design guidelines that need to be updated.

@ericquinones ericquinones changed the title Initial design commit for new pattern: aws-iot-sqs feat(aws-iot-sqs): initial implementation Aug 3, 2021
fixes awslabs#266

Added implementation of new pattern

Added unit and integration tests
@aws-solutions-constructs-team
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
  • Commit ID: d4ce63f
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-solutions-constructs-team
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
  • Commit ID: 2bd1647
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-solutions-constructs-team
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
  • Commit ID: c48af21
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-solutions-constructs-team
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
  • Commit ID: da61bfa
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-solutions-constructs-team
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
  • Commit ID: e782f4a
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

test/*.js
*.d.ts
coverage
test/lambda/index.js
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can drop this, no lambda function in these tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed line 5

@@ -0,0 +1,16 @@
lib/*.js
test/*.js
!test/lambda/*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can drop this, no lambda function in these tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed line 3

* and limitations under the License.
*/

import { Queue, QueueProps, DeadLetterQueue } from '@aws-cdk/aws-sqs';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We haven't codified conventions, and occasionally do inexplicable things, but in general imports from cdk get a namespace applied, eg

import * as sqs from '@aws-cdk/aws-sqs'

Let's do that here.

(FYI - inexplicable is lines 20 and 21 of this - that hurts to look at)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed imports to:

import * as cdk from '@aws-cdk/core';
import * as sqs from '@aws-cdk/aws-sqs';
import * as iot from '@aws-cdk/aws-iot';
import * as kms from '@aws-cdk/aws-kms';
import * as iam from '@aws-cdk/aws-iam';
import * as defaults from '@aws-solutions-constructs/core';

Used defaults for @aws-solutions-constructs/core because I saw that used other places in the project.

@@ -0,0 +1,101 @@
# aws-iot-sqs module
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't leave a comment on the binary png file that is your architecture diagram. It doesn't match our style, we use a different icon set (perhaps an older set, but it's where we are).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the architecture diagram. Used the SQS service icon instead of the Queue icon.

* @param {cdk.App} scope - represents the scope for all the resources.
* @param {string} id - this is a a scope-unique id.
* @param {IotToSqsProps} props - user provided props for the construct
* @since 1.110.1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drop the @SInCE line

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed


// Creates a dead letter queue
expect(stack).toHaveResource("AWS::SQS::Queue", {
KmsMasterKeyId: "alias/aws/sqs"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

???
This is the check you use when enableEncryptionWithCustomerManagedKey is false. This test is supply keyProps - shouldn't you be looking for the key alias in those props? And, if so, why does this test pass?

Copy link
Contributor Author

@ericquinones ericquinones Aug 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you pass enableEncryptionWithCustomerManagedKey=false, you are indicating that you don't want to use a customer managed key but the queue will still have the default encryption: AWS managed CMK for SQS (the alias for which is alias/aws/sqs). So that's why this test passes.

The test for enableEncryptionWithCustomerManagedKey=false similarly tests that the queues have the default SQS key but also specifically checks that a Key was not created

@aws-solutions-constructs-team
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
  • Commit ID: e86c532
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

* Removed references to lambda from .gitignore and .eslintignore
* Use service namespaces instead of named imports
* Updated architecture diagram
@aws-solutions-constructs-team
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
  • Commit ID: 3d0fde1
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@ericquinones ericquinones requested a review from biffgaut August 4, 2021 12:54
@aws-solutions-constructs-team
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
  • Commit ID: 6d8b2da
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@biffgaut biffgaut merged commit 5411ab7 into awslabs:main Aug 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants