support for DynamicPartitioningConfiguration in firehose resource

[bozerkins]

Closes #20763

Added support for dynamic partitioning configuration. The syntax is very straight forward

dynamic_partitioning_configuration {
  enabled = true
  retry_options {
    duration_in_seconds = 300
  }
}

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Relates OR Closes #0000

[github-actions]

Welcome @bozerkins 👋

It looks like this is your first Pull Request submission to the Terraform AWS Provider! If you haven’t already done so please make sure you have checked out our CONTRIBUTING guide and FAQ to make sure your contribution is adhering to best practice and has all the necessary elements in place for a successful approval.

Also take a look at our FAQ which details how we prioritize Pull Requests for inclusion.

Thanks again, and welcome to the community! 😃

[bozerkins]

My first PR here. Any feedback would be appreciated. I'm interested into seeing this functionality in a release as soon as possible.

[esvm]

I'm interested in this feature. How could we define if it's inline parsing or from aws lambda and the options to define the query or the lambda arn?

[bozerkins]

How dynamic partitioning work is defined by 1) processors that you specify in the configuration 2) prefix key

For example dynamic partitioning by using jq query language would like this

dynamic_partitioning_configuration {
    enabled = true
    retry_options {
        duration_in_seconds = 300
    }
}

prefix  = "custom-prefix/customerId=!{partitionKeyFromQuery:customerId}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/"

processing_configuration {
    enabled = true
    # processor for partitioning
    processors {
        type = "MetadataExtraction"

        parameters {
            parameter_name  = "MetadataExtractionQuery"
            parameter_value = "{customerId:.customer_id}"
        }
        parameters {
            parameter_name  = "JsonParsingEngine"
            parameter_value = "JQ-1.6"
        }
    }
    # record file unification
    processors {
        type = "RecordDeAggregation"

        parameters {
            parameter_name  = "SubRecordType"
            parameter_value = "JSON"
        }
    }
    # record delimited unification
    processors {
        type = "AppendDelimiterToRecord"
    }
}

When you want to enable processing from lambda, the configuration would be similar to this

dynamic_partitioning_configuration {
    enabled = true
    retry_options {
        duration_in_seconds = 300
    }
}
prefix  = "custom-prefix/customerId=!{partitionKeyFromLambda:customerId}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/"

processing_configuration {
    enabled = true
    # lambda function for processing and partitioning
    processors {
        type = "Lambda"

        parameters {
            parameter_name  = "LambdaArn"
            parameter_value = "${aws_lambda_function.processor.arn}:$LATEST"
        }
    }

    # record file unification
    processors {
        type = "RecordDeAggregation"

        parameters {
            parameter_name  = "SubRecordType"
            parameter_value = "JSON"
        }
    }
}

[esvm]

Hey @bozerkins thanks for the response, I appreciate it! Do those processors exist today? I looked into the docs and the unique processor existent is the Lambda processor.

[bozerkins]

Luckily, processors in provider are generated automatically from the aws-go client, and are available since this commit
a321b3b

Here are the release notes
https://github.com/aws/aws-sdk-go/releases/tag/v1.40.33

[avegao]

Any news about this?

[bozerkins]

Curious myself. Would really like to get feedback on this.

On the bright side - I've deployed this functionality to production and it's working brilliantly. With dynamic partitioning I managed to remove a decent chunk of the data partitioning logic (previously made with lambdas and S3 event notifications) and place it into firehouse.

[RonPenton]

Any updates on this? Would be great to have!

[tonyf]

Any updates here?

[zhelding]

Pull request #21306 has significantly refactored the AWS Provider codebase. As a result, most PRs opened prior to the refactor now have merge conflicts that must be resolved before proceeding.

Specifically, PR #21306 relocated the code for all AWS resources and data sources from a single aws directory to a large number of separate directories in internal/service, each corresponding to a particular AWS service. This separation of code has also allowed for us to simplify the names of underlying functions -- while still avoiding namespace collisions.

We recognize that many pull requests have been open for some time without yet being addressed by our maintainers. Therefore, we want to make it clear that resolving these conflicts in no way affects the prioritization of a particular pull request. Once a pull request has been prioritized for review, the necessary changes will be made by a maintainer -- either directly or in collaboration with the pull request author.

For a more complete description of this refactor, including examples of how old filepaths and function names correspond to their new counterparts: please refer to issue #20000.

For a quick guide on how to amend your pull request to resolve the merge conflicts resulting from this refactor and bring it in line with our new code patterns: please refer to our Service Package Refactor Pull Request Guide.

[PG-Daniel-Andrews]

Any updates on this? It's great functionality but can't deploy new pipelines until this is available in Terraform.

[alexnovak]

I've slightly modified the code in this diff on a personal branch to get things working here: https://github.com/alexnovak/terraform-provider-aws/tree/adds-dynamic-configuration
This is compliant with the refactor mentioned in the comment above, and I'm currently using it in my local builds to interact with firehose.
I don't want to step on the toes of the original author, but given that this has been hanging for a little while, I'd love to help in any way I can to get some version of this pushed through. Do you happen to still know if you're able to get around to this @bozerkins ? If so, is there a way we can prioritize giving this diff some feedback @zhelding ? (Apologies to you both for the @'s)

[bozerkins]

Thanks for the input! I'll try to get around to this in recent days :) Would appreciate some help with additional testing after the changes.

[YakDriver]

@bozerkins Thank you for your contribution!! I would like to help out with getting this up to speed post-refactor. I may force push to your branch. If so, please pull before pushing any changes. What we really need are acceptance tests.

[YakDriver]

@bozerkins I've updated your branch so it should be working. Are you willing/able to add acceptance tests?

@alexnovak I looked at your changes. I like how you simplified but I think it's best if we stick with the structure as given by AWS. If they make changes later, it's easier if we're already aligned. Also, if we're aligned, the AWS Go SDK docs are like our docs and vice versa.

[alexnovak]

Thanks for your feedback @YakDriver! I went forward with that change for two reasons:

1. Using the diff as-is, when planning / applying after the resource is created, you will incorrectly have to update if you don't specify a retry_options within your code. We cannot (to my knowledge! Let me know if I'm wrong) specify a default for a schema-type object for retry_options, meaning it either has to be marked as required to circumvent the problem, or we simply permit the "unnecessary update" pattern.
2. I noticed a number of other objects within this file required a RetryOptions block (eg redshift configuration). In all of those examples - the author chose to bring retry_duration to the parent instead of creating a separate retry_options block. It seemed more consistent with the patterns established elsewhere in the file.

I'm totally up for whatever pattern you feel is more appropriate! Just want to give some color as to why I thought those changes were necessary.

[alexnovak]

Lol sorry - I just got back from vacation but looks like @YakDriver 's got this.

[YakDriver]

@adutane PR #7026 added error_output_prefix to the extended_s3_configuration. In testing, dynamic partitioning appears to work correctly using it. s3_configuration does not have error_output_prefix but should not need it in order for dynamic partitioning to work.

[YakDriver]

Awesome job! 🎉

Output from acceptance tests (us-west-2):

% make testacc TESTS='TestAccFirehoseDeliveryStream' PKG=firehose
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/firehose/... -v -count 1 -parallel 20 -run='TestAccFirehoseDeliveryStream' -timeout 180m
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3_kinesisStreamSource (81.54s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversionOpenXJSONSerDe_empty (106.58s)
--- PASS: TestAccFirehoseDeliveryStream_missingProcessing (109.65s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3Processing_empty (110.35s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversionParquetSerDe_empty (117.16s)
--- PASS: TestAccFirehoseDeliveryStream_HTTPEndpoint_retryDuration (120.47s)
--- PASS: TestAccFirehoseDeliveryStreamDataSource_basic (138.24s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversionDeserializer_update (140.39s)
--- PASS: TestAccFirehoseDeliveryStream_extendedS3KMSKeyARN (140.95s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversionOrcSerDe_empty (141.61s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3_errorOutputPrefix (143.41s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversionSerializer_update (147.56s)
--- PASS: TestAccFirehoseDeliveryStream_splunkUpdates (149.52s)
--- PASS: TestAccFirehoseDeliveryStream_extendedS3Updates (172.49s)
--- PASS: TestAccFirehoseDeliveryStream_httpEndpoint (176.82s)
--- PASS: TestAccFirehoseDeliveryStream_extendedS3DynamicPartitioning (186.68s)
--- PASS: TestAccFirehoseDeliveryStream_s3KinesisStreamSource (72.53s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversionHiveJSONSerDe_empty (130.30s)
--- PASS: TestAccFirehoseDeliveryStream_s3WithCloudWatchLogging (97.64s)
--- PASS: TestAccFirehoseDeliveryStream_basic (90.19s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3_externalUpdate (108.36s)
--- PASS: TestAccFirehoseDeliveryStream_s3Updates (140.55s)
--- PASS: TestAccFirehoseDeliveryStream_disappears (116.98s)
--- PASS: TestAccFirehoseDeliveryStream_s3basicWithTags (152.89s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversion_enabled (150.63s)
--- PASS: TestAccFirehoseDeliveryStream_s3basic (119.51s)
--- PASS: TestAccFirehoseDeliveryStream_extendedS3basic (197.18s)
--- PASS: TestAccFirehoseDeliveryStream_redshiftUpdates (379.67s)
--- PASS: TestAccFirehoseDeliveryStream_s3basicWithSSEAndKeyType (271.87s)
--- PASS: TestAccFirehoseDeliveryStream_s3basicWithSSEAndKeyARN (272.37s)
--- PASS: TestAccFirehoseDeliveryStream_s3basicWithSSE (299.28s)
--- PASS: TestAccFirehoseDeliveryStream_elasticSearchUpdates (1099.87s)
--- PASS: TestAccFirehoseDeliveryStream_elasticSearchEndpointUpdates (1176.48s)
--- PASS: TestAccFirehoseDeliveryStream_elasticSearchWithVPCUpdates (1744.71s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/firehose	1745.955s

Output from acceptance tests (GovCloud):

% make testacc TESTS=TestAccFirehoseDeliveryStream PKG=firehose 
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/firehose/... -v -count 1 -parallel 20 -run='TestAccFirehoseDeliveryStream' -timeout 180m
--- PASS: TestAccFirehoseDeliveryStream_missingProcessing (81.86s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3Processing_empty (82.81s)
--- PASS: TestAccFirehoseDeliveryStreamDataSource_basic (84.57s)
--- PASS: TestAccFirehoseDeliveryStream_extendedS3basic (95.13s)
--- PASS: TestAccFirehoseDeliveryStream_extendedS3KMSKeyARN (99.66s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversionOpenXJSONSerDe_empty (100.03s)
--- PASS: TestAccFirehoseDeliveryStream_s3WithCloudWatchLogging (102.67s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversionParquetSerDe_empty (103.74s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversionHiveJSONSerDe_empty (104.65s)
--- PASS: TestAccFirehoseDeliveryStream_HTTPEndpoint_retryDuration (111.84s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversionOrcSerDe_empty (112.50s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3_errorOutputPrefix (120.76s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversionDeserializer_update (125.89s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3_externalUpdate (126.74s)
--- PASS: TestAccFirehoseDeliveryStream_s3Updates (128.97s)
--- PASS: TestAccFirehoseDeliveryStream_extendedS3DynamicPartitioning (135.72s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversion_enabled (138.36s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3DataFormatConversionSerializer_update (141.30s)
--- PASS: TestAccFirehoseDeliveryStream_s3KinesisStreamSource (76.67s)
--- PASS: TestAccFirehoseDeliveryStream_s3basic (64.86s)
--- PASS: TestAccFirehoseDeliveryStream_ExtendedS3_kinesisStreamSource (75.70s)
--- PASS: TestAccFirehoseDeliveryStream_s3basicWithSSEAndKeyType (197.20s)
--- PASS: TestAccFirehoseDeliveryStream_disappears (97.52s)
--- PASS: TestAccFirehoseDeliveryStream_s3basicWithTags (123.60s)
--- PASS: TestAccFirehoseDeliveryStream_splunkUpdates (123.37s)
--- PASS: TestAccFirehoseDeliveryStream_basic (104.76s)
--- PASS: TestAccFirehoseDeliveryStream_httpEndpoint (131.76s)
--- PASS: TestAccFirehoseDeliveryStream_extendedS3Updates (127.34s)
--- PASS: TestAccFirehoseDeliveryStream_s3basicWithSSEAndKeyARN (193.12s)
--- PASS: TestAccFirehoseDeliveryStream_s3basicWithSSE (225.62s)
--- PASS: TestAccFirehoseDeliveryStream_redshiftUpdates (315.93s)
--- PASS: TestAccFirehoseDeliveryStream_elasticSearchEndpointUpdates (953.68s)
--- PASS: TestAccFirehoseDeliveryStream_elasticSearchUpdates (1032.05s)
--- PASS: TestAccFirehoseDeliveryStream_elasticSearchWithVPCUpdates (3633.01s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/firehose	3634.619s

[YakDriver]

@Olaktal This should be part of v3.66.0 which should be released around Nov. 18-19.

[github-actions]

This functionality has been released in v3.66.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.