Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

internal/ini: Failing test cases for #2800 #2801

Closed
wants to merge 1 commit into from

Conversation

bflad
Copy link
Contributor

@bflad bflad commented Aug 29, 2019

Reference: #2800

My apologies I won't have time to dive into the INI parsing code for awhile to fully resolve this. It appears anything between the first new line token and next new line token when in the skip state is completely skipped, even if it contains a separator.

--- FAIL: TestParser (0.00s)
    --- FAIL: TestParser/complex_section_statement#02 (0.00s)
        ini_parser_test.go:302: expected same length 4, but received 3
        ini_parser_test.go:317: expected:
            	0: {completed_stmt {0 NONE 0 []} false [{section_stmt {1 STRING 0 [100 101 102 97 117 108 116]} true []}]}
            	1: {skip {0 NONE 0 []} false [{ {4 NONE 0 [61]} true [{expr {1 STRING 0 [115 51]} true []}]}]}
            	2: {completed_stmt {0 NONE 0 []} false [{section_stmt {1 STRING 0 [97 115 115 117 109 101 114 111 108 101]} true []}]}
            	3: {expr_stmt {0 NONE 0 []} false [{ {4 NONE 0 [61]} true [{expr {1 STRING 0 [111 117 116 112 117 116]} true []} {expr {1 STRING 0 [106 115 111 110]} true []}]}]}

            received:
            	0: {completed_stmt {0 NONE 0 []} false [{section_stmt {1 STRING 0 [100 101 102 97 117 108 116]} true []}]}
            	1: {skip {0 NONE 0 []} false [{ {4 NONE 0 [61]} true [{expr {1 STRING 0 [115 51]} true []}]}]}
            	2: {expr_stmt {0 NONE 0 []} false [{ {4 NONE 0 [61]} true [{expr {1 STRING 0 [111 117 116 112 117 116]} true []} {expr {1 STRING 0 [106 115 111 110]} true []}]}]}
--- FAIL: TestValidDataFiles (0.01s)
    walker_test.go:59: could not find profile bar

Reference: aws#2800

```
--- FAIL: TestParser (0.00s)
    --- FAIL: TestParser/complex_section_statement#02 (0.00s)
        ini_parser_test.go:302: expected same length 4, but received 3
        ini_parser_test.go:317: expected:
            	0: {completed_stmt {0 NONE 0 []} false [{section_stmt {1 STRING 0 [100 101 102 97 117 108 116]} true []}]}
            	1: {skip {0 NONE 0 []} false [{ {4 NONE 0 [61]} true [{expr {1 STRING 0 [115 51]} true []}]}]}
            	2: {completed_stmt {0 NONE 0 []} false [{section_stmt {1 STRING 0 [97 115 115 117 109 101 114 111 108 101]} true []}]}
            	3: {expr_stmt {0 NONE 0 []} false [{ {4 NONE 0 [61]} true [{expr {1 STRING 0 [111 117 116 112 117 116]} true []} {expr {1 STRING 0 [106 115 111 110]} true []}]}]}

            received:
            	0: {completed_stmt {0 NONE 0 []} false [{section_stmt {1 STRING 0 [100 101 102 97 117 108 116]} true []}]}
            	1: {skip {0 NONE 0 []} false [{ {4 NONE 0 [61]} true [{expr {1 STRING 0 [115 51]} true []}]}]}
            	2: {expr_stmt {0 NONE 0 []} false [{ {4 NONE 0 [61]} true [{expr {1 STRING 0 [111 117 116 112 117 116]} true []} {expr {1 STRING 0 [106 115 111 110]} true []}]}]}
--- FAIL: TestValidDataFiles (0.01s)
    walker_test.go:59: could not find profile bar
```
@bflad bflad force-pushed the internal-ini-failing-2800 branch from fc6b90b to 77a60a5 Compare August 29, 2019 11:53
@jasdel
Copy link
Contributor

jasdel commented Aug 29, 2019

Thanks for reporting this issue, and providing the testcase @bflad. I think this bug is an artifact for the SDK's built in parser handling nested key/value pairs. Though that behavior should handle the case where a field is started but no nested key/value pairs are provided.

I think the SDK's INI parser should terminate its parsing of nested block when the next line starts with no whitespace, or is a new section.

Exampled of nested key/value pairs the parser should recognize.

[section]
parentKey = 
    nestedKey = abc

Example of an INI file the parser should also correctly parse where someKey in both first and othersections have empty value, andotherKeyinother` section is also empty.

[first]
someKey =
[second]

[other]
someKey = 
otherKey =

skotambkar added a commit that referenced this pull request Oct 1, 2019
The ini parser incorrectly decided whether a statement should be skipped. As a result, valid statements in the ini files were being squashed. The PR fixes incorrect modifications to the previous token value of the skipper. We also add checks for cases where a skipped statement should be marked as complete and not be ignored.

Adds test cases for cases for statements that need to be skipped. Also adds suggested tests from #2801 . Fixes #2800
skotambkar added a commit to aws/aws-sdk-go-v2 that referenced this pull request Oct 1, 2019
The ini parser incorrectly decided whether a statement should be skipped. As a result, valid statements in the ini files were being squashed. The PR fixes incorrect modifications to the previous token value of the skipper. We also add checks for cases where a skipped statement should be marked as complete and not be ignored.

Adds test cases for cases for statements that need to be skipped. Also adds suggested tests from aws/aws-sdk-go#2801 .
skmcgrail added a commit to skmcgrail/aws-sdk-go-v2 that referenced this pull request Oct 1, 2019
* Synced the V2 SDK with latest AWS service API definitions.

* This update includes breaking changes to how the DynamoDB AttributeValue (un)marshier handles empty collections.

* `service/s3/s3crypto`: Deprecates the crypto client from the SDK ([aws#394](aws#394))
  * s3crypto client is now deprecated and may be removed from the future versions of the SDK.
* `aws`: Removes plugin credential provider ([aws#391](aws#391))
  * Removing plugin credential provider from the v2 SDK developer preview. This feature may be made available as a separate module.
* Removes support for deprecated Go versions ([aws#393](aws#393))
  * Removes support for Go version specific files from the SDK. Also removes irrelevant build tags, and updates the README.md file.
  * Raises the minimum supported version to Go 1.11 for the SDK. Older versions may work, but are not actively supported

* `service/s3/s3manager`: Add Upload Buffer Provider ([aws#404](aws#404))
  * Adds a new `BufferProvider` member for specifying how part data can be buffered in memory.
  * Windows platforms will now default to buffering 1MB per part to reduce contention when uploading files.
  * Non-Windows platforms will continue to employ a non-buffering behavior.
* `service/s3/s3manager`: Add Download Buffer Provider ([aws#404](aws#404))
  * Adds a new `BufferProvider` member for specifying how part data can be buffered in memory when copying from the http response body.
  * Windows platforms will now default to buffering 1MB per part to reduce contention when downloading files.
  * Non-Windows platforms will continue to employ a non-buffering behavior.
* `service/dynamodb/dynamodbattribute`: New Encoder and Decoder Behavior for Empty Collections ([aws#401](aws#401))
  * The `Encoder` and `Decoder` types have been enhanced to support the marshaling of empty structures, maps, and slices to and from their respective DynamoDB AttributeValues.
  * This change incorporates the behavior changes introduced via a marshal option in V1 ([aws#2834](aws/aws-sdk-go#2834))

* `internal/awsutil`: Add suppressing logging sensitive API parameters ([aws#398](aws#398))
  * Adds suppressing logging sensitive API parameters marked with the `sensitive` trait. This prevents the API type's `String` method returning a string representation of the API type with sensitive fields printed such as keys and passwords.
  * Related to [aws/aws-sdk-go#2310](aws/aws-sdk-go#2310)
  * Fixes [aws#251](aws#251)
* `aws/request` : Retryer is now a named field on Request. ([aws#393](aws#393))
* `service/s3/s3manager`: Adds `sync.Pool` to allow reuse of part buffers for streaming payloads ([aws#404](aws#404))
  * Fixes [aws#402](aws#402)
  * Uses the new behavior introduced in V1 [aws#2863](aws/aws-sdk-go#2863) which allows the reuse of the sync.Pool across multiple Upload request that match part sizes.

* `service/s3/s3manager`: Fix index out of range when a streaming reader returns -1 ([aws#378](aws#378))
  * Fixes the S3 Upload Manager's handling of an unbounded streaming reader that returns negative bytes read.
* `internal/ini`: Fix ini parser to handle empty values [aws#406](aws#406)
  * Fixes incorrect modifications to the previous token value of the skipper. Adds checks for cases where a skipped statement should be marked as complete and not be ignored.
  * Adds tests for nested and empty field value parsing, along with tests suggested in [aws/aws-sdk-go#2801](aws/aws-sdk-go#2801)
skmcgrail added a commit to skmcgrail/aws-sdk-go-v2 that referenced this pull request Oct 1, 2019
### Services
* Synced the V2 SDK with latest AWS service API definitions.

### SDK Breaking changes
* This update includes breaking changes to how the DynamoDB AttributeValue (un)marshier handles empty collections.

### Deprecations
* `service/s3/s3crypto`: Deprecates the crypto client from the SDK ([aws#394](aws#394))
  * s3crypto client is now deprecated and may be removed from the future versions of the SDK.
* `aws`: Removes plugin credential provider ([aws#391](aws#391))
  * Removing plugin credential provider from the v2 SDK developer preview. This feature may be made available as a separate module.
* Removes support for deprecated Go versions ([aws#393](aws#393))
  * Removes support for Go version specific files from the SDK. Also removes irrelevant build tags, and updates the README.md file.
  * Raises the minimum supported version to Go 1.11 for the SDK. Older versions may work, but are not actively supported

### SDK Features
* `service/s3/s3manager`: Add Upload Buffer Provider ([aws#404](aws#404))
  * Adds a new `BufferProvider` member for specifying how part data can be buffered in memory.
  * Windows platforms will now default to buffering 1MB per part to reduce contention when uploading files.
  * Non-Windows platforms will continue to employ a non-buffering behavior.
* `service/s3/s3manager`: Add Download Buffer Provider ([aws#404](aws#404))
  * Adds a new `BufferProvider` member for specifying how part data can be buffered in memory when copying from the http response body.
  * Windows platforms will now default to buffering 1MB per part to reduce contention when downloading files.
  * Non-Windows platforms will continue to employ a non-buffering behavior.
* `service/dynamodb/dynamodbattribute`: New Encoder and Decoder Behavior for Empty Collections ([aws#401](aws#401))
  * The `Encoder` and `Decoder` types have been enhanced to support the marshaling of empty structures, maps, and slices to and from their respective DynamoDB AttributeValues.
  * This change incorporates the behavior changes introduced via a marshal option in V1 ([aws#2834](aws/aws-sdk-go#2834))

### SDK Enhancements
* `internal/awsutil`: Add suppressing logging sensitive API parameters ([aws#398](aws#398))
  * Adds suppressing logging sensitive API parameters marked with the `sensitive` trait. This prevents the API type's `String` method returning a string representation of the API type with sensitive fields printed such as keys and passwords.
  * Related to [aws/aws-sdk-go#2310](aws/aws-sdk-go#2310)
  * Fixes [aws#251](aws#251)
* `aws/request` : Retryer is now a named field on Request. ([aws#393](aws#393))
* `service/s3/s3manager`: Adds `sync.Pool` to allow reuse of part buffers for streaming payloads ([aws#404](aws#404))
  * Fixes [aws#402](aws#402)
  * Uses the new behavior introduced in V1 [aws#2863](aws/aws-sdk-go#2863) which allows the reuse of the sync.Pool across multiple Upload request that match part sizes.

### SDK Bugs
* `service/s3/s3manager`: Fix index out of range when a streaming reader returns -1 ([aws#378](aws#378))
  * Fixes the S3 Upload Manager's handling of an unbounded streaming reader that returns negative bytes read.
* `internal/ini`: Fix ini parser to handle empty values [aws#406](aws#406)
  * Fixes incorrect modifications to the previous token value of the skipper. Adds checks for cases where a skipped statement should be marked as complete and not be ignored.
  * Adds tests for nested and empty field value parsing, along with tests suggested in [aws/aws-sdk-go#2801](aws/aws-sdk-go#2801)
skmcgrail added a commit to aws/aws-sdk-go-v2 that referenced this pull request Oct 2, 2019
### Services
* Synced the V2 SDK with latest AWS service API definitions.

### SDK Breaking changes
* This update includes breaking changes to how the DynamoDB AttributeValue (un)marshier handles empty collections.

### Deprecations
* `service/s3/s3crypto`: Deprecates the crypto client from the SDK ([#394](#394))
  * s3crypto client is now deprecated and may be removed from the future versions of the SDK.
* `aws`: Removes plugin credential provider ([#391](#391))
  * Removing plugin credential provider from the v2 SDK developer preview. This feature may be made available as a separate module.
* Removes support for deprecated Go versions ([#393](#393))
  * Removes support for Go version specific files from the SDK. Also removes irrelevant build tags, and updates the README.md file.
  * Raises the minimum supported version to Go 1.11 for the SDK. Older versions may work, but are not actively supported

### SDK Features
* `service/s3/s3manager`: Add Upload Buffer Provider ([#404](#404))
  * Adds a new `BufferProvider` member for specifying how part data can be buffered in memory.
  * Windows platforms will now default to buffering 1MB per part to reduce contention when uploading files.
  * Non-Windows platforms will continue to employ a non-buffering behavior.
* `service/s3/s3manager`: Add Download Buffer Provider ([#404](#404))
  * Adds a new `BufferProvider` member for specifying how part data can be buffered in memory when copying from the http response body.
  * Windows platforms will now default to buffering 1MB per part to reduce contention when downloading files.
  * Non-Windows platforms will continue to employ a non-buffering behavior.
* `service/dynamodb/dynamodbattribute`: New Encoder and Decoder Behavior for Empty Collections ([#401](#401))
  * The `Encoder` and `Decoder` types have been enhanced to support the marshaling of empty structures, maps, and slices to and from their respective DynamoDB AttributeValues.
  * This change incorporates the behavior changes introduced via a marshal option in V1 ([#2834](aws/aws-sdk-go#2834))

### SDK Enhancements
* `internal/awsutil`: Add suppressing logging sensitive API parameters ([#398](#398))
  * Adds suppressing logging sensitive API parameters marked with the `sensitive` trait. This prevents the API type's `String` method returning a string representation of the API type with sensitive fields printed such as keys and passwords.
  * Related to [aws/aws-sdk-go#2310](aws/aws-sdk-go#2310)
  * Fixes [#251](#251)
* `aws/request` : Retryer is now a named field on Request. ([#393](#393))
* `service/s3/s3manager`: Adds `sync.Pool` to allow reuse of part buffers for streaming payloads ([#404](#404))
  * Fixes [#402](#402)
  * Uses the new behavior introduced in V1 [#2863](aws/aws-sdk-go#2863) which allows the reuse of the sync.Pool across multiple Upload request that match part sizes.

### SDK Bugs
* `service/s3/s3manager`: Fix index out of range when a streaming reader returns -1 ([#378](#378))
  * Fixes the S3 Upload Manager's handling of an unbounded streaming reader that returns negative bytes read.
* `internal/ini`: Fix ini parser to handle empty values [#406](#406)
  * Fixes incorrect modifications to the previous token value of the skipper. Adds checks for cases where a skipped statement should be marked as complete and not be ignored.
  * Adds tests for nested and empty field value parsing, along with tests suggested in [aws/aws-sdk-go#2801](aws/aws-sdk-go#2801)
@aws-sdk-go-automation aws-sdk-go-automation mentioned this pull request Oct 2, 2019
@jasdel
Copy link
Contributor

jasdel commented Oct 10, 2019

Closing this as the fixed was released in v1.25.4 PR #2860

@jasdel jasdel closed this Oct 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants