Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Solve unordered container dependency problem #2615

Merged
merged 1 commit into from
Sep 9, 2020

Conversation

ubhattacharjya
Copy link
Contributor

@ubhattacharjya ubhattacharjya commented Sep 3, 2020

Summary

This PR solves a problem of container ordering feature not working as expected when a container has multiple dependencies. This issue is mentioned in #2579 .

Implementation details

With unordered multiple dependencies - stuck as PENDING
{
    "containerDefinitions": [
        {
            "command": [
                "cat",
                "123"
            ],
            "image": "alpine",
            "startTimeout": 10,
            "name": "exit1",
            "essential": false
        },
        {
            "image": "nginx",
            "startTimeout": 60,
            "dependsOn": [
                {
                    "containerName": "mysql",
                    "condition": "START"
                },
                {
                    "containerName": "exit1",
                    "condition": "SUCCESS"
                }
            ],
            "name": "nginx"
        },
        {
            "image": "mysql:5.7",
            "startTimeout": 30,
            "dependsOn": [
                {
                    "containerName": "exit1",
                    "condition": "SUCCESS"
                }
            ],
            "name": "mysql"
        }
    ],
    "memory": "100",
    "family": "reproduce-dependency-problem",
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "128"
}
With logically ordered multiple dependencies - task STOPPED as expected ✅
{
    "containerDefinitions": [
        {
            "command": [
                "cat",
                "123"
            ],
            "image": "alpine",
            "startTimeout": 10,
            "name": "exit1",
            "essential": false
        },
        {
            "image": "nginx",
            "startTimeout": 60,
            "dependsOn": [
                {
                    "containerName": "exit1",
                    "condition": "SUCCESS"
                },
                {
                    "containerName": "mysql",
                    "condition": "START"
                }
            ],
            "name": "nginx"
        },
        {
            "image": "mysql:5.7",
            "startTimeout": 30,
            "dependsOn": [
                {
                    "containerName": "exit1",
                    "condition": "SUCCESS"
                }
            ],
            "name": "mysql"
        }
    ],
    "memory": "100",
    "family": "reproduce-dependency-problem",
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "128"
}
Here, in the first task definiton, the task should have stopped much earlier, as exit1 has stopped unsuccessfully. But it was stuck in pending because the first mentioned dependency did not yet meet requirements.

This problem was happening because whenever a container was waiting on a dependency to resolve, instead of continuing to check for other dependencies, it would just return with error as dependency not resolved. This PR returns this error at the end of checking all dependencies. If we can fail faster, we will be able to do so after this fix.

Testing

Tested that the above mentioned tasks behaved as expected and both transitioned to STOPPED status

Description for the changelog

Bug: Fix the container ordering bug where the agent did not iterate through all the dependencies of a particular container

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@ubhattacharjya ubhattacharjya changed the title [WIP]: Solve unordered container dependency problem Solve unordered container dependency problem Sep 3, 2020
@ubhattacharjya ubhattacharjya marked this pull request as ready for review September 3, 2020 23:41
@ubhattacharjya ubhattacharjya requested a review from a team September 3, 2020 23:43
Copy link
Member

@fierlion fierlion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a test to cover this specific case?
(I could approve if you wanted to break such a test into a separate PR)

agent/engine/dependencygraph/graph.go Outdated Show resolved Hide resolved
agent/engine/dependencygraph/graph.go Outdated Show resolved Hide resolved
@ubhattacharjya
Copy link
Contributor Author

Can we add a test to cover this specific case?
(I could approve if you wanted to break such a test into a separate PR)

Added integration test

@ubhattacharjya ubhattacharjya requested a review from a team September 8, 2020 21:49
Copy link
Member

@fierlion fierlion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. Thanks for adding the test here.

@ubhattacharjya ubhattacharjya merged commit 417b7d6 into aws:dev Sep 9, 2020
@mssrivas mssrivas added this to the 1.44.4 milestone Sep 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants