Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for version and deduplication #1

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add support for version and deduplication #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1abd88e
add support for version and deduplication
Aug 28, 2024
8231cf4
update result count
Aug 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ All parameters need to be provided as environment variables:
| DD_PRODUCT_TYPE_NAME | Mandatory | Mandatory | If a product type with this name does not exist, it will be created |
| DD_PRODUCT_NAME | Mandatory | Mandatory | If a product with this name does not exist, it will be created |
| DD_ENGAGEMENT_NAME | Mandatory | - | If an engagement with this name does not exist for the given product, it will be created |
| DD_ENGAGEMENT_VERSION | Optional | - | If provided, the version is used as an additional filter to the name to find the matching engagement |
| DD_ENGAGEMENT_DEDUPLICATION | Optional | - | Default: false |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What did you need a deduplication flag?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

Different version of a "product" (redact-pipeline, redact-infer) will be tracked as different engaements. I don't want defect dojo to count identical finding across engagements as duplicates. Or otherway around each engagementt representing a differentt version should have it's independent findings.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are still only setting a portion of all possible fields.
For reference, these are all the field of an engagement:

{
      "id": 15,
      "tags": [],
      "name": "Trivy Cron Audit",
      "description": null,
      "version": null,
      "first_contacted": null,
      "target_start": "2023-11-01",
      "target_end": "2999-12-31",
      "reason": null,
      "updated": "2024-08-27T23:46:28.803182Z",
      "created": "2023-11-01T10:42:19.375365Z",
      "active": true,
      "tracker": null,
      "test_strategy": null,
      "threat_model": true,
      "api_test": true,
      "pen_test": true,
      "check_list": true,
      "status": "In Progress",
      "progress": "threat_model",
      "tmodel_path": "none",
      "done_testing": false,
      "engagement_type": "CI/CD",
      "build_id": null,
      "commit_hash": null,
      "branch_tag": null,
      "source_code_management_uri": null,
      "deduplication_on_engagement": false,
      "lead": null,
      "requester": null,
      "preset": null,
      "report_type": null,
      "product": 3,
      "build_server": null,
      "source_code_management_server": null,
      "orchestration_engine": null,
      "notes": [],
      "files": [],
      "risk_acceptance": []
    },

| DD_ENGAGEMENT_TARGET_START | Optional | - | Format: YYYY-MM-DD, default: `today`. The target start date for a newly created engagement. |
| DD_ENGAGEMENT_TARGET_END | Optional | - | Format: YYYY-MM-DD, default: `2999-12-31`. The target start date for a newly created engagement. |
| DD_TEST_NAME | Mandatory | - | If a test with this name does not exist for the given engagement, it will be created |
Expand Down Expand Up @@ -176,6 +178,13 @@ Another example, showing how to use `dd-import` within a GitHub Action, can be f

`./bin/runDockerUnitTests.sh` - First creates the docker image and then starts a docker container in which the unit tests are executed.

### Publish new Image

```
docker build -f docker/Dockerfile -t brightercore.azurecr.io/dd-import:latest .
docker push brightercore.azurecr.io/dd-import:latest
```

## License

Licensed under the [3-Clause BSD License](LICENSE.txt)
4 changes: 2 additions & 2 deletions bin/runDockerUnitTests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/sh

if docker build -f docker/Dockerfile -t dd-import:latest .; then
docker run --rm dd-import:latest ./bin/runUnitTests.sh
if docker build -f docker/Dockerfile -t brightercore.azurecr.io/dd-import:latest .; then
docker run --rm brightercore.azurecr.io/dd-import:latest ./bin/runUnitTests.sh
else
exit 1
fi
7 changes: 7 additions & 0 deletions dd_import/dd_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,9 @@ def new_product(self, product_type):
def get_engagement(self, product):
payload = {'name': self.environment.engagement_name,
'product': product}
if self.environment.engagement_version is not None:
payload['version'] = self.environment.engagement_version

r = requests.get(self.engagement_url,
headers=self.headers,
params=payload,
Expand All @@ -126,7 +129,11 @@ def new_engagement(self, product):
'target_start': self.environment.engagement_target_start,
'target_end': self.environment.engagement_target_end,
'engagement_type': 'CI/CD',
'deduplication_on_engagement': self.environment.engagement_deduplication,
'status': 'In Progress'}
if self.environment.engagement_version is not None:
payload['version'] = self.environment.engagement_version

if self.environment.source_code_management_uri is not None:
payload['source_code_management_uri'] = self.environment.source_code_management_uri
r = requests.post(self.engagement_url,
Expand Down
4 changes: 4 additions & 0 deletions dd_import/environment.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ def __init__(self):
self.product_name = os.getenv('DD_PRODUCT_NAME')
self.product_type_name = os.getenv('DD_PRODUCT_TYPE_NAME')
self.engagement_name = os.getenv('DD_ENGAGEMENT_NAME')
self.engagement_version = os.getenv('DD_ENGAGEMENT_VERSION', None)
self.engagement_deduplication = os.getenv('DD_ENGAGEMENT_DEDUPLICATION', 'False').lower() in ['true']
self.engagement_target_start = os.getenv('DD_ENGAGEMENT_TARGET_START', datetime.date.today().isoformat())
self.engagement_target_end = os.getenv('DD_ENGAGEMENT_TARGET_END', '2999-12-31')
self.test_name = os.getenv('DD_TEST_NAME')
Expand Down Expand Up @@ -60,6 +62,8 @@ def check_environment_reimport_findings(self):
print('DD_PRODUCT_TYPE_NAME: ', self.product_type_name)
print('DD_PRODUCT_NAME: ', self.product_name)
print('DD_ENGAGEMENT_NAME: ', self.engagement_name)
print('DD_ENGAGEMENT_VERSION: ', self.engagement_version)
print('DD_ENGAGEMENT_DEDUPLICATION: ', self.engagement_deduplication)
print('DD_ENGAGEMENT_TARGET_START: ', self.engagement_target_start)
print('DD_ENGAGEMENT_TARGET_END: ', self.engagement_target_end)
print('DD_TEST_NAME: ', self.test_name)
Expand Down
73 changes: 69 additions & 4 deletions unittests/test_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ def test_new_product(self, mockPost, mockEnv):
def test_get_engagement_found(self, mockGet, mockEnv):
response = Mock(spec=Response)
response.status_code = 200
response.text = '{\"count\": 2, \"results\": [{\"id\": 2, \"name\": \"engagement_dev\"}, {\"id\": 3, \"name\": \"engagement\"}]}'
response.text = '{\"count\": 3, \"results\": [{\"id\": 2, \"name\": \"engagement_dev\", \"version\": \"null\"}, {\"id\": 3, \"name\": \"engagement\", \"version\": \"null\"}, {\"id\": 4, \"name\": \"engagement\", \"version\": \"1.0.1\"}]}'
mockGet.return_value = response

api = Api()
Expand All @@ -168,6 +168,27 @@ def test_get_engagement_found(self, mockGet, mockEnv):
mockGet.assert_called_once_with(url, headers=self.header, params=payload, verify=True)
response.raise_for_status.assert_called_once()

@patch('dd_import.environment.Environment')
@patch('requests.get')
@patch.dict('os.environ', {'DD_URL': 'https://example.com',
'DD_API_KEY': 'api_key',
'DD_ENGAGEMENT_NAME': 'engagement',
'DD_ENGAGEMENT_VERSION': '1.0.1'})
def test_get_engagement_with_version_found(self, mockGet, mockEnv):
response = Mock(spec=Response)
response.status_code = 200
response.text = '{\"count\": 3, \"results\": [{\"id\": 2, \"name\": \"engagement_dev\", \"version\": \"null\"}, {\"id\": 3, \"name\": \"engagement\", \"version\": \"null\"}, {\"id\": 4, \"name\": \"engagement\", \"version\": \"1.0.1\"}]}'
mockGet.return_value = response

api = Api()
id = api.get_engagement(self.product_id)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this function return the first result?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess you need to select someway. Taking the first is a reasonable option. In the end there is no clear way what to select if there are more than one matches. The only other reasonable way would be to fail the script if more than one entry is found. However this was already like this from the original author,


self.assertEqual(id, self.engagement_id)
url = 'https://example.com/api/v2/engagements/'
payload = {'name': 'engagement', 'product': self.product_id, 'version': '1.0.1'}
mockGet.assert_called_once_with(url, headers=self.header, params=payload, verify=True)
response.raise_for_status.assert_called_once()

@patch('dd_import.environment.Environment')
@patch('requests.get')
@patch('dd_import.dd_api.Api.new_engagement')
Expand All @@ -191,6 +212,50 @@ def test_get_engagement_not_found(self, mockNewEngagement, mockGet, mockEnv):
mockNewEngagement.assert_called_once_with(self.product_id)
response.raise_for_status.assert_called_once()

@patch('dd_import.environment.Environment')
@patch('requests.post')
@patch.dict('os.environ', {'DD_URL': 'https://example.com',
'DD_API_KEY': 'api_key',
'DD_ENGAGEMENT_NAME': 'engagement',
'DD_ENGAGEMENT_VERSION': '1.0.1'})
def test_new_engagement_with_version(self, mockPost, mockEnv):
response = Mock(spec=Response)
response.status_code = 200
response.text = '{\"id\": 3}'
mockPost.return_value = response

api = Api()
id = api.new_engagement(self.product_id)

self.assertEqual(id, self.engagement_id)
today = datetime.date.today().isoformat()
url = 'https://example.com/api/v2/engagements/'
payload = f'{{"name": "engagement", "product": 2, "target_start": "{today}", "target_end": "2999-12-31", "engagement_type": "CI/CD", "deduplication_on_engagement": false, "status": "In Progress", "version": "1.0.1"}}'
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does deduplication_on_engagement influence the version?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No why?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure what payload does, but there is here "version": "1.0.1" and line 255, there is no version

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

payload is the expected body of the internal rest request. This tool is "configured" using env vars and creates rest requests accordingly. In line 220 you see that the DD_ENGAGEMENT_VERSION is set. That's why the tool includes version in the rest request. DD_ENGAGEMENT_DEDUPLICATION is not set so the request includes the default value. For Version there is no default. If it isn't defined it will not be in the body.

mockPost.assert_called_once_with(url, headers=self.header, data=payload, verify=True)
response.raise_for_status.assert_called_once()

@patch('dd_import.environment.Environment')
@patch('requests.post')
@patch.dict('os.environ', {'DD_URL': 'https://example.com',
'DD_API_KEY': 'api_key',
'DD_ENGAGEMENT_NAME': 'engagement',
'DD_ENGAGEMENT_DEDUPLICATION': 'True'})
def test_new_engagement_with_deduplication(self, mockPost, mockEnv):
response = Mock(spec=Response)
response.status_code = 200
response.text = '{\"id\": 3}'
mockPost.return_value = response

api = Api()
id = api.new_engagement(self.product_id)

self.assertEqual(id, self.engagement_id)
today = datetime.date.today().isoformat()
url = 'https://example.com/api/v2/engagements/'
payload = f'{{"name": "engagement", "product": 2, "target_start": "{today}", "target_end": "2999-12-31", "engagement_type": "CI/CD", "deduplication_on_engagement": true, "status": "In Progress"}}'
mockPost.assert_called_once_with(url, headers=self.header, data=payload, verify=True)
response.raise_for_status.assert_called_once()

@patch('dd_import.environment.Environment')
@patch('requests.post')
@patch.dict('os.environ', {'DD_URL': 'https://example.com',
Expand All @@ -208,7 +273,7 @@ def test_new_engagement_without_target(self, mockPost, mockEnv):
self.assertEqual(id, self.engagement_id)
today = datetime.date.today().isoformat()
url = 'https://example.com/api/v2/engagements/'
payload = f'{{"name": "engagement", "product": 2, "target_start": "{today}", "target_end": "2999-12-31", "engagement_type": "CI/CD", "status": "In Progress"}}'
payload = f'{{"name": "engagement", "product": 2, "target_start": "{today}", "target_end": "2999-12-31", "engagement_type": "CI/CD", "deduplication_on_engagement": false, "status": "In Progress"}}'
mockPost.assert_called_once_with(url, headers=self.header, data=payload, verify=True)
response.raise_for_status.assert_called_once()

Expand All @@ -230,7 +295,7 @@ def test_new_engagement_with_target(self, mockPost, mockEnv):

self.assertEqual(id, self.engagement_id)
url = 'https://example.com/api/v2/engagements/'
payload = '{"name": "engagement", "product": 2, "target_start": "2023-02-01", "target_end": "2023-02-28", "engagement_type": "CI/CD", "status": "In Progress"}'
payload = '{"name": "engagement", "product": 2, "target_start": "2023-02-01", "target_end": "2023-02-28", "engagement_type": "CI/CD", "deduplication_on_engagement": false, "status": "In Progress"}'
mockPost.assert_called_once_with(url, headers=self.header, data=payload, verify=True)
response.raise_for_status.assert_called_once()

Expand All @@ -252,7 +317,7 @@ def test_new_engagement_with_source_code_management_uri(self, mockPost, mockEnv)
self.assertEqual(id, self.engagement_id)
today = datetime.date.today().isoformat()
url = 'https://example.com/api/v2/engagements/'
payload = f'{{"name": "engagement", "product": 2, "target_start": "{today}", "target_end": "2999-12-31", "engagement_type": "CI/CD", "status": "In Progress", "source_code_management_uri": "https://github.com/MyOrg/MyProject/tree/main"}}'
payload = f'{{"name": "engagement", "product": 2, "target_start": "{today}", "target_end": "2999-12-31", "engagement_type": "CI/CD", "deduplication_on_engagement": false, "status": "In Progress", "source_code_management_uri": "https://github.com/MyOrg/MyProject/tree/main"}}'
mockPost.assert_called_once_with(url, headers=self.header, data=payload, verify=True)
response.raise_for_status.assert_called_once()

Expand Down