Add guidance on how to write security test cases (or threat verification tests)

[itscooper]

Guidance, to be added to section 5 (reporting), on how a tester can write programmatic test cases as a form of output, that developers can re-run to determine if an issue has been fixed.

[victoriadrake]

Is instruction on regression testing in scope for the WSTG? Or have I got the wrong idea?

[ThunderSon]

@victoriadrake we believe (me and @kingthorin ) that it could be a nice to have, nothing core to the project or its latest release. The project focuses more on guidance a tester through the stages in a manual approach more so than an automated one.

[patrickceg]

The project focuses more on guidance a tester through the stages in a manual approach more so than an automated one.

@ThunderSon Does that mean there's another OWASP project that focuses on adversarial unit tests that we should make sure to link to?

On this issue in general, the framework of Defect Dojo seems to work as a bit of a guide to write security tests: https://defectdojo.readthedocs.io/en/latest/models.html#engagements / https://owasp.org/www-project-defectdojo/

Personally, I handle security type tests in the same way as any other test (maybe I saw too much DevSecOps idealism marketing), so I'm not sure what this guide has to say about what makes a "security test case" different from any other test. Everything I can think of (business importance, which team is responsible for it, who to call if the feature underlying said test fails in production and forces people in at 02:00 on a Sunday) apply to both "security" and "non-security" tests.

[ThunderSon]

What I read from DD (defect dojo) doesn't actually fit what is being discussed in here.
The project can contain these points, but at this stage, we see a bigger value in actually updating the tests written, adding the missing tests, and mapping to the ASVS.
Tests discussed in this issue are part of the automated pipeline that could be run.
If you feel like tackling this point, or anyone, we don't mind it. If this is something that is of interest to you or any other future reader, please take it. 😄

[kingthorin]

@itscooper any chance you want to contribute some content to address this?

[Hsiang-Chih]

"how a tester can write programmatic test cases as a form of output, that developers can re-run to determine if an issue has been fixed."?
I can help if it's the "BDD Security testing". It will require the integration of BDD framework.
However, is the topic "How to do Behavior-Based Driven Security Testing" what OWASP testing guide would like to add?

i.e. By applying the BDD security, and testing cases/report will be -
Testing Scenario - There is no port 80 open listening
Testing Steps

1. Use Nmap to scan the target website
2. Check if there is port 80 open

Results
There is no port 80 open

[alp1n3-eth]

I saw this in the v4.3 release milestones and wanted to double-check a few things ensure I have the experience necessary to cover this. The original issue is:

on how a tester can write programmatic test cases as a form of output, that developers can re-run to determine if an issue has been fixed.

This would be nice, but I wonder about the actual application of it. Normally these would be solved with a remediation test, as compliance in some corps requires a 3rd party to confirm a vulnerability has been remediated. It would also add a significant amount of overhead to time-boxed testing, and would require decisions to be made surrounding format. Would it be a Go script, Python script, or would it possibly even use newer tooling such as Hurl? It also could be a vuln that is not easily programmatically reproducible for a number of reasons: browser-based, a large amount of intricate steps involved, etc.

The reporting section does hit on reproduction steps:

Detailed description of what the vulnerability is, how to exploit it, and the damage that may result from its exploitation. Any possibly-sensitive data should be masked, for example, passwords, personal information, or credit card details.

And mentions extra ways to help readers of the report:

Additional resources that could help the reader to understand the vulnerability, such as an image, a video, a CVE, an external guide, etc.

If a smaller change is desired, it could be appended to the last quote posted above, as I think it qualifies as "Additional Resources" that are technically optional on top of the required items, like text-based reproduction steps.

If a larger change is desired, I could also add these into either different sections, or new files, as just the act of writing reproduction steps can get pretty deep. I don't know if stylistic guidelines would be welcome there as well, such as suggesting how to format the screenshots to make their intent clear, wording choices for clarity, etc.

Smaller change could also be v4.3, larger change planned for v5. Just let me know! 😁

[kingthorin]

I think it can be something small.

We don't need to suggest a specific language or solution as they might be different for different issues/vulns. It could simply (IMHO) list things like generating a curl command, generating a CSRF form/PoC, creating a HAR, creating a selenium or puppeteer test/script, etc.