statusCode to reflect what is available in the openAPI

[savage-alex]

I am trying to create a generic catchall contract test starter.

One of the things I noticed was that when using statusCode

{
                "openApiOperation": "*::/*",
                "statusCode": {
                    "enabled": true
                }
            },

The test created in postman always requires 200 to be returned even if that is not what is in the OpenAPI.
I understand statusCode takes in a code parameter but would it be beneficial if that is not present to allow any of the codes from that operations openAPI to be present?

Love to hear your thoughts?

[thim81]

Hi @savage-alex,

The Portman contract tests are organised around the following concepts:

• contract test = Happy path, verify the succes response which uses by default the 200 request OpenAPI specification to generate the tests: statusCode, JSON schema, ...
• Variation contract test = test variations of the 200 OpenAPI specification (like different query params, headers, ...) + unhappy path (4xx, 5xx) tests, for which you can define the targeted OpenAPI response.

For variation tests, you can instruct Portman which OpenAPI response you want to test against by setting the openApiResponse

openApiResponse (String | optional) : References to the OpenAPI response object code/name for which a variation will be created. (example: "404"). If not defined, the 1st response object from OpenAPI will be taken as expected response. If the configured openApiResponse code is not defined in the OpenAPI document, Portman will not generate a variation for the targeted operations.

If the openApiResponse is defined, the "statusCode" test does not require the "code" parameter, since it will verify against the expected openApiResponse code, but you can overwrite it if you want the tests to be generated based on the OpenAPI response but you expect your API to return a different HTTP code.

We have a variation example that showcases the usage openApiResponse, which hopefully makes it a bit clearer.
https://github.com/apideck-libraries/portman/tree/main/examples/testsuite-variation-tests#example-explained

[savage-alex]

Ok. Its not to do with the 400x responses on this operation but actually when you add another operation on that path such as GET, it impacts the 201
AdvancedComputerSoft-desks-post.yaml.txt
.

[thim81]

That seems like a bug. Using your YAML, I can reproduce it and I'll try to figure out where the bug happens.

[savage-alex]

Thanks @thim81, We are doing training on it so trying to make a super generic contract test we can throw at any API to get people started. Was a lightbulb moment today - can we check status codes as currently just doing the "statusSuccess" check as I had sadly seen OAI saying return a 201 and the API returning a 200 which it wouldnt currently be able to warn.
Always a pleasure. Hope we can share a beer at APIDays or similar one day

[AliKhalili]

@thim81 Thank you for your amazing tool.
I tried to fix this issue here 241. Hope you find this useful.

[thim81]

The fix from @AliKhalili got merged in and will be part of the next release, which is planned in the coming days with some other small bug fixes.

[thim81]

FYI: In theory, you don't need the combination of statusCode + statusSuccess for a contract test, since the Postman assertion used: pm.response.to.be.success; will cover any HTTP 2xx (200,201,2..) response.

By adding the statusCode check, you make the validation stricter, which can be desired if needed.

1. statusSuccess
2. statusCode

{
  "version": 1.0,
  "tests": {
    "contractTests": [
      {
        "openApiOperation": "*::/*",
        "statusSuccess": {
          "enabled": true
        }
      },
      {
        "openApiOperation": "*::/*",
        "statusCode": {
          "enabled": true
        }
      },

[savage-alex]

100% we are intending to move from the success to the statusCode function for more stricter validation.

[thim81]

@savage-alex The statusCode fix from @AliKhalili has landed in the latest release 1.13.1 from Portman.

[thim81]

@savage-alex FYI: We inject contract tests (happy path) for all OpenAPI responses ranging from 200 till 302

So if you have a 201 or 204 or even a 302, this will be considered the happy path for the normal contract tests, if there are multiple "happy path" responses are defined, the 1ste happy response will be used as expected response for which Portman will generate the contract tests.

For variation tests, you can define the expected OpenAPI request using openApiResponse setting.

[savage-alex]

Answered my next question of what if you have more than one 200/300. It takes the first. Thank you

[savage-alex]

Tested this with the latest release and it works great. How can we close this dicussion please?

[thim81]

We can leave it here, for people we run into similar thoughts.