Access to OpenAPI example value in portman?

[jpjpjp]

Hi,

I've noticed that, without any particular configuration, Portman generates Contract Tests that use the values specified in the examples provided in the OpenAPI spec for request parameter and request body attribute values.

Can these example values be accessed in Portman configuration rules or Postman scripts?

To provide some context in the ask, I've been trying to generate a single test configuration that I can use while I'm designing a new API with a mock service. Our goal is to have a single test configuration that we can run against the mock service in the design phase and then hand it off to the implementation team to validate that their implementation is correct.

The one area where we have struggled is with the Contract Tests. In a real service you typically want your contract tests to dynamically use the id created in a POST request in subsequent RUD operation tests. Portman provides this syntax for making this happen:

# Set a general rule to assign a postman id variable from the response bodies
# of any POST or GET requests
assignVariables:
  - openApiOperation: POST::*
    collectionVariables:
      - responseBodyProp: id
        name: <tag>Id
  - openApiOperation: GET::*
    excludeForOperations:
      - "GET::/*/{id}"
    collectionVariables:
      - responseBodyProp: <tag>[0].id
        name: <tag>Id
# Set a general rule to use the most recently assigned id variable from the 
# previous POST or GET request as the id to use in any requests that take an id
overwrites:
  - openApiOperation: "*::/*/{id}"
    overwriteRequestPathVariables:
      - key: id
        value: '{{<tag>Id}}'
        overwrite: true

However, I don't want to use this dynamic ID from the POST response if I run my test against a mock service with canned data. Instead, I'd like to use the default example values from the spec. In a previous question, I learned that there isn't a way to add any conditional logic to the overwriteRequestPathVariables so I decided to work around this problem by using a prerequest script to overwrite, the previously overwritten variable. To do this I add the following to the configuration shown above:

operationPreRequestScripts:
  - openApiOperation: "*::/*/{id}"
    scripts:
      - file:portman/preMockRequestIdOverwrite.js

Which generates adds this as pre-request script:

// Pre-request script to check for isMockService variable and update categoriesId if true
pm.collectionVariables.get("isMockService") === "true" && pm.collectionVariables.set("categoriesId", 83);

Here I'm setting the value of the environment variable that will be used to set the request path parameter on the request back to the example value from my OpenAPI spec. But it is hard coded. This isn't the end of the world, but Portman seems to have the examples available somewhere, I wondered if it was possible access those values in order to make my script more dynamic.

Thanks!

[thim81]

hi @jpjpjp

I'm trying to fully understand how your setup looks like.
It sounds complicated to use preRequests to overwrite variables.

Some dummy questions/ideas:

• Do you have a pipeline to run the tests against? Or do you want to use the Postman collection manually for both scenario's?
• Would it be an option to generate 2 postman collections? One for using the real data (with the CRUD assign variables flow) and another postman collection that uses the default examples with only the contract test but without the assign var/overwrites that has the mock service as the base URL?
• Which Mock service are you using? So I can experiment with it to better understand your setup. I've used in the past Prism in combination with Portman.
• Would it not be easier to align the Mock canned data with your default examples? Like you would setup DB seeding for resetting any staging/pre-prod environment. That way you provide guides (and public facing mocks) based on your examples in your OpenAPI and any public docs that you render from it, with example values that are realistic.
• Another path, you could explore is assigning variables from the requestBody? As a way to "Can these example values be accessed in Portman configuration rules or Postman scripts?"

requestBodyProp (String) : The property for which the value will be taken from the request body and set the value as the pm.collectionVariables value.

[jpjpjp]

hi @jpjpjp

I'm trying to fully understand how your setup looks like. It sounds complicated to use preRequests to overwrite variables.

Let me try to explain a bit more. I'm working on a project where we are taking a "Design First" approach to the next version of an API. As the designer, I write the spec, generate the docs, write the tests, and (to answer one of your questions below) implement a node.js express mock service with static mock data that hopefully demonstrates how the API should handle all the expected/possible error conditions. Part of the reason I chose this approach was to be able to hand off this "mock code" to an implementation team and tell them that all they need to do is "change the implementation of the handlers".

It's probably worth being explicit that in my mock API server, when I make a POST /xxx request, the server generates a response as if the new xxx element is created, but in fact, it isn't. The data is static. Because of this, the "out of the box" contract tests pass when running using input data from the examples. Of course when running tests on the real service we want to use the id of the recently created object on the remainder of the contract tests.

My goal is to create a Portman config that exercises contact tests and variation tests against all the mock endpoints, but also give this same config to the implementation team and tell them "your implementation isn't complete until all these tests pass". I guess this adds a little bit of complexity to the Portman config, but hopefully it's helpful for the implementation team to get a pre-built set of tests.

In fact, we are already following this process. Today, I comment out the lines in the overwrites section in my example above when running the tests locally, to have the requests use the example values in the requests. Then, before I commit any changes to our shared repo, I uncomment out the directive to set those request parameters based on the previous response.

However, when I read about the possibility of running preRequest tests I realized that I could avoid doing that. An, in fact it works! However, to make this my standard way of doing this I need to hard code the correct example values for each of the endpoints, perhaps as postman variables in configured in my private .env-portman file.

It occurred to me that Portman does know what the example values are, though, so I thought I'd ask here if they are available to access in the configuration. It's OK if the answer is no. I can certainly work around it ;-)

I hope that clarifies the WHY of the request. For your other questions:

• Do you have a pipeline to run the tests against? Or do you want to use the Postman collection manually for both scenario's?

They run in a pipeline with newman. We do tend to load the tests into Postman to debug failures though.

• Would it be an option to generate 2 postman collections? One for using the real data (with the CRUD assign variables flow) and another postman collection that uses the default examples with only the contract test but without the assign var/overwrites that has the mock service as the base URL?

Yes, that would be possible, but I always worry that when you have two files, they will get out of sync.

• Which Mock service are you using? So I can experiment with it to better understand your setup. I've used in the past Prism in combination with Portman.

I use a manually written mock service, but I'd like to use Prism going forward since the implementation team now knows how to implement handlers and doesn't need me to provide templates. Exploring this more is on my to-do list. However, I'm somewhat skeptical that I can (or should) put enough examples in the spec to generate all the strange error conditions I want to ensure our implementation handles.

If you'd ever like to have a more detailed conversation about your experience with Prism, I'd be happy to do that!

• Would it not be easier to align the Mock canned data with your default examples? Like you would setup DB seeding for resetting any staging/pre-prod environment. That way you provide guides (and public facing mocks) based on your examples in your OpenAPI and any public docs that you render from it, with example values that are realistic.

Yes, I've considered having a mock service with a real DB. Still, since we plan to make the mock service available to our developer community to get feedback on the design, I thought managing that would become more complicated than it was worth.

• Another path, you could explore is assigning variables from the requestBody? As a way to "Can these example values be accessed in Portman configuration rules or Postman scripts?"

requestBodyProp (String) : The property for which the value will be taken from the request body and set the value as the pm.collectionVariables value.

Yes, I can think of other ways to return to the examples, so again, this was more of a curiosity-based question.

As always, I really appreciate both your attentiveness to these discussions and, of course, Portman!

[thim81]

hi @jpjpjp

About Prism, I'm a fan and include them in my Spec-Driven API Development but there are some limitations/quirks you have keep in mind. For quick prototyping is really handy. For cases where business logic is involved, any pure OpenAPI mock service would probably be too limited.

Always happy to have a conversation, so let me know when and how so we can set it up.

About accessing the OpenAPI examples directly using a Portman directive, I have thinking of options or angles but can't find a straightforward way. The most straightforward option would be assign Variables from the requestBodyProp (which would be the OpenAPI example).