Context property annotations

[darrelmiller]

JSON Schema is not aware of the context of usage of the schemas that it defines. It has no notion that a schema is being used to create a resource, update it, or read it.

This leads people today to create multiple schemas so that there can be an accurate schema for each request or response content. However, this introduces duplication or requires splitting the schema in order to reuse parts of the schema that don't change between responses. This gets hard for documentation and code generators to process.

I am proposing that we introduce four new keywords that are boolean flags into the OAS JSON Schema vocabulary:

• computed - Generated by the server
• immutable - Can be provided on create but then read-only
• requiredForCreate - must be present when creating
• writeOnly - Can only be provided on create or update

Example:

   todo:
        type: "object"
        title: "Todo"
        properties:
            dateCreated: 
                type: string
                computed: true  
            owner:  
                type: string
                requiredForCreate: true 
                immutable: true
            title:   
                type: string
                requiredForCreate: true  
            secret:
                type: string
                writeOnly: true  

While this is a proposal aimed at Moonwalk, it would be possible to introduce these in OpenAPI 3.1.

[kevin-postman]

I have mixed feelings about this to be honest. On the one hand.. I agree that it would be very helpful to understand on a per property level how it could be used. II think along the lines of the variety of tools that will consume this document and do something with it. Doc tools are now going to need to add logic to handle each of these additions and what it means for the generated documentation. Then you have the variety of code gen tools, mock tools, etc.. are ALL of them by every vendor/author going to utilize these in the same way so that it's guaranteed consistent across tooling? Doubtful.

But then you can take it further.. what about adding properties for things like RBAC? Maybe I want my "owner" to be mutable if the user is an Admin.. but if it's a normal user.. they don't even get this property. Similar to the idea that we have individual objects for different operations based on the role of the API consumer. Admin gets the full shabang.. every property.. full control. But a manager role only gets some of the properties.. and a standard user gets even less.. read only.. with only their name/email being able to be changed. How do you account for all those? Granted you could have RBAC specific endpoints instead.. but you're still going to likely end up having duplicate definitions or overlap to cover the different scenarios that could exist.

[darrelmiller]

I think the goal is to come to a common set of terms and standardize them as part of a specification in order to try and drive consistent usage across the tooling ecosystem. I have no illusions that it is going to be easy, but the inability to describe these characteristics in a standardized way has been getting in the way of people describing their APIs for quite a while now. It seems like a valuable problem to try and solve.
I think RBAC is certainly an additional challenge and something that could be used to further constrain what is possible for app/user. I don't think RBAC rules should ever be used to relax rules defined on types. That would just get unmanageable.

[MikeRalphson]

writeOnly is already defined as an annotation in Json Schema 2020-12.

[ralfhandl]

As is readOnly, which seems to have the same meaning as computed without the baggage of "this field is not computed on the fly, it is persisted and you just can't change it, so I don't like to tag it as computed".

[handrews]

Here is an edited-down version of a similar idea I described on the JSON Schema Slack, and fortunately saved. I've put a few things in bold italics for folks who just want to skim to the interesting bits.

As noted below, this was very much off the top of my head, so it's not so much a serious alternative proposal as a sketch of some things to consider. It overlaps with @darrelmiller's proposal, but takes into account and explains the mechanics of JSON Schema validation and annotation interactions, which requires a different keyword design.

---

Annotations are how an application (including what we call "downstream specifications" like OAS) can build additional functionality on top of JSON Schema's instance-driven and context-unaware behavior.

So let's look at the HTTP context cases here:

• read-write optional: the field can be there or not, changed or not, no matter the method
• read-write required: the field must always be there, but can be changed
• create-only required: (on a POST, or on a PUT or even PATCH that creates a resource) you have to include it in the creation representation, it will always be present in GET, but you can't change it after that.
• write-once: same as above, except that you can set the field once at any time, and then can't change it further. But you might omit it from the create call and set it on a later PUT or PATCH
• read-only required: it's always present in GET but cannot be changed
• read-only optional: it may or may not be present in a GET, but cannot be changed if present or added if absent
• write-only required: it's never present in GET but always has to be sent in PUT (or PATCH, I guess) - this would be a password field on a resource specifically for setting the password that also has some other data
• write-only optional: it's never present in GET but can be set in POST/PUT/PATCH = password fields on resources that have many settable fields, but treat a missing password as a non-change, even in a PUT (because omitting it from the GET doesn't mean there isn't a password, therefore omitting it from PUT doesn't mean it's deleting the field)
  write-after-create: You can't include it on a POST but you can change it later (I'm not going keep doing required vs optional, you get the idea)
• server-managed: The field is present in a GET, but no matter what you send in a PUT, PATCH, or POST it won't affect the value (e.g. the timestamp case mentioned before)

readOnly and writeOnly alone can't handle these combinations, and required would cause failures that would prevent you from collecting the necessary annotations. You need to start with a permissive schema, plus new annotations.

These new annotations would, like required, have to be set on the object schema rather than the individual fields. This is because an annotation can only be applied to an instance that exists.

So, with a good vocabulary that expressed the above semantics you would do this:

1. Use a JSON Schema validator that supports annotations and validate the request/response payload with it, getting the annotations as part of the result
2. In a separate piece of code that is aware of the HTTP method, whether it's a request or response, what headers were present, etc., read the annotations and determine if any of the context-sensitive constraints were violated.

Annotation results include a JSON Pointer for the instance value, so you can check the value for further processing if needed. In this approach, you'd never use required for conditionally-required fields. You'd use something like this (which I am making up on the spot, don't take it too seriously):

{
  "properties": {
    "id": {
      "type": "integer",
      "minimum": 1
    },
    "lastModified": {
      "type": "string",
      "format": "date-time"
    }
  },
  "httpConstraints": {
    "id": {
      "required": ["GET", "PUT"],
      "forbidden": ["POST"],
      "unmodified": ["PUT", "PATCH"]
    },
    "lastModified": {
      "required": ["GET"],
      "serverManaged": true
    }
  }
}

httpConstraints here annotates the whole object, grouping the specifics by field. We avoid having to name too many possible combinations by indicating the HTTP method(s) to which the constraints apply. There would need to be some thoughts on how to handle re-use (the value of httpConstraints is not a schema - the whole blob is a single keyword with a complex value).

You would have a separate HTTP-aware validation tool that would consume the annotation output from JSON Schema validation and perform additional validation. Annotation vocabularies are easy to implement (and some implementations treat unknown keywords as annotations by default, as the 2020-12 spec recommends, meaning they do not require any custom code).

Because annotation output includes JSON Pointers to the schema and instance locations, this 2nd HTTP-aware validation pass would be much simpler to implement than JSON Schema. But it would correctly keep the instance data concerns and HTTP context concerns separate.

[darrelmiller]

The httpConstraints property is interesting. It is a bit verbose but it is very explicit and I find it easy to read.
Using HTTP methods doesn't quite capture the "requiredForCreate" semantics because using a PUT or a PATCH, only the server would be able to identify if a create occurs. It means that a client would not be able to enforce the presence of the property which makes it a bit useless for JSON Schema validation, but it could be useful for server code generators.

I'm not quite sure I get the difference between "forbidden" and "unmodified". Is the distinction whether the client allows the property to be included in the request?

[handrews]

@darrelmiller I believe:

    "id": {
      "required": ["GET", "PUT"],
      "forbidden": ["POST"],
      "unmodified": ["PUT", "PATCH"]
    }

means that this "id" field is a typical SQL-assigned identifier. It MUST NOT be present on POST, MUST be present and unmodified on GET or PUT. I had to think about that for a moment, so I agree it's not the most elegant way to capture that. I did want to distinguish between server-assigned (which is this "id" use case) and server-managed (which is the "lastModified" or similar use case). The distinction being that trying to write to a server-managed timestamp is generally ignored, but trying to change a server-assigned identifier is definitely an error.

These would definitely be useful for HTTP-aware server-side validation (where you could check the database state). They could also help with client-side validation in a GET-modify-PUT cycle, which could catch a modification in PUT from the last GET. The client could also catch it being used in a PATCH which would be either pointless or an error.

IIRC, I came up with this as I was typing it and just maximized flexibility. Which breaks things down into very small constraints. An alternative would be to define some of these common patterns (like server-assigned field) into their own keywords. requiredForCreate could be another one — and I agree that httpConstraints cannot express that as shown, because it's dependent entirely on server state, and only incidentally related to HTTP method choice.

There's a weirdness here that I probably didn't think about, which is that the POST here is on the collection resource rather than the item resource. I'm sure I can come up with a way to justify that 🙃, but keywords for usage patterns would probably be more intuitive.

There's probably some sort of balance to be struck between usage patterns that are best identified by a concise name, and flexibility to cover cases that aren't recognizable patterns but map predictably to HTTP methods.

The one critical thing is that it has to go at the object level to be used for runtime validation (via annotation output) of any sort. The current readOnly and writeOnly keywords are problematic for that reason, although in practice since it's the presence of the field in some contexts that causes problems, they mostly work out.

[garethj-msft]

"There's probably some sort of balance to be struck between usage patterns that are best identified by a concise name, and flexibility to cover cases that aren't recognizable patterns but map predictably to HTTP methods."

I think this is where I'd like a bit of a middle ground. Cos although it is generally readable, when it requires a somewhat complex combinatorial to express a fairly common case, then I'd prefer a more direct representation such as Darrel's 'requiredForCreate'

Is there a sort of aliasing that might work for that? It would make it easier to compare values at scale too.

[handrews]

@garethj-msft we don't particularly need to include httpConstraints in any proposal. I posted it to show one extreme- maybe it's a useful advanced mode sort of thing, or maybe it's just a conversation starter here.

On the other hand, requiredForCreate poses challenges for generic tooling. It's one thing for the API implementation itself to use that field - it presumably knows how the resource is created. But would a generic validation tool need to check all PUT and PATCH requests to the resource? And how would it figure out which POST is relevant? (The challenge with POST is shared by the httpConstraints approach). We might need to think about how collection and item resources are connected, and how any of this impacts API requests that don't map smoothly onto collection+item CRUD.

[ralfhandl]

This leads people today to create multiple schemas so that there can be an accurate schema for each request or response content.

Fully agree, we are generating three schemas of type:object from each API model structure: for read, for create, and for update, for example https://api.sap.com/api/BANK_0002/schema.

The generator is controlled by model annotations that are very similarly named to Darrel's proposal 😄

The four four keywords above cover the most common cases.

There's a fifth keyword one whose necessity I initially doubted and then learned better:

• updateOnly - Can only be provided on update