Improve explanation of Reference Object

[jharmn]

With the addition of relative file references (notably in swagger-js, and samples), there should be an explanation of how that resolves (based on JSON Reference). Samples/petstore-separate provides a reference point that could be utilized.

[jharmn]

On further review, the REUSE guidelines should be updated too:
https://github.com/swagger-api/swagger-spec/blob/master/guidelines/REUSE.md

[webron]

@jasonh-n-austin - are you referring specifically to how relative references are resolved?

[jharmn]

Right, so perhaps examples of what $ref links should/shouldn't look like, and what they should be relative to in pathing.

One example:

swagger.yaml
$ref: 'models/Thing.yaml'

Thing.yaml
$ref: 'models/AnotherThing.yaml' # note path is relative to swagger.yaml, not Thing.yaml

Also, deep links such as:

$ref: 'models/Common.yaml#Address' # an object within a schema file

All of this is really just how JSON Reference works, but some practical examples would probably help devs implementing the Swagger 2 spec get it right more consistently.

[john-kurkowski]

👍 been burning a lot of cycles reverse engineering what references are supported. REUSE.md seems to indicate only absolute references.

[mission-liao]

@jasonh-n-austin Hi, I'm refactoring my python client for external file reference, and feel confused with these:

swagger.yaml
$ref: 'models/Thing.yaml'

Thing.yaml
$ref: 'models/AnotherThing.yaml' # note path is relative to swagger.yaml, not Thing.yaml

In another discussion, one of your comment:

Each file resolves it's own refs, regardless where they're included from. In the swagger.yaml->Category->Person example, the $ref in Category is resolved from it's own location to find Person (as opposed to the location of swagger.yaml).

In the first comment, the base URI used to resolve 'models/AnotherThing.yaml' should be derived from swagger.yaml, although the $ref resides in Thing.yaml. But the comment in another issue states that the base URI should be derived from Thing.yaml.

Which one of my understanding is correct? Thanks for any feedback in advance. 👍

[whitlockjc]

I would expect relative references to be based on the location of the document containing the reference.

[jharmn]

I tend to agree with you @whitlockjc (clearly contradicting my earlier statements). I think I was basing the examples on the way that @BigstickCarpet 's swagger-parser resolves JSON references (which could be incorrect here).

Per the RFC for JSON Reference (https://tools.ietf.org/html/draft-pbryan-zyp-json-ref-03):

Resolution is performed relative to the referring document.

[whitlockjc]

When working on swagger-tools and sway, I created json-refs just for this. While I'm not saying it's perfect or should be the guide, it's been well received. Ever since supporting relative references in json-refs, relative references have always been resolved as I mentioned above and I'm yet to have anyone complain about it or suggest otherwise. While the sample set is small, this tells me that the intuitive approach matches what I suggested above as well.

[jharmn]

FWIW, I put together a sample spec which has separated files (https://github.com/OAI/OpenAPI-Specification/tree/master/examples/v2.0/json/petstore-separate). IIRC I ran swagger-tools across this to validate that everything was resolving right. This should serve as a working example for other implementers for the time being.

[mission-liao]

I'm still confused with that RFC :(

Resolution is performed relative to the referring document.

In the example above, should the referring document be swagger.yaml or Thing.yaml when resolving 'models/AnotherThing.yaml'?

@BigstickCarpet very clear, thanks.

[JamesMessinger]

I agree with @jasonh-n-austin and @whitlockjc that relative references should be based on the location of the document containing the reference. Swagger Parser 1.x and 2.x both resolved relative paths based on the main document rather than the containing document, but there were at least three problems with that:

1. It proved to be very fragile. Any time you moved files around or changed your folder structure, you had to go find and change every relative reference. It was a pain in the ass.

2. It didn't match the way people were using Swagger in the real world. It was a constant point of confusion/frustration for people who were using Swagger Parser.

3. And this is the big one - the JSON Reference specfication is very clear about how relative path resolution should behave:

   If the URI contained in the JSON Reference value is a relative URI,
   then the base URI resolution MUST be calculated according to
   [RFC3986], section 5.2. Resolution is performed relative to the
   referring document.

Swagger Parser 3.0 was a complete rewrite, and one of the main goals was to make it fully compliant with all of the specifications involved (Swagger 2.0, JSON Schema, JSON Hyper-Schema, JSON Reference, and JSON Pointer). As a result, relative references are now resolved relative to the containing document.

[jharmn]

I'll see what I can do about finally getting around to writing something up in REUSE.md...this discussion helped nail down some of the potential ambiguity.

[whitlockjc]

I'm not sure there was ambiguity other than at some point, we have to call these what they are: JSON References. Thankfully, if we can do that the JSON Reference specification is very clear on things. I've had no problem understanding the spec when working on my tooling and I'm sure @BigstickCarpet has had similar success writing his.

At the same time, I doubt anyone would complain with more documentation if it meant making things simpler to understand.

[JamesMessinger]

👍 to that

[jharmn]

Used part of my open dev day and did some writing for PR #535, I'd love feedback @whitlockjc & @BigstickCarpet

[webron]

@jasonh-n-austin Thanks for PR! Overall it looks good to me. I would reorganize the guide a bit, but I'd suggest we'll merge your PR first, and then I'll submit another PR reorganizing it for everyone here to review. Basically, I'd split it into cases, and change some headers and topic order, just to create a clearer flow, not changing the content itself.

Per your request, I'll wait for the feedback from @whitlockjc and @BigstickCarpet before merging.

[JamesMessinger]

@jasonh-n-austin - All of your edits look good to me. But there are a few other parts of the document that I feel are a bit unclear:

1. I'm not sure exactly what this line under "Operations" means:

Remember, you cannot override the definitions, but in this case, you can add additional operations on the same path level.

What exactly is it referring to? It doesn't seem to pertain to anything in the code sample immediately preceding it. Maybe it just needs to be reworded?

2. In the "Responses" section, there's an example file responses.json with the following content:

{
  "NotFoundError": {
    "description": "Entity not found",
    "schema": {
      "$ref": "#/definitions/ErrorModel"
    }
  }
}

The reference to #/definitions/ErrorModel seems a bit confusing, since there's no such definition in the example. Is it just that this example is not meant to represent the entire file? Or is the reference incorrect?

3. At the very end, in the "Constraints" section, it says:

Referenced objects must be to JSON structures. YAML reuse structures may be supported in a future version

I assume it means that OAI 2.0 only supports references to JSON files, not to YAML files? If so, why? The format supports YAML for the main file, so why not support it for external files as well? This seems like an arbitrary limitation, especially since YAML is simply an alternate syntax for JSON, not something more advanced that requires special consideration or handling.

[webron]

@BigstickCarpet In Swagger/OpenAPI 2.0 we made an unfortunate mistake by allowing $ref inside the Path Item Object where we allowed $ref to co-exist with other Operation Objects breaking the rules of JSON Reference. However, by the time we learned that, it was too late to change the spec so the rule is this - any Operation Objects defined in the $ref will be included under the Path Item Object. You can add more operation definitions such as get, post, etc. but those cannot override whatever is already defined in the $ref. Defining something that overrides that reference is considered invalid by the spec. I realize this is a parsing nightmare. That's the meaning of that line under "Operations".

[webron]

Sorry, should have replied to everything. For point 2, that's not @jasonh-n-austin's fault, it's mine :) It can definitely be expanded.

As for 3, that's a constraint that may no longer relevant and is related to the official Swagger tools around the spec. Now that the spec has migrated to the OAI and the tools are separate, that constraint should be removed (it may also be irrelevant now, I'd need to check).

[whitlockjc]

I provided my responses to the PR. Long story short, I think we really should avoid adding too much to this. Something like "Swagger reuse is enabled through JSON References." would be a good start. After that, some example of the different types of JSON Reference locations (local, relative, remote, relative or remote with a hash, ...). I also think we should remember that JSON Reference processors will not necessarily be Swagger aware and will gladly resolve references to locations within the Swagger document regardless of whether or not the JSON Reference points to a collection of definitions or not. (Collection of definitions like we Swagger supports for definitions, parameters and responses.)

[jharmn]

@whitlockjc reuse is also enabled with internal references to parameters, definitions etc, not just JSON references. A single schema definition is likely to be referenced multiple times in a spec, even in normal CRUD resource scenarios.

[webron]

Perhaps it would be better to split the REUSE into two docs? One explaining what can be reused, and one giving more info about how to use JSON References within the current spec (first doc pointing to the other)?

[whitlockjc]

@jasonh-n-austin Yep...using JSON References to get to them. ;)

[JamesMessinger]

lol. What @whitlockjc said.

[whitlockjc]

My personal opinion is that it makes no sense to say "We use JSON References but we only allow them to point to certain locations when they are local references." Over the last two years, my work and open source life has been Swagger tooling. I have read every shred of Swagger specification documentation and their JSON Schema files so many times I can't even begin to count. I never once have made the connection that just because these containers were created that it implied JSON References to locations within the same document could only point to those containers. Not only that but never once has anyone brought it up when working on any of the Swagger tooling I've been involved with.

But that is probably a topic for a different issue since I guess the proposed verbiage is fine based on the unwritten rules that I've just become privy to. I just thought you should know.

[jharmn]

I was thinking that the internal references were JSON Pointer, and external was JSON Reference (which uses JSON Pointer for fragments). Maybe I just didn't realize JSON reference would do the local resolution stuff too. Now that I think about it, the $ref part is JSON reference, but it's using JSON Pointer for the local fragment.
#istandcorrected

I'll work on rewording this a bit. @webron are you concerned this is too long already, re: 2 docs?

[whitlockjc]

Just to avoid confusion for anyone reading this later, the JSON Reference spec says that the $ref value is a URI and only the fragment portion, which would mean the whole $ref value for local references, is a JSON Pointer. JSON Reference itself doesn't treat $ref differently whether it's a fragment only (local) or not, it is always just a URI. At evaluation time this information matters but the end user shouldn't care about these things because it's a tooling problem.

To make things simpler, I think mentioning that JSON References are used to enable reuse should be good enough as long as we have good examples. The reason being is that the differences between the different "types" of JSON References used in Swagger documents are just URI semantics. For example, see the three "main" JSON Definition definition types:

• Local references: These are references to somewhere within the document where the reference is defined (Example: {"$ref": "#/definitions/Pet")
• Remote reference (whole document): These are references to a remote document (Example: {"$ref": "http://petstore.swagger.io/v2/swagger.json")
• Remote reference (sub document): These are references to a sub portion of the remote document (Example: {"$ref": "./swagger.json#/definitions/Pet")

Are we over complicating things?

[whitlockjc]

I hope this is reading right. There is no emotion or attitude in any of this. I'm just trying to help.

[JamesMessinger]

Oh the joys of text-based communication. :-/

It reads fine to me. Very informative and clarifying too. Thanks for elaborating!

[whitlockjc]

Any time I communicate via text, I always err on the side of caution just in case. Thanks for the feedback.

[webron]

Okay, as @whitlockjc mentioned we always use JSON References inside the spec. JSON References use JSON Pointers in some cases, as defined in the JSON Reference RFC. I really don't think we need to get into the subtleties of how and when JSON Pointers are used inside JSON References - if anyone wants to know, they should read the RFC. I know @whitlockjc have done so, I've done so, and I'm sure @BigstickCarpet and @jasonh-n-austin have done so as well. Since we use JSON References, we just need to mention where we deviate from it or what additional restrictions we add to it. In most cases, we use it as-is. Overloading the spec or our docs with information that can be easily attained from reading other specs seems wasteful to me.

@jasonh-n-austin - If it's ok with you, please keep it in one doc and concentrate on the content. I'll rework the docs afterwards and submit a PR for the contributors here to review.

@whitlockjc - I assume I annoy people by default with my comments (even though it's not my intent), and am ready to be hit by rotten tomatoes at any given time (virtually and in real life).

[jharmn]

Excellent stuff @whitlockjc. No negative emotion read...remember I work with anti-social developers all day...I read that you care ;)

Totally agree it's all just JSON Reference...my apologies for confusing the matter.
I like the local, remote, remote (sub-document) language, I think I'll relabel the sections in the guidance.
I'll try to steal some time to spit polish this PR this afternoon based on all the additional input.

[jharmn]

@webron PR should be ready for merge. This was a fun one, with all of the collaborative input! Glad to finally have that topic covered.

[webron]

Thanks everyone for the great work on this. The PR has been merged. I'd like to keep the issue open for a bit more, if you don't mind. Please wait for an additional comment to come soon.

[webron]

So following the recent changes, I'd propose splitting the guides into two as shown in #547. Just a proposal, can be scrapped as quickly as it was created. @jasonh-n-austin, @whitlockjc, @BigstickCarpet - thoughts?

[handrews]

@webron I think this can be folded in under #2092 or similar $ref-related things tracked under #2099?

[philsturgeon]

@handrews I've got this tracked under #2099 now and it needs somebody to be in charge. Any volunteers?

[philsturgeon]

@webron @darrelmiller what is the status of this one?

[darrelmiller]

@philsturgeon It is closed