Relative JSON-pointer extension

[epoberezkin]

The proposal is to extend relative JSON pointer (https://tools.ietf.org/html/draft-luff-relative-json-pointer-00) to be able to refer to the items in array with some relative position.

Originally filed as: json-schema/json-schema#223

The syntax extension can be really simple, from the current N/JSONpointer to N+M/JSONPointer (to traverse N levels up and M items forward) or N-M/JSONPointer (M items backward), where N == 0 can possibly be dropped.

That would allow, for example, to implement restrictions on the array such as ascending/descending order (using $data v5 proposal).

Ascending order without duplicates:

{
  "type": "array",
  "items": {
    "type": "integer",
    "minimum": { "$data": "-1/" },
    "exclusiveMinimum": true
  }
}

Objects ordered in ascending order by unique id property:

{
  "type": "array",
  "items": {
    "properties": {
      "id": {
        "type": "integer",
        "minimum": { "$data": "1-1/id" },
        "exclusiveMinimum": true
      }
    }
  }
}

[handrews]

Should we first have an issue somewhere to revive the Relative JSON Pointer I-D? Is that part of this project or does it need to be revived separately?

[Anthropic]

@handrews I think if we can agree on what to modify then it would need a revival issue...

@epoberezkin can I suggest that we consider bracketing like N{-M}/ and N{+M}/ to allow for the same syntax at depth. I'm dealing with a UI at the moment so complex it has three arrays depth and many depths of object intertwined. So if I am working with the UI-Schema and not just the JSON-Schema then I would need to be able to define multiple array positions at depth not just in the first position.

The baz example where "0#" gets the current array position doesn't suffice when I need to define something more inline. If I have team[3].player[4].stats.game[4].plays and I want to get to team[3].player[5].stats.game[4].plays from plays, I would need "4{#+1}/stats/game/{#}/plays" to indicate back to the next player for the same details if we keep hash as meaning the current index.

[handrews]

For reviving the RFC I-D at all, see #126 (we need it for various things whether we want to change it or not).
[EDIT: It has since been revived and two further drafts published with a third underway.]

[handrews]

I have a few concerns about this proposal.

• Only allowing relative array indices on the first token of the JSON pointer is limiting and confusing.
• Allowing relative array indices anywhere requires some sort of escaping as any string can be a JSON property name.

[EDIT: The following definitely no longer true, and the point was dubious and contested in the first place]

• Adding more escaping will make Relative JSON Pointer incompatible with JSON Pointer (currently, since relative JSON pointers are proposed as an extension, all regular JSON pointers are also legal relative JSON pointers with the same meaning).

[epoberezkin]

@Anthropic I think relying on the sequential number being the same in different arrays is an uncommon (and also fragile) use case, usually IDs are used to cross-reference the objects.

Only allowing relative array indices on the first token of the JSON pointer is limiting and confusing.

@handrews that's the most commonly needed use case that allows to implement several common constraints (such as ascending/descending, other sequences)

all regular JSON pointers are also legal relative JSON pointers with the same meaning

@handrews that is incorrect - regular pointers must start with / while relative start with a number, regular are from root, relative are from the current point.

While we can consider an even more radical extension to something like JSON-pointer expression (that would need to cover some arithmetics maybe?) a simple extension allowing to navigate to sibling items from the current item would have been very useful (even if not covering everything).

So essentially, I believe that "up" -> "left/right" -> "JSON pointer" is much more powerful than "up" -> "JSON pointer" and still doesn't require any additional escaping characters or other dramatic changes.

[epoberezkin]

It really only makes sense to move laterally when the array is at the "top" of your path, right?

For the common data designs, yes. As I wrote it is uncommon (and fragile) design when items in different trees are correlated by the index rather than by some id.

[handrews]

@epoberezkin as I have resolved the grammatical argument [EDIT: meaning the argument over my interpretation of "extension"] in your favor with #446 (assuming you agree), once that is merged I plan to delete the comments involved in that argument so that future readers can focus on the actual point of this issue. There are plenty of examples of the handrews/epoberezkin argument show elsewhere ;-P

[EDIT: GitHub makes it look like I just deleted epoberezkin's comments, for some reason it does not show that I deleted my own comments as well]

[handrews]

I am fairly certain this is the only substantial Relative JSON Pointer issue in the repo, so I'd like to figure out what to do with it (there are some trivial wording bugs and the like filed elsewhere, which don't need discussion).

Reading back through this, I am open to supporting the original proposal, keeping in mind that even though the current JSON Schema Core and Validation do not require such things, Relative JSON Pointer is being produced as its own specification and should not be artificially limited by JSON Schema's current feature set. Particularly now that we have modular vocabularies.

I'm not entirely sold on dropping the leading 0 in pointers such as 0+1/foo being writable as +1/foo, but I'm not dead-set against it either.

Regarding @Anthropic 's further proposal, I am less convinced. If there is consensus on the basic proposal and not on the further proposal, I would prefer to split @Anthropic 's part out for separate discussion. Returning to this after (wow) over two years, the original proposal immediately made sense to me, while the more complex proposal is making my head hurt a bit even though I get the use case. I would want to kick it around more, at minimum.

Thoughts? Is any of this still of interest at all? I'm not pushing this, I'm just saying I'm open to it and I'd like to resolve it.

[gregsdennis]

I'm not sure I follow exactly how this is supposed to work. The only examples are in the original post and only show an example of usage but not actually what it does. I'd appreciate a breakdown of the syntax along with a step-by-step description of how it works.

[handrews]

@gregsdennis I'd also forgotten that the original proposal referenced $data, and @awwright and I closed that proposal finally yesterday as it's clear at this point that it's not where we want to go. 😁

I will use the not-seriously-named keywords mentioned in #855 e.g. dataMinimum where the dataFoo keyword works just like foo except that the value is determined by a relative JSON pointer instead of a literal.

I'm also going to always use 0+1 and 0-1 instead of the proposed shorthand of +1 or -1 when the leading number would otherwise be zero, because I think having +1 and 1 mean different things would be very confusing. Hmm.. actually a number of things need to be tweaked.
With that in mind:

{
  "type": "array",
  "items": [
    {
      "type": "integer"
    }
  ],
  "additionalItems": {
    "dataExclusiveMinimum": "0-1",
  }
}

Given this instance:
[0, 1, 4, 2]

The schema in the tuple items form applies to 0 and passes
additionalItems is applied to the remaining items, as follows:

For each additional item:

• the 0 at the beginning of 0-1 means "start from the current instance location"
• the -1 that follows it assumes that the current location is an element of an array (an immediate child element), and tells us to look at the previous element

So...

• applied to the value 1 (at index 1) we look back to 0 (at index 0) which is strictly smaller, so that passes
• applied to the value 4 (at index 2) we look back to 1 (at index 1) which is strictly smaller, so that passes
• applied to the value 2 (at index 3) we look back to 4 (at index 2) which is larger, so that fails.

The other example works similarly:

{
  "type": "array",
  "items": [
    {
      "type": "object",
      "properties": {
        "id": {
          "type": "integer"
        }
      }
    }
  ],
  "additionalItems": {
    "type": "object",
    "properties": {
      "id": {
        "type": "integer",
        "dataExclusiveMinimum": "1-1/id"
      }
    }
  }
}

given the instance:

[{"id": 0}, {"id": 1}, {"id": 4}, {"id": 2}]

this works the same, except instead of the current instance location being the whole value of the array element, the current instance location is the value of the "id" property in the array element.

So we have to go up a level (that first 1), and then back an element, and then down the "id" property. The numeric comparison works the same.

Does that make sense?

[handrews]

If we wanted to get rid of having to special-case the first element, keywords could be defined to always pass if the pointer goes over the end of the array. Which might be necessary as you can't use tuple items to hit the end of the array, so going in the opposite direction won't work with the tuple items technique.

[gregsdennis]

Thank you for the explanation. The syntax now makes sense.

So we have to go up a level

This adds an entirely new level of complexity that my (and I would guess many others) validator isn't equipped to handle. Up to this point, traversing the instance and schema has been strictly "down" the tree, meaning there was no need for parent references (I have none in my JSON model).

That said...

There may be a way around this: relative pointer transforms. Given a discrete pointer (the normal kind) and a relative pointer, I might be able to "add" them in a way that produces a new discrete pointer. That may save me from having to have parent references.

Additionally (no pun intended) we could include this "add" operation in the (Relative) JSON Pointer spec. Examples would be given, a test suite could be created... It'd be fun!

[handrews]

This adds an entirely new level of complexity that my (and I would guess many others) validator isn't equipped to handle. Up to this point, traversing the instance and schema has been strictly "down" the tree, meaning there was no need for parent references (I have none in my JSON model).

I was writing up the schema loading/processing guidelines, specifically a section on what keywords can do, and ran into this. It's one of the reasons I haven't put up a PR for it yet.

Tentatively, I was going with the idea that evaluating a keyword might require access to the whole instance, plus the pointer for the current location. Then if a keyword wants to do things like adjust the location with a relative pointer, they can "add" the relative pointer to the current location pointer, and then apply that to the instance.

Which I think matches what your saying with transforms? If so that would be a good sign if we're looking at the same sort of solution.

BTW, the links keyword in Hyper-Schema already does this sort of stuff with relative pointers, which is why I was trying to sort out what that meant for a generic implementation.

[gregsdennis]

Yeah, I think "adding" a relative pointer to a discrete pointer to yield a new discrete pointer is the way to go. We are in agreement. I was just thinking out loud for the other bit.

[Relequestual]

I like the concept, but this goes back to referencing data in the instance, which I thought we were (at least currently) trying to avoid (at least for non-hyperschema vocabs)?

If @handrews you're looking at this in terms of "We accept a future vocabulary MAY wish to reference instance data by relative JSON pointer" and making sure that's possible in core, then fine.

BUT, I would say THAT is a different issue than the one than where this comment resides.

For clarity, I'm not opposed to having a new vocabulary to allow referencing of instance data for validation, just as far as I understand today, we're not looking into it right now. So my question is, does THIS issue belong in the draft-08-patch-1 milestone, or should it be removed from the milestone?

[handrews]

@Relequestual the whole point of #855 , which you commented that you were OK with, is to spec out what extension keywords might do with data access, even though we are not adding any specific keywords to do that. That is a good time to set the scope of what Relative JSON Pointer can do.

[Relequestual]

@handrews that was only 8 days ago... Sorry. ugh -_-

OK. Yup, sounds reasonable.

Am I right in thinking that the originally mentioned RFC has been updated by you @handrews ? and as such we COULD extend that RFC also? (I don't know if that's worth doing right now vs focusing on getting the draft 8 patch 1 milestone out the door)

[handrews]

Relative JSON Pointer is our draft, we usually release it with the others. We've only done really minor clarifications up to this point but we could do more.