Splitting spec files still throws schema and file not found errors (json_schema exceptions) but still works with Connexion <= 3.0.2

[eharvey71]

I have my yml files divided into subdirectories for paths, parameters, schemas, etc. This worked fine (and still works fine) on Connexion 3.0.4 3.0.2.
Connexion 3.0.5 and 3.0.6 3.0.3 - 3.0.6 are still throwing errors. The prior fixes for $refs don't seem to have fixed issues. I thought this was just a Windows filesystem or jsonschema issue but it's happening all over - in my MacOS, Windows and docker container deployments running Debian.

---

If I downgrade to Connexion 3.0.4 3.0.2 or prior, the error goes away, in all cases.

---

I am working with a separate swagger UI deployment, so I can do customizations.

In my configuration, I'm initializing my app like this:

basedir = pathlib.Path(__file__).parent.resolve()
swagoptions = SwaggerUIOptions(swagger_ui = True, swagger_ui_template_dir = basedir / 'swagger-ui')
connex_app = FlaskApp(__name__, specification_dir=basedir / "apispecs")

Paths are like this to specs, where my swagger.yml contains $refs to the proper paths of each of the divided spec files:

project/
      │── apispecs/
               |── swagger.yml
               |── parameters/
                         |── _index.html
               │── securitySchemas/
                         |── _index.html
               │── schemas/
                         |── _index.html

swagger.yml sample:

components:
  schemas:
    $ref: "./schemas/_index.yml"
  parameters:
    $ref: "./parameters/_index.yml"
  securitySchemes:
    $ref: "./security/_index.yml"

Full contents of error:

_RefResolutionError(_cause=FileNotFoundError(2, 'No such file or directory'))
Traceback (most recent call last):
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/connexion/json_schema.py", line 88, in _do_resolve
    retrieved = deep_get(spec, path)
                ^^^^^^^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/connexion/utils.py", line 112, in deep_get
    return deep_get(obj[keys[0]], keys[1:])
                    ~~~^^^^^^^^^
KeyError: 'schemas'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/jsonschema/validators.py", line 1102, in resolve_from_url
    document = self.store[url]
               ~~~~~~~~~~^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/jsonschema/_utils.py", line 20, in __getitem__
    return self.store[self.normalize(uri)]
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
KeyError: './schemas/_index.yml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/jsonschema/validators.py", line 1105, in resolve_from_url
    document = self.resolve_remote(url)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/jsonschema/validators.py", line 1202, in resolve_remote
    result = self.handlers[scheme](uri)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/connexion/json_schema.py", line 41, in __call__
    with open(filepath) as fh:
         ^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/schemas/_index.yml'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/connexion/middleware/swagger_ui.py", line 110, in _get_openapi_json
    content=jsonifier.dumps(self._spec_for_prefix(request)),
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/connexion/middleware/swagger_ui.py", line 73, in _spec_for_prefix
    return self.specification.with_base_path(base_path).raw
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/connexion/spec.py", line 209, in with_base_path
    new_spec = self.clone()
               ^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/connexion/spec.py", line 199, in clone
    return type(self)(copy.deepcopy(self._raw_spec))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/connexion/spec.py", line 83, in __init__
    self._spec = resolve_refs(raw_spec, base_uri=base_uri)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/connexion/json_schema.py", line 106, in resolve_refs
    res = _do_resolve(spec)
          ^^^^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/connexion/json_schema.py", line 100, in _do_resolve
    node[k] = _do_resolve(v)
              ^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/connexion/json_schema.py", line 100, in _do_resolve
    node[k] = _do_resolve(v)
              ^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/connexion/json_schema.py", line 96, in _do_resolve
    with resolver.resolving(node["$ref"]) as resolved:
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/jsonschema/validators.py", line 1044, in resolving
    url, resolved = self.resolve(ref)
                    ^^^^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/jsonschema/validators.py", line 1091, in resolve
    return url, self._remote_cache(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eharvey71/dev/integration-bridge/venv/lib/python3.12/site-packages/jsonschema/validators.py", line 1107, in resolve_from_url
    raise exceptions._RefResolutionError(exc) from exc
jsonschema.exceptions._RefResolutionError: [Errno 2] No such file or directory: '/schemas/_index.yml'
INFO:     127.0.0.1:55767 - "GET /api/openapi.json HTTP/1.1" 500 Internal Server Error

Output of the commands:

• Python is 3.12

[eharvey71]

Actually, when I test with 3.04 or 3.03, I have a different issue with specs being invalid:

Failed validating 'oneOf' in schema['properties']['paths']['patternProperties']['^\\/']['patternProperties']['^(get|put|post|delete|options|head|patch|trace)$']['properties']['requestBody']:
    {'oneOf': [{'$ref': '#/definitions/RequestBody'},
               {'$ref': '#/definitions/Reference'}]}

3.0.2 seems to be the most stable for my needs and works perfectly, but it would be great to run the latest and greatest.

[eharvey71]

3.1 has fixed my issue. Thank you!

[dmariaa]

@eharvey71 How are you splitting your files? I have a similar setup to the one in the beginning of this post and I still find errors when version is higher than 3.0.2 including 3.1... thanks!!!

[eharvey71]

I think I may have closed this prematurely, thought it was working, gave up, and went with the single flat file spec.

I WAS using this: https://github.com/eharvey71/video-platform-integration-bridge/blob/main/apispecs/swagger-refs-version.yml ... and it seemed to be working well until the underlying jsonschema issues arose once again. This is starting to get unworkable in the case of scaling out specs and I may have to look at something else. This was perfect but my app has to grow to support more abstractions.

I love Connexion, I'm even a sponsor, but so much for "spec first" ¯_(ツ)_/¯
.. it's not a priority: #1897

[RobbeSneyders]

#1897 is not related to this issue. It is about a clearer error message, but since this error is developer-facing and not user-facing, I indeed do not consider it a priority. As mentioned, I would welcome a PR from someone who does think it is a priority.

Can you confirm that this issue is related to the /openapi.json endpoint and the swagger UI? Those indeed break when splitting specs, however based on my testing, any paths defined in the spec work as expected.