Openapi spec split #447

usmansaleem · 2021-11-19T01:14:31Z

Extract OpenAPI specs from classpath resources to disk
Adjust relative ref paths to absolute paths for OpenApi3Router as it doesn't work well with relative paths
Use relative paths for /swagger-ui by loading contents in memory
Update .openapidoc nodejs project so that it publishes specs with version in directory name instead of filenames.

Merge #444 into this PR.

…ger api integration

-- update swagger-ui resource reading Signed-off-by: Usman Saleem <[email protected]>

…irectory name Signed-off-by: Usman Saleem <[email protected]>

Signed-off-by: Usman Saleem <[email protected]>

joaquim-verges

Thanks for doing this!

joaquim-verges · 2021-11-19T06:36:16Z

core/src/main/java/tech/pegasys/web3signer/core/Runner.java

+            // adjust the ref (which works with OpenApi3Router) to relative that works with
+            // swagger-ui
+            final String adjustedFile =
+                StringUtils.replace(file, "/openapi/eth2/signing/schemas.yaml", "../schemas.yaml");


quite annoying to have to do this twice, both in the gradle task and here :(

Once I add the new API endpoints I'll have another path to replace too on top of this one. Will be /openapi/eth2/keymanager/schemas.yaml.

Would be great to make this replace logic generic so I don't have to touch this code once I add the new endpoints. Maybe with a regex? If starts with / -> replace everything before the last slash with .. or something like that.

Also realized that this flattening of files into one folder means that all filenames must be unique right? Kind of a bummer. Anyway to preserve some of the folder hierarchy and still use relative paths? Like ../signing/schemas.yaml and ../keymanager/schemas.yaml

Would also make the string replace logic simpler that way. Would just be replace /openapi/eth2 with .. and done

@joaquim-verges

I am about to re-do this logic slightly ... apparently the Vertx's OpenApi3Router (which internally uses swagger-parser) is not fool-proof (i.e. fails to parse) when it deals with relative paths in $ref (such as sign.yaml referencing ../schema.yaml#.... where it sits two folder deep. It works if all files are on same level. On the other hand, tools like swagger-ui and redoc (which are javascript based) works fine with such relative paths.

I am adding a bit of logic to modify the relative $ref and discriminator entries to absolute paths for OpenApi3Router handling.

The end file structure that should work would look like:

openapi ├── eth1 │ └── web3signer.yaml ├── eth2 │ ├── signing │ │ ├── paths │ │ │ ├── public_keys.yaml │ │ │ ├── reload.yaml │ │ │ ├── sign.yaml │ │ │ └── upcheck.yaml │ │ └── schemas.yaml │ └── web3signer.yaml ├── filecoin │ └── web3signer.yaml └── index.html

file structure looks good to me! Def had a hard time making relative paths work with Vertx, if there's a robust way to make it work that would be the cleanest for sure.

…d by OpenApi3Router from file system instead of from within the jar Signed-off-by: Usman Saleem <[email protected]>

Signed-off-by: Usman Saleem <[email protected]>

joaquim-verges

Huge progress! Much more robust solution. I'm still baffled that we have to do this relative/absolute paths gymnastic in the first place. Really wish the $ref standard was more standardized across the different OpenAPI tools.

Getting super close though!

joaquim-verges · 2021-11-23T01:51:15Z

acceptance-tests/src/test/java/tech/pegasys/web3signer/tests/SwaggerUIAcceptanceTest.java

@@ -78,7 +78,7 @@ void web3signerYamlEndPointRespondsWith200() {
    given()
        .baseUri(signer.getUrl())
        .when()
-        .get(SWAGGER_UI_ENDPOINT + "/web3signer.yaml")
+        .get(SWAGGER_UI_ENDPOINT + "/eth2/web3signer.yaml")


this is the failing test. Can't see exactly what's wrong with this though, at first glance seems like it should work 🤔

Its Vertx 3.x handling of trailing / in route paths. I've added quick fix and a note in Runner class.

ah I see, nice!

joaquim-verges · 2021-11-23T01:54:54Z

btw @usmansaleem feel free to close my PR and do the API split in yours! Probably simpler to test and merge confidently.

Signed-off-by: Usman Saleem <[email protected]>

usmansaleem · 2021-11-23T03:25:21Z

btw @usmansaleem feel free to close my PR and do the API split in yours! Probably simpler to test and merge confidently.

Thank you, @joaquim-verges , I'll proceed with split openapi specs in this PR 👍🏼

usmansaleem · 2021-11-23T03:33:26Z

Huge progress! Much more robust solution. I'm still baffled that we have to do this relative/absolute paths gymnastic in the first place. Really wish the $ref standard was more standardized across the different OpenAPI tools.

Its the swagger-parser-v3 which has issues with relative paths (specially when involved with multi-level folders), going to create a reproducer and file an issue with them.

joaquim-verges

lgtm! I'll let @jframe take the final look, but this should unblock the rest of the tasks in #439

Thank you @usmansaleem !

core/src/main/java/tech/pegasys/web3signer/core/Runner.java

jframe · 2021-11-23T05:16:44Z

core/src/main/java/tech/pegasys/web3signer/core/Runner.java

+      final Path openApiSpec =
+          openApiSpecsExtractor
+              .getSpecFilePathAtDestination(getOpenApiSpecResource())
+              .orElseThrow();


Be good throw with a nice error message

core/src/main/java/tech/pegasys/web3signer/core/util/OpenApiSpecsExtractor.java

core/src/test/java/tech/pegasys/web3signer/core/util/OpenApiSpecsExtractorTest.java

core/src/main/java/tech/pegasys/web3signer/core/Runner.java

Signed-off-by: Usman Saleem <[email protected]>

joaquim-verges and others added 3 commits November 12, 2021 17:15

Split up OpenAPI spec into multiple files to prepare for new key mana…

cf28f05

…ger api integration

Add reading split openapi spec files

b695030

-- update swagger-ui resource reading Signed-off-by: Usman Saleem <[email protected]>

Update .openapidoc nodejs project to publish specs using version as d…

2bf68fd

…irectory name Signed-off-by: Usman Saleem <[email protected]>

usmansaleem requested a review from jframe November 19, 2021 01:14

usmansaleem added 5 commits November 19, 2021 11:24

code formatting

ebf72d5

Signed-off-by: Usman Saleem <[email protected]>

task name fix

877af8d

Signed-off-by: Usman Saleem <[email protected]>

reverting tuweni version

332f137

Signed-off-by: Usman Saleem <[email protected]>

fix swagger-ui AT

baad816

Signed-off-by: Usman Saleem <[email protected]>

Upgade gatling gradle plugin version

c4f7ab8

Signed-off-by: Usman Saleem <[email protected]>

joaquim-verges reviewed Nov 19, 2021

View reviewed changes

usmansaleem added 4 commits November 20, 2021 14:34

Extract openapi resources to temp directory so that they can be loade…

af664ba

…d by OpenApi3Router from file system instead of from within the jar Signed-off-by: Usman Saleem <[email protected]>

Update .openapidoc nodejs

d2de638

Signed-off-by: Usman Saleem <[email protected]>

Fix core/build.gradle version to update in specs

c2c7b7f

Signed-off-by: Usman Saleem <[email protected]>

attempt to fix relative ref

59af17c

Signed-off-by: Usman Saleem <[email protected]>

usmansaleem marked this pull request as draft November 21, 2021 02:03

usmansaleem added 3 commits November 22, 2021 17:19

openapi resource extractor to handle relative ref

b75bbe7

Signed-off-by: Usman Saleem <[email protected]>

openapidoc readme

ea4044a

Signed-off-by: Usman Saleem <[email protected]>

spotless fix

cc72e09

Signed-off-by: Usman Saleem <[email protected]>

joaquim-verges reviewed Nov 23, 2021

View reviewed changes

Handle trailing /swagger-ui correctly

1422a16

Signed-off-by: Usman Saleem <[email protected]>

usmansaleem marked this pull request as ready for review November 23, 2021 03:24

joaquim-verges approved these changes Nov 23, 2021

View reviewed changes

jframe reviewed Nov 23, 2021

View reviewed changes

usmansaleem added 4 commits November 23, 2021 15:52

fix extracted path to work outside jar

cb50b49

Signed-off-by: Usman Saleem <[email protected]>

change openapi resourcee to openapi-spec to avoid any confusion

c05d592

Signed-off-by: Usman Saleem <[email protected]>

Extract SwaggerUI route

0c17244

Signed-off-by: Usman Saleem <[email protected]>

method name refactoring

4385a92

Signed-off-by: Usman Saleem <[email protected]>

usmansaleem added 2 commits November 23, 2021 17:24

rename method to be more meaningful

33dc7a6

Signed-off-by: Usman Saleem <[email protected]>

additional assertions

fcc9437

Signed-off-by: Usman Saleem <[email protected]>

usmansaleem mentioned this pull request Nov 23, 2021

Split up OpenAPI spec into multiple files to prepare for new key manager api integration #444

Closed

jframe approved these changes Nov 24, 2021

View reviewed changes

usmansaleem mentioned this pull request Nov 24, 2021

[skip ci] Use version in directory name #451

Merged

usmansaleem self-assigned this Nov 24, 2021

usmansaleem added the TeamCerberus Under active development by TeamCerberus @Consensys label Nov 24, 2021

usmansaleem changed the title ~~Openapi spec split preparation~~ Openapi spec split Nov 24, 2021

usmansaleem merged commit 8c5dd34 into Consensys:master Nov 24, 2021

usmansaleem linked an issue Nov 24, 2021 that may be closed by this pull request

Update openapi publish script and gh-pages branch to handle spec file split #446

Closed

usmansaleem mentioned this pull request Nov 24, 2021

Update openapi publish script and gh-pages branch to handle spec file split #446

Closed

joaquim-verges mentioned this pull request Nov 25, 2021

Implementing a standard key manager API #439

Open

26 tasks

usmansaleem deleted the openapi_split_prep branch March 16, 2022 01:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Openapi spec split #447

Openapi spec split #447

usmansaleem commented Nov 19, 2021 •

edited

Loading

joaquim-verges left a comment

joaquim-verges Nov 19, 2021

joaquim-verges Nov 19, 2021

usmansaleem Nov 21, 2021

joaquim-verges Nov 21, 2021

joaquim-verges left a comment

joaquim-verges Nov 23, 2021

usmansaleem Nov 23, 2021

joaquim-verges Nov 23, 2021

joaquim-verges commented Nov 23, 2021

usmansaleem commented Nov 23, 2021

usmansaleem commented Nov 23, 2021 •

edited

Loading

joaquim-verges left a comment

jframe Nov 23, 2021

Openapi spec split #447

Openapi spec split #447

Conversation

usmansaleem commented Nov 19, 2021 • edited Loading

joaquim-verges left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joaquim-verges left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joaquim-verges commented Nov 23, 2021

usmansaleem commented Nov 23, 2021

usmansaleem commented Nov 23, 2021 • edited Loading

joaquim-verges left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

usmansaleem commented Nov 19, 2021 •

edited

Loading

usmansaleem commented Nov 23, 2021 •

edited

Loading