Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lazy decode messages #17

Merged
merged 11 commits into from
Jun 27, 2022
Merged

lazy decode messages #17

merged 11 commits into from
Jun 27, 2022

Conversation

guplersaxanoid
Copy link
Contributor

No description provided.

@guplersaxanoid guplersaxanoid linked an issue Jun 15, 2022 that may be closed by this pull request
@stwiname
Copy link
Contributor

It would be good to do the same for decoding events and transactions where possible and have some tests that show that the lazy loading works where possible. We would have to use jest spy features.

@@ -169,7 +192,9 @@ function wrapCosmosMsg(
block: block,
msg: {
typeUrl: rawMessage.typeUrl,
...api.decodeMsg<any>(rawMessage),
get decodedMsg() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this changing the CosmosMessage type? Will this require updating other projects?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like that the data were originally accessed through msg property, will have to be accessed through msg.decodedMsg

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could look into a way of transforming it so that the projects or dictionary remains unaffected

Comment on lines 126 to 130
evt.msg.msg = {
...evt.msg.msg,
...evt.msg.msg.decodedMsg,
};
delete evt.msg.msg.decodedMsg;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this hoisting the change from the wrappers? I think this is quite hard to follow.

Would we be better off making a breaking change to get more usable types?

.filter((message) => {
filters.find((filter) => filterMessageData(message, filter));
})
.map((msg) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this need to be done if there are no filters too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized that this function will be only be called for custom datasources. It's a seperate mechanism for runtime datasources:

const handlers = ds.mapping.handlers.filter(

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And it wouldn't be done if there are no filters but, it has to be done.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this transformation doesn't happen for runtime datasources. msg.decodedMsg would still exist.

msg: any;
msg: {
typeUrl: string;
decodedMsg: any;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this generic? CosmosMessage<T = any> and decodedMsg: T

@stwiname
Copy link
Contributor

@DeveloperInProgress I think this is looking good but I think wrapping the data types will cause everything to be loaded.

Problem is here: https://github.com/subquery/subql-cosmos/blob/main/packages/node/src/utils/cosmos.ts#L245-L255

I think we might need to lazy load the events/messages/transactions, that way if there's no event handlers then we don't wrap them.

Can we have a test that calls fetchBlocksBatches please.

@stwiname stwiname merged commit ecb6c78 into main Jun 27, 2022
@guplersaxanoid guplersaxanoid deleted the http-keep-alive branch June 28, 2022 05:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Lazy decoding
2 participants