Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/speed up chainfollower #899

Merged
merged 6 commits into from
Nov 6, 2024
Merged

Feat/speed up chainfollower #899

merged 6 commits into from
Nov 6, 2024

Conversation

Kammerlo
Copy link
Member

@Kammerlo Kammerlo commented Nov 4, 2024

Context

The chainfollower took multiple days to sync the assets. This wasn't practically since the time will likely increase in the future. Additionally it was needed to set the starting point of the chainfollower with two environment variables, which was just seen as a work around.
The problem were:

  • The chainfollower got stuck - that's why the custom starting point was needed
  • Saving the assets were very slow

This hindered users to use the background easily without days of syncing.

Proposed Solution

We are saving assets in a local cache and save them batch wise. Additionally I adjusted the insertAssets function to skip already existing assets. Through this solution we are able to remove the check hasAsset for each asset (which takes a few seconds for every asset)
Additionally catching errors from the Chainfollower and continuing the chainfollower.

This solution increases the speed on mainnet to 10% Assets per hour + removes the need of the custom starting point.

@Kammerlo Kammerlo self-assigned this Nov 4, 2024
@Kammerlo Kammerlo merged commit 881896c into master Nov 6, 2024
1 check passed
@Kammerlo Kammerlo deleted the feat/speed-up-chainfollower branch November 6, 2024 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant