Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider removing @watchable/queue from nevermore #84

Open
cefn opened this issue Jun 15, 2024 · 1 comment
Open

Consider removing @watchable/queue from nevermore #84

cefn opened this issue Jun 15, 2024 · 1 comment

Comments

@cefn
Copy link
Owner

cefn commented Jun 15, 2024

Currently @watchable/queue is used by nevermore to sequence job settlements. The implementation of @watchable/queue employs immutable arrays (meaning every insertion copies items into a new array). This could lead to a significant performance impact when there is a backlog of e.g. 1000-plus items. It may explain why bulk tests slow down after e.g. 10000 items

A better design would be to employ a FIFO queue employing a linked list with head and tail pointers. This has an O(1) insertion and removal cost regardless of the length of the queue.

@cefn cefn changed the title Consider removing watchable/queue from nevermore Consider removing @watchable/queue from nevermore Jun 15, 2024
@cefn
Copy link
Owner Author

cefn commented Jun 15, 2024

Another related intervention might be to switch @watchable/queue to use FIFO internally and find a different way to introduce immutability (e.g. with a queue subscription that can snapshot the sequence for debugging purposes).

@cefn cefn added the nevermore label Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant