Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

XIM Reader speedup #546

Open
purepani opened this issue Feb 17, 2025 · 0 comments
Open

XIM Reader speedup #546

purepani opened this issue Feb 17, 2025 · 0 comments

Comments

@purepani
Copy link

Is your feature request related to a problem? Please describe.
Currently the speed of the XIM parser is quite slow due to being mostly written in pure python, and also partially being written with some missing optimizations(I.e. converting numbers to strings).

Describe the solution you'd like
I've began writing an XIM parser in rust: https://github.com/purepani/xim_reader. This is able to be imported as a python package through pyo3. It isn't fully finished(it only supports i16 at the moment, but I've gotten it to work well enough to run some tests, and it reduces the read time from ~1.1s to 7.5 ms:
Image

If this is something you'd like to add, I think it'd be best to vendor it, especially since you recently switched to uv with some nice workspace features. I'd be willing to finish it up and vendor it if it's something you'd like to add.

Describe alternatives you've considered
I also tried speeding up the parsing by just improving the python and trying to do more array operations. This, however, was much harder. I got some speed up with using packbits, but to improve the main decompression algorithm, I'd have to be a lot more clever with array operations, whereas the rust implementation is basically a mirror of the python version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant