You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Currently the speed of the XIM parser is quite slow due to being mostly written in pure python, and also partially being written with some missing optimizations(I.e. converting numbers to strings).
Describe the solution you'd like
I've began writing an XIM parser in rust: https://github.com/purepani/xim_reader. This is able to be imported as a python package through pyo3. It isn't fully finished(it only supports i16 at the moment, but I've gotten it to work well enough to run some tests, and it reduces the read time from ~1.1s to 7.5 ms:
If this is something you'd like to add, I think it'd be best to vendor it, especially since you recently switched to uv with some nice workspace features. I'd be willing to finish it up and vendor it if it's something you'd like to add.
Describe alternatives you've considered
I also tried speeding up the parsing by just improving the python and trying to do more array operations. This, however, was much harder. I got some speed up with using packbits, but to improve the main decompression algorithm, I'd have to be a lot more clever with array operations, whereas the rust implementation is basically a mirror of the python version.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Currently the speed of the XIM parser is quite slow due to being mostly written in pure python, and also partially being written with some missing optimizations(I.e. converting numbers to strings).
Describe the solution you'd like

I've began writing an XIM parser in rust: https://github.com/purepani/xim_reader. This is able to be imported as a python package through
pyo3
. It isn't fully finished(it only supportsi16
at the moment, but I've gotten it to work well enough to run some tests, and it reduces the read time from ~1.1s to 7.5 ms:If this is something you'd like to add, I think it'd be best to vendor it, especially since you recently switched to
uv
with some nice workspace features. I'd be willing to finish it up and vendor it if it's something you'd like to add.Describe alternatives you've considered
I also tried speeding up the parsing by just improving the python and trying to do more array operations. This, however, was much harder. I got some speed up with using packbits, but to improve the main decompression algorithm, I'd have to be a lot more clever with array operations, whereas the rust implementation is basically a mirror of the python version.
The text was updated successfully, but these errors were encountered: