Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

read!(::IO, ::ReinterpretArray) performance #51838

Closed
chrstphrbrns opened this issue Oct 24, 2023 · 2 comments
Closed

read!(::IO, ::ReinterpretArray) performance #51838

chrstphrbrns opened this issue Oct 24, 2023 · 2 comments
Labels
io Involving the I/O subsystem: libuv, read, write, etc. performance Must go faster

Comments

@chrstphrbrns
Copy link
Contributor

julia> a=Vector{UInt8}(undef,1_000_000_000);

julia> b=reinterpret(UInt16, a);

julia> f=open("data", "r"); # 1 GB

julia> @time read!(f,a);
  0.307432 seconds (5.56 k allocations: 407.016 KiB, 6.52% compilation time)

julia> f=open("data", "r");

julia> @time read!(f,b);
  8.080147 seconds (15.09 k allocations: 1.029 MiB, 0.33% compilation time)

julia> function Base.read!(s::IO, a::Base.ReinterpretArray{T}) where T
           GC.@preserve a unsafe_read(s, pointer(a), length(a) * sizeof(T))
       end

julia> f=open("data", "r");

julia> @time read!(f,b);
  0.149527 seconds (6.76 k allocations: 456.250 KiB, 5.54% compilation time)

This poor performance is easily explained, since the fallback in read! is a for-loop that reads T-by-T, but it's a surprising limitation

Not sure if pointer(::ReinterpretArray) is legitimate here, but it does seem to work

(views have the same issue)

@N5N3
Copy link
Member

N5N3 commented Oct 24, 2023

We have a stale PR for this issue (#42593).
It only covers "dense" array known at compile time.
We can fuse the code from strides(::ReshapedArray) to extend the acceleration to more StridedArrays

@N5N3 N5N3 added performance Must go faster io Involving the I/O subsystem: libuv, read, write, etc. labels Oct 24, 2023
@N5N3
Copy link
Member

N5N3 commented Oct 25, 2023

close by #42593

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
io Involving the I/O subsystem: libuv, read, write, etc. performance Must go faster
Projects
None yet
Development

No branches or pull requests

2 participants