Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support positional vectored IO for unix files #71055

Open
vimpunk opened this issue Apr 12, 2020 · 5 comments
Open

Support positional vectored IO for unix files #71055

vimpunk opened this issue Apr 12, 2020 · 5 comments
Labels
C-feature-request Category: A feature request, i.e: not implemented / a PR. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.

Comments

@vimpunk
Copy link

vimpunk commented Apr 12, 2020

Next to the existing write_vectored and read_vectored APIs, it would be great to have write_vectored_at and read_vectored_at methods for Files on UNIX systems, corresponding to the pwritev and preadv syscalls. This would be an easy way to add means for high performance file IO using the standard library File type.

As far as I understand, the equivalent does not exist on Windows, so this would probably have to live in the unix FileExt trait.

If this is deemed desirable, I'd be happy to send a PR for this.

@jonas-schievink jonas-schievink added C-feature-request Category: A feature request, i.e: not implemented / a PR. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. labels Apr 12, 2020
@retep998
Copy link
Member

Windows actually does not support vectored reads and writes for anything other than sockets, so write_vectored and read_vectored for files just defaults to doing a single read/write of the first buffer. Windows does support reading and writing at a specific position, so I see no reason why read_vectored_at and write_vectored_at cannot be implemented in a similar fashion, where it does a simple read_at or write_at of the first buffer.

@vimpunk
Copy link
Author

vimpunk commented Apr 15, 2020

@retep998 Thanks for the clarification. In that case this could indeed be implemented for Windows as well.

I suppose the next step would be an RFC?

@alecmocatta
Copy link
Contributor

Positional reading/writing on Windows has the unfortunate effect of modifying the file offset cursor. IIRC the positioned-io crate has either of two workarounds: storing the cursor explicitly, or using a temporary memory mapping to read data rather than FileExt::seek_read.

@superobertking
Copy link

Citing from Linux man page, preadv and pwritev are "nonstandard, but present also on the modern BSDs". In fact, not all unix support preadv and pwritev - macOS does not yet.

Further, "preadv and pwritev first appeared in Linux 2.6.30 and library support was added in glibc 2.10", while the Rust platform support requires the minimal Linux version of 2.6.18. This would break backward compatibility support.

For non-POSIX standard API, it's better to reside in an external crate rather than in the std as suggested (in somewhere else I cannot find now). Libc already provides this syscall function, and nix provides a pretty wrapper function around it. I guess we could have a FileExt-like trait in some crate and implement it for the std File.

@artob
Copy link

artob commented Oct 10, 2023

This was implemented in #89518, feature gated by #![feature(unix_file_vectored_at)] (#89517).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-feature-request Category: A feature request, i.e: not implemented / a PR. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

6 participants