-
Notifications
You must be signed in to change notification settings - Fork 262
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for an MVP of mmap #304
Comments
It might be a good idea to add an argument for the offset in the file where mapping should start from -- that's supported on POSIX and Win32. The offset will probably need to be page-aligned. This will also allow reading from files that are bigger than 4G. |
As an aside, we do have a simple emulated A potentially tricky issue is what happens to the map if a file is truncated or deleted while mapped. In POSIX, accessing the memory gets a |
Ah that's a good point that this'd need an offset as well, and seems reasonable to provide! Also thanks for pointing out the emulation, I had actually missed that but see the way to get it now ( It may be reasonabley to say that in a situation like where the file is concurrently deleted that it's somewhat unspecified what happens, where it may no be the most portable of behaviors. Engines could presumably catch the |
Yeah Silently filling in zeros is a little risky, because then applications could be fooled into thinking they read the actual file contents when they didn't. For example, a serialized NUL-terminated string could be truncated if the file is deleted while the application is in the middle of reading it, and a naive application wouldn't suspect anything was wrong. I don't yet have an opinion about whether this is a show-stopper; I'm just bringing it up to be considered. |
The I/O can also fail. i.e. if the network goes down. |
In theory, yes, an |
Just because people might find it interesting, I had a requirement to map files read-write to guest linear memory, for my Go bindings to SQLite using wazero. This was achieved, and is working on Linux/macOS by using a custom linear memory allocator, then doing an |
Is there a solution for Windows and C++? |
I didn't see a previously filed issue about this so I wanted to make sure that one was filed here. In the applications I've seen
mmap
(or the equivalent for Windows) is a pretty common operation and would be nice to support in WASI. I believe, though, thatmmap
covers quite a large range of functionality so this initially is a very large feature request, so I'd like to winnow it down a bit more to see if others feel like it might be possible to add to WASI.I think a pretty useful (but still minimal) implementation of
mmap
might be to allow to map files into memory as read-write, but not allowing changes to get persisted back to disk. This encompasses a pretty common usage ofmmap
, used to read files (e.g. text searching, parsing, etc), where the contents need to be readable but not writable.One important thing to consider with
mmap
I think as well is the compatibility or ability to implement this on the web. The web (and wasm in general) don't have a way of making a region of memory read-only, for example, so I don't think it's possible to expose the full power ofmmap
.A possible proposal for this would be:
This would mmap a file descriptor into the process address space at the (required)
$map_base
and$map_len
parameters. This is different from themmap
syscall found on Unix in a number of ways:memory.grow
, not through this API.The goal here is that engines (at least out-of-browser ones) could implement this with OS-level
mmap
calls. With proper alignment requirements on$map_base
and$map_len
this could translate to a host-levelmmap
to actually map the contents of a file into a wasm address space natively. This should bring all the expected performance wins ofmmap
to a wasm guest as well.I'm curious about what others think of this strawman, or if others know where other discussions have happened where this could be linked to as well.
The text was updated successfully, but these errors were encountered: