Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emitting &[u8] literal is very slow #2

Closed
SOF3 opened this issue Aug 14, 2019 · 1 comment
Closed

Emitting &[u8] literal is very slow #2

SOF3 opened this issue Aug 14, 2019 · 1 comment
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@SOF3
Copy link
Owner

SOF3 commented Aug 14, 2019

Issue

rustc takes forever to parse the generated &[u8] literal. I have checked that it is indeed compiled into a raw byte array (no delimiters, no boxing, etc.) in the output executable (thus satisfying the size requirements), but rustc doesn't seem to know to parse the contents in a faster way.

Alternatives

An alternative would be to write the deflated stream into a temp file and use include_bytes!(), but I don't know where the file should be created.

  • If I write it to the OS temp directory, it would result in an OS-level resource leak before rebooting (because nobody is deleting the temp files).
  • If I always write to the same file in the target directory, concurrent compilations might overwrite each other.
  • If I write to somewhere next to the included file, it corrupts the user filesystem. Rust does not recommend build scripts to affect files beyond the OUT_DIR. However, for macros, we don't even have OUT_DIR or anything within the target/ directory.

A solution (still bad but affects less negatively) is to create a hash based on the full file name in the operating system, and save the file content hash + compressed contents using a file within target uniquely identified by the hash. A unique file lock can be acquired while writing. If the file content hash is already identical to the compressed contents, this means (although unlikely) that another compilation has already compressed the file, and there is no need to recompile, preventing reducing the chance of race condition between rustc's handling include_bytes!() and another process's macro compilation.
But where within the target directory? I am not sure.

@SOF3 SOF3 added bug Something isn't working help wanted Extra attention is needed labels Aug 14, 2019
@SOF3
Copy link
Owner Author

SOF3 commented Aug 14, 2019

include_bytes!() appears to use a bytestring rather than a &[u8]. Benchmarks can be tried again with byte strings.

@SOF3 SOF3 closed this as completed in 535b763 Aug 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant