New handler to set mtime embedded in pyc file to 0 #27
Conversation
Co-authored-by: Luke Yang <[email protected]>
This is in preparation for future changes. The code for 'struct Pyc' is put together in one place, separate from the more general code for PycParser. parser.clear_unused_flag_refs() modifies parser in place. A scratch copy of the data is still used, so that modifications can be discarder on failure. Returning a new copy of data would be awkward if we want to do further processing on that data.
Pyc files store the mtime, size, and optionally hash of the original py file. Mtime is used to validate the cached bytecode: the mtime of the .py file (if present) must be equal to the stored value in the .pyc file. If not, the pyc file is considered stale. On ostree systems, all mtimes are set to 0. The problem is that the .py file is created first, then the byte compilation step creates .pyc with some embedded timestamp, and later ostree discards mtimes on all files. On the end system, the bytecode cache is considered invalid. This new handler does the two very simple things: 1. zero out the mtime in the .pyc file 2. set mtime (in the filesystem) to zero for the .py file This second step is also done by ostree, so strictly speaking, it's not necessary. But it's easy to do it, and this way the bytecode still matches the file on disk, so if we called Python after this operation, it would be able to use the bytecode and wouldn't try to rewrite it. Replaces #26. See ostreedev/ostree#1469, https://gitlab.com/fedora/bootc/tracker/-/issues/3, https://pagure.io/workstation-ostree-config/pull-request/505.
Usage: add-determinism --handler pyc-zero-mtime /some/path
|
Added a comment on the other pr, and closed it, wanted to share it here because i know how messages/notifications can get lost. #26 (comment) |
This is something that I haven't considered so far… The API is not designed for external consumption, it currently exposes all kinds of implementation details that are in flux as new functionality is added. I have never implemented a rust library… I guess it could be useful to have a top-level entry point like that'd take
This could be done, but I think it'd make the most sense to do it in tandem with a caller on your side, so that we don't end up with an API that is akward to use. |
|
That looks great as well. Thanks |
|
Once we have that in a release in Fedora, we can give it a try in the Atomic Desktops, replacing the current workaround. |
No description provided.