Remove hot uses of pathlib #14110
Remove hot uses of pathlib #14110
Conversation
bonzini
force-pushed
the
speedup-pathlib
branch
2 times, most recently
from
0b2ef7f to
bf03ed2
Compare
Isn't this just os.path.commonpath? This is a well-known problem with pathlib -- it loves to reimplement os.path via complicated and fiddly data structures with questionable correctness characteristics. And apparently questionable performance characteristics too... |
|
It's similar to
Yeah, I like |
Faster, maybe. I haven't timed it. But with regard to returning a boolean etc. the point of commonpath isn't "this function returns whether a path is an ancestor" but rather that it's the trivial building block for such a check. It's code we already use in several places: if os.path.commonpath(x, y) == x:
do_thing_with(y) |
|
Yes, it is faster: :) Also it doesn't crash and burn if you have different drives on Windows. :) Should I adjust other uses of |
Yup, we catch ValueError for those. So I suppose it may anyways be worth extracting into a dedicated function. :P Should I adjust other uses of Yes, please. If we need this function for performance in a hot path anyway, then we might as well use it everywhere the same code pattern is used. |
bonzini
force-pushed
the
speedup-pathlib
branch
2 times, most recently
from
d1eaebd to
7d781c8
Compare
| if InterpreterRuleRelaxation.ALLOW_BUILD_DIR_FILE_REFERENCES in self.relaxations and path_starts_with(norm, self.environment.build_dir): | ||
| return | ||
| if srcdir not in norm.parents: | ||
|
|
||
| if not path_starts_with(norm, self.environment.source_dir): |
There was a problem hiding this comment.
Commit message for this is no longer accurate. :D
"Introduce a new function" -> use our recently added function?
Introduce an alternative to os.path.commonpath(name, path) == path, which is a common idiom throughout Meson. Call it is_parent_path just like the existing static method in Generator. It is a bit faster and handles drives on Windows without the need for an exception handler. Signed-off-by: Paolo Bonzini <[email protected]>
This saves about 1% of execution time; path manipulation is expensive. About 1% more could be saved by avoiding usage of "foo in bar.parents", which is an expensive way to say "bar.startswith(foo)". Signed-off-by: Paolo Bonzini <[email protected]>
pathlib's implementation of Path iteration is expensive;
"foo.parents" has quadratic complexity when building it:
return self._path._from_parsed_parts(self._drv, self._root,
self._tail[:-idx - 1])
and comparisons performed by "path in file.parents" potentially
have the same issue. Introduce a new function that checks whether
a file is under a path in the same way, removing usage of Path
from the biggest hotspot.
Signed-off-by: Paolo Bonzini <[email protected]>
bonzini
force-pushed
the
speedup-pathlib
branch
from
7d781c8 to
8ace12e
Compare
pathlib's implementation of Path iteration is expensive; "foo.parents" has quadratic complexity when building it:
and comparisons performed by "path in file.parents" potentially have the same issue. Introduce a new function that checks whether a file is under a path in the same way, removing usage of Path from the biggest hotspot.