-
Notifications
You must be signed in to change notification settings - Fork 360
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Read windows path error - 2 #557
Conversation
The kubernetes test does not have to pass as you would not run kubernetes on windows. They do some symlink magic there to update files atomically, that's why we have that special case. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would actually be very nice if you could add a CIrcleCI job or GitHub workflow to run the tests on windows. I think it would be enough to run go test ./...
. If you have no experience with that, let me know.
I don't have any CIrcleCI or GitHub Workflow experience at all, but I can give it a try. |
Ok, So I was not successful in doing a Windows CircleCI test, not sure how to do it. Sorry. |
The error output is:
|
Ok, I should have been more clear, I can see the logs but I am not sure what part of the output is the error that fails the test.
All the relevant test seems to pass when I run them locally, but this log doesn't really run those tests at all. Maybe I need to run some other test script locally to find out where the issue is? As I am not so familiar with your build/test system some help here would be appreciated. Edit: Edit2: |
…ndard variants of file URLs
Please note that I have made some small change to rule/rule_test.go that corrected an issue that made the tests (or was it linting?) fail. I am not sure this change is allowed or should be in here. Should I remove it? |
ci: windows unit test
The PatseURL function will handle parsing URLs with special handling of file:// URLs The updated GetURLFilePath function will return a correct path baed on the updated URL parsing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you re-request a review when your done? Thx 😉
x/fileurl.go
Outdated
|
||
// ParseURL parses rawurl into a URL structure with special handling for file:// URLs | ||
func ParseURL(rawurl string) (*url.URL, error) { | ||
lcRawurl := strings.ToLower(rawurl) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rename
lcRawurl := strings.ToLower(rawurl) | |
lcRawURL := strings.ToLower(rawurl) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't it be less error prone to do
lcRawurl := strings.ToLower(rawurl) | |
if runtime.GOOS == "windows" { | |
rawURL = strings.ToLower(rawurl) | |
} |
and avoid the decision what variable to use for what check?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you are lower casing rawURL
there because windows filepaths are case insensitive right? But they are case sensitive on POSIX
That was what I was referring to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure exactly what you mean here, the lcRawURL is only used to compare the initial part if the rawURL string so see if it starts with file:// (or any upper/lower case version of it).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah ok, could you then maybe use strings.Split(rawURL, "://")
to get the scheme and use that for the checks?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At this point in the code I am first looking for file:/// (three slashes), if that is the case I am parsing it as a normal URL with an early exit. If that is not the case and it instead starts with file:// (two slashes) we trim the first 7 chars off the beginning and then continues the function. The lower case comparison has nothing to do with Windows, it just makes sure that FILE:// and file:// are treated the same.
I can change it to a regexp case insensitive comparison if that feels better.
x/fileurl.go
Outdated
if len(parts) > 2 { | ||
host = parts[2] | ||
} | ||
p := "/" | ||
if len(parts) > 4 { | ||
p += strings.Join(parts[3:], "/") | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This took me quite some time to understand this part. Can you maybe rewrite it a bit? Or add comments? It is especially not clear why parts[3]
is discarded when len(parts) == 4
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have reworked this part now
x/fileurl.go
Outdated
return nil, err | ||
} | ||
if u.Scheme == "file" || u.Scheme == "" { | ||
u.Path = toSlash(u.Path) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here also, on POSIX \
could be part of the file name and the conversion is not necessary at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good call! I didn't think of this. I will make some tests and try to fix it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I have fixed this issue now
x/fileurl.go
Outdated
} | ||
|
||
func stripFistPathSeparators(fPath string) string { | ||
for len(fPath) > 0 && (fPath[0] == '/' || fPath[0] == '\\') { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here also, on POSIX \
could be part of the file name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will fix
x/fileurl_test.go
Outdated
{"file://file7.txt", "file7.txt", "file7.txt"}, | ||
{"file://path/file8.txt", "path/file8.txt", "path/file8.txt"}, | ||
{"file://C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\9ccf9f68-121c-451a-8a73-2aa360925b5a386398343/access-rules.json", "/C:/Users/RUNNER~1/AppData/Local/Temp/9ccf9f68-121c-451a-8a73-2aa360925b5a386398343/access-rules.json", "file:///C:/Users/RUNNER~1/AppData/Local/Temp/9ccf9f68-121c-451a-8a73-2aa360925b5a386398343/access-rules.json"}, | ||
{"file:///C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\9ccf9f68-121c-451a-8a73-2aa360925b5a386398343/access-rules.json", "/C:/Users/RUNNER~1/AppData/Local/Temp/9ccf9f68-121c-451a-8a73-2aa360925b5a386398343/access-rules.json", "file:///C:/Users/RUNNER~1/AppData/Local/Temp/9ccf9f68-121c-451a-8a73-2aa360925b5a386398343/access-rules.json"}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not quite sure this is what we want. Is this following some spec or standard?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, this can be discussed of course, but I have determined that this is a reasonable solution since this style of non standard file URLs are used throughout Oathkeeper. And since the file scheme doesn't support relative paths the solution is to use an empty scheme. This is also documented in the standard library documentation: https://golang.org/pkg/net/url/#Parse
I have tried to combine the standard and also the way many people are probably using file URLs already without having any regressions.
x/fileurl.go
Outdated
return "" | ||
} | ||
|
||
if u.Scheme != "file" && u.Scheme != "" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's invert this condition to make it more readable
if u.Scheme != "file" && u.Scheme != "" { | |
if !(u.Scheme == "file" || u.Scheme == "") { |
Edit: I forgot the not 🤦♂️
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi Mr. De Morgan 👋
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is fixed
strings.Index() == 0 -> strings.HasPrefix() Co-authored-by: Patrik <[email protected]>
Namingconvention Co-authored-by: Patrik <[email protected]>
Also fixed a few issues with \ on POSIX where it should not be interpreted as a path separator.
So I have made some changes after your suggestions now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, this looks good now 🎉
Let's merge it in ory/x
x/fileurl.go
Outdated
return url.Parse(rawURL) | ||
} | ||
|
||
if strings.HasPrefix(lcRawURL, "file://") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just realized you can avoid the if
statement and use strings.TrimPrefix(lcRawURL, "file://")
, sry 😅
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I has the same thought, but it turns out that we are comparing the lower case converted string but modifying the non converted string so unfortunately that won't work. Unless there is a way to TrimPrefix without case sensitivity?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah no, you are actually trimming rawURL
, nevermind.
I added a trimPrefixIC function to improve the readability of the code. Also removed some unessesary code
What is the current state of this @PatrLind ? Do you think it is enough to bump the |
The code is now in urlx, I changed the ParseOrXXX functions to use urlx.Parse() instead of url.Parse() as it still is in urlx.
I have been a bit busy on other things so this was kind of forgotten. |
Hm, in essence this means that file watching does not work on windows? Although it worked for the first case. Maybe it is flaky? Can you reproduce that locally? I don't have a windows machine at hand. |
Damn, the windows tests are flaky AF... I just reran it and |
Github actions overwrites logs on rerun 😪 here are at least some logs I could gather: Maybe you can have a look if you happen to have time? But I think this is not related to this PR anyway... |
Ok, I have dived deep in the rabbit hole...
The problem with using So to recap. The reason the tests fail on Windows is because the old configuration is cached. This happens randomly depending on the timing of the tests. It will probably fail more on a fast computer than a slow computer. So what is the solution? I am not exactly sure, but I think a good way to start would be to not use the configChangedAt in viper for anything related to caching or to tell if the configuration has changed. It would be better to use a configuration version number that increases monotonically for each change instead.
Edit:
|
Thanks for your debugging effort 👍 |
I guess you can merge, but no rush if you want to do more validation. |
Hm so I looked into this and while your observations will certainly be correct, However, I think this shouldn't be part of this PR anyways. Thank you so, so much for your hard work! @vinckr will send some goodies your way :) |
@aeneasr oh, thanks! :) |
Ah, my bad - I still think that this is not what is causing the issue (even though it definitely is an issue!) as the problem with the configs not being properly read in also appears in hydra, which does not use this method or value. In any case, I will try to debug this in ORY Hydra now and then will probably come back to the windows tests here! |
Related PR: #514
I asked in the Slack chat if I should make this PR and it seems like @zepatrik and @jfcurran gave their permission with an emoji.
Proposed changes
I had the same issue as the person creating PR 514. That file:// URLs wasn't working correctly under Windows.
I looked into it a bit and it seems like the original author of the PR and also Ory have misunderstood how file:// URLs work.
I have added a helper function that can extract the file path from the URL for as many situations I can think of.
I have also added a few non standard ways (relative paths and incomplete URLs) to deal with file:// URLs because those were used in the original tests from Ory.
Checklist
vulnerability. If this pull request addresses a security. vulnerability, I
confirm that I got green light (please contact
[email protected]) from the maintainers to push
the changes.
works.
Further comments
The tests didn't work on Windows before my fix, but now they work except for a test called TestFetcherWatchRepositoryFromKubernetesConfigMap. I am not sure how to make this test succeed since it seems to be written for non windows systems only. I have also run the tests under WSL (Linux) and they all pass, so I think this change should not cause any regressions.
I don't particularly like all the workarounds in the helper function, but it seems like it is common for people to write incorrect file:// URLs so I think they are needed.