Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Profile IO operations. #807

Closed
rwblair opened this issue Jul 30, 2019 · 3 comments
Closed

Profile IO operations. #807

rwblair opened this issue Jul 30, 2019 · 3 comments
Assignees
@nellh

Comments

@rwblair
Copy link
Member

rwblair commented Jul 30, 2019 •
edited
Loading

Validator may be hitting filesystems too hard:
OpenNeuroOrg/openneuro#1219
@nellh
Copy link
Member

nellh commented Jul 30, 2019

Specifically we're seeing this on NFSv4 with datasets that have many (200+) participants.

@nellh nellh self-assigned this Jul 30, 2019
@nellh
Copy link
Member

nellh commented Aug 1, 2019 •
edited
Loading

The worst case dataset spends 65% of the time running lstat syscalls in https://github.com/bids-standard/bids-validator/blob/master/bids-validator/utils/files/readDir.js#L129 - 25% is spent on the actual readdir calls. We do make an extra lstat call for every annexed file (since they are symlinks), which could be avoided for OpenNeuro where directories will never be symlinks but is useful generally.

@nellh nellh mentioned this issue Aug 2, 2019
Merged
@nellh
Copy link
Member

nellh commented Aug 14, 2019

I think this is solved for now.

@nellh nellh closed this as completed Aug 14, 2019
@nellh nellh reopened this Aug 22, 2019
@nellh nellh closed this as completed Aug 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nellh nellh

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nellh @rwblair