Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve how open file limits & blocking threads are handled #513

Closed
MarcusSorealheis opened this issue Dec 19, 2023 · 0 comments
Closed

Improve how open file limits & blocking threads are handled #513

MarcusSorealheis opened this issue Dec 19, 2023 · 0 comments
Assignees
@MarcusSorealheis
Labels
enhancement New feature or request

Comments

@MarcusSorealheis
Copy link
Collaborator

MarcusSorealheis commented Dec 19, 2023 •
edited by allada
Loading

If the number of blocking threads > the number of open files, we can encounter deadlocks. Currently we get around this issue by allowing the tokio runtime to spawn 10x the number of threads as there are background threads. A more proper solution is to refactor the way filesystem api works and have it interact with blocking threads more stable.
@MarcusSorealheis MarcusSorealheis self-assigned this Dec 19, 2023
@allada allada mentioned this issue Dec 19, 2023
Merged
8 tasks
@allada allada changed the title Deadlocks happen due to FS issue Handle how open file limits & blocking threads are handled better Dec 19, 2023
@allada allada added the enhancement New feature or request label Dec 19, 2023
@MarcusSorealheis MarcusSorealheis changed the title Handle how open file limits & blocking threads are handled better imeive how open file limits & blocking threads are handled better Dec 25, 2023
@MarcusSorealheis MarcusSorealheis changed the title imeive how open file limits & blocking threads are handled better Improve how open file limits & blocking threads are handled Dec 25, 2023
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
b74b3fc 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513
@allada allada mentioned this issue Feb 9, 2025
Merged
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
43044cc 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
d7a3655 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
d81c645 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
f2a2385 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
c8c51ba 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
a9302dc 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
3c8d578 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
8579c00 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
4bcf7c1 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
5c657c5 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
8e3ad06 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
2e86f91 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
fc58824 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
bc8b804 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
72fc743 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
511f002 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
af64864 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
a73bcab 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
a47c20e 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
c2298d4 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
3e2df54 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 11, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
56f427d 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit that referenced this issue Feb 12, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits (#1582)
8b89c31 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: #1288, #513, #1298, #527
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MarcusSorealheis MarcusSorealheis

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@allada @MarcusSorealheis