Deleting a large quantity of files makes the UI unresponsive

[federicobozzini]

Bug Description:

Steps to Reproduce:

1. Add a large quantity of files and folders in the Theia workspace (>10GB)
2. Delete the files all at once (from the UI, select all, open the context menu and click on the delete menu item)
3. Interact with any other item left in the workspace (EG: expanding folder). Nothing happens

Depending on the nature of the files delete, an error notification might be shown in the UI:

Unable to watch for file changes in this large workspace. The information you see may not include recent file changes.

Sometimes an error is also thrown by NSFW here. The error is:
Service shutdown: root path changed (renamed or deleted)
This is probably misleading because the root path (the Theia workspace) has not been renamed or deleted.

Additional Information

• Operating System: Ubuntu 20.04
• Theia Version: 1.27

[tsmaeder]

I was not able to reproduce the problem on Windows; here's what I did:

1. created a workspace with 10 copies of the theia node_modules folder for a total of > 10Gb.
2. Expanded the node_modules folders and selected everything.
3. Select "Delete" from the context menu
4. Observe: the delete took around a minute, but proceeded without problems.

[tsmaeder]

@AlexandraBuzila is your scenario different?

[AlexandraBuzila]

Hi @tsmaeder,
I was able to reproduce the problem in Ubuntu, with the electron example.
My workspace also contained packages from the node_modules folder, as you described in your test steps, but I pasted them directly into the workspace directory. There were about 1200 roots.
If I interacted with the Explorer while the delete operation was running (e.g. expanding folders), sometimes an error popped up (Unable to watch for file changes in this large workspace. The information you see may not include recent file changes.) and the Explorer was not refreshing properly anymore.

[tsmaeder]

I ran into a problem where the delete would fail with an unrelated error

Error: ENOENT: no such file or directory, mkdir '/var/snap/firefox/common/host-hunspell./.Trash-1000'

probably due to this:

https://askubuntu.com/questions/1431815/lsblk-root-partition-mounted-on-and-var-snap-firefox-common-host-hunspell

[tsmaeder]

I remove Firefox from my system and now I'm getting

Error: Service shutdown: root path changed (renamed or deleted)

after getting around 50 correct DELETED notifications. Debugging through the C++ code reveals that nsfw tries to remove the watched root directory when it should be removing the subdirectory that is being removed. Currently not sure why this is happening, but I can reproduce in a simple nodejs program, so the problem should be debuggable.

The gist here is this seems to be a nsfw problem. I'll be away for a week and pick this up again then.

[tsmaeder]

The error case occurs when more than one "MOVED_FROM" event is in the buffer delivered to InotifyEventLoop#work(). The algorithm then decides that it's not a delete of a directory form the watched directory with handle 1, but a remove of the watched directory with handle 1. Investigating as to the cause.

[tsmaeder]

Methinks I have a fix: Axosoft/nsfw#171. Now waiting for feedback from nsfw folks.

[marcdumais-work]

@tsmaeder I see this error message with a significant portion of the CI failures we have in this repo [1]. It would be awesome if your fix helped.

[1]: e.g.: https://github.com/eclipse-theia/theia/actions/runs/3524336163/jobs/5909603214#step:7:4339

[tsmaeder]

PR has been merged. Waiting for a new release of nsfw.

[tsmaeder]

We have a new release: https://github.com/Axosoft/nsfw/releases/tag/v2.2.4