Problems with "Smart remove" in "Run in background on remote Host. EXPERIMENTAL!" when to much snapshots to delete

[buhtz]

Thanks to @aryoda comment in PR #1351 I found that issue.

BIT has an "EXPERIMENTAL" feature where the "smart remove" is done in background via SSH.

Usually it isn't done in background. And based on my observations it isn't done with SSH even if an SSH snapshot profiles is used. The remote is mounted and then a simple rm (more precise shutil.rmtree()) is done. No SSH is involed.

The problem is that the feature seems really to be experimental. In my test scenario it creates a command more than 10.000 characters long which seems to be to long for SSH. Here you see just a snipped example.

['ssh', '-o', 'ServerAliveInterval=240', '-o', 'LogLevel=Error', '-o', 'IdentityFile=/home/user/.ssh/id_rsa', '-p', '22', 'user@localhost', 'screen -d -m bash -c "(TMP=\\$(mktemp -d); test -z \\"\\$TMP\\" && exit 1; test -n \\"\\$(ls \\$TMP)\\" && exit 1; logger -t \\"backintime smart-remove [$BASHPID]\\" \\"start\\"; flock -x 9; logger -t \\"backintime smart-remove [$BASHPID]\\" \\"got exclusive flock\\"; test -e \\"/home/user/dest blank/backintime/localhost/user/1/20221106-214507-663\\" && (logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221106-214507-663 still exist\\"; sleep 1; rsync -a --delete -s \\"\\$TMP/\\" /home/user/dest blank/backintime/localhost/user/1/20221106-214507-663; rmdir \\"/home/user/dest blank/backintime/localhost/user/1/20221106-214507-663\\"; logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221106-214507-663 remove done\\"); test -e \\"/home/user/dest blank/backintime/localhost/user/1/20221105-214500-483\\" && (logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221105-214500-483 still exist\\"; sleep 1; rsync -a --delete -s \\"\\$TMP/\\" /home/user/dest blank/backintime/localhost/user/1/20221105-214500-483; rmdir \\"/home/user/dest blank/backintime/localhost/user/1/20221105-214500-483\\"; logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221105-214500-483 remove done\\"); test -e \\"/home/user/dest blank/backintime/localhost/user/1/20221104-221056-108\\" && (logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221104-221056-108 still exist\\"; sleep 1; rsync -a --delete -s \\"\\$TMP/\\"

SNIPPED

&& (logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221019-230601-797 still exist\\"; sleep 1; rsync -a --delete -s \\"\\$TMP/\\" /home/user/dest blank/backintime/localhost/user/1/20221019-230601-797; rmdir \\"/home/user/dest blank/backintime/localhost/user/1/20221019-230601-797\\"; logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221019-230601-797 remove done\\"); rmdir \\$TMP) 9>\\"/home/user/dest blank/backintime/localhost/user/1/smartremove.lck\\""']

The problem is obvious and also some solutions are possible. But when this problem is solved maybe there are other problems coming up with the experimental feature. I suspect that this will take much more time and testing resources.

My first suggestion was to "deactivate" that feature for the 1.3.3 release. And after that release we have more time to dive into it.

Maybe @Germar can give us more background info about that feature.

A problem could be. That the feature was introduced with 1.1.6 in the year 2015 ([CHANGELOG](* add option to run Smart Remove in background on remote host (https://launchpad.net/bugs/1457210)) and #257). so it is 7 years old and could be more stable than experimental.

[aryoda]

There is a "history" of issues for this feature:

• SSH backup mode: option to launch smart removal "in the background" #257
• Question: How dangerous is the experimental removal of remote snapshots? #744
• SSH max arg length choke on SSH banner #581
• Deleting snapshots in the background: be less agressive #150

[buhtz]

Thanks for your research. After asking my pillow that night I wouldn't categorize that code as "experimental" in the meaning of "new and unstable". It is quite old like a good wine and there are some fixed issues about it. So deactivating it for the next release is a bad choice.

I assume I triggered a rare case with my tests scripts. Via faketimelib and other helper scripts I can generate snapshots for the last N days. Then activating smart remove will of course cause a lot of snapshots to be deleted at once.

I will investigate that further but treat the "to many snapshots at once" in a separate PR.

EDIT: The question is if we should touch that before the next release or better after it. Beside my special testing-use-case it seems that no one else is having problems with that feature. The last issue about that was in the year 2017.

[Germar]

This feature is working fine as far as I can tell. But I called it experimental because it need to control commands running on remote without having full control over remote (if that makes any sense 🤔)

The command is splitt into parts that fit into valid ssh commands ( BiT tests the max possible length for ssh commands for this)

[buhtz]

The command is splitt into parts that fit into valid ssh commands ( BiT tests the max possible length for ssh commands for this)

Maybe there is potential to optimize that maxlength feature. I'll investigate that further and first try to reproduce my problem again.

@Germar I haven't tested it yet but did you ever heard of the getconf ARG_MAX? If so, what was the reason that you didn't used that instead of creating sshMaxArg.py? And what does the acronym mid stands for? I understand that variable in ``sshMaxArg.py` but don't get behind its name. 😄

[Germar]

I'm not sure if I knew getconf ARG_MAX back in the days. But I don't think this would be helpfull because it's about the combination of the local ssh USER@HOST COMMAND and the remote COMMAND length.

Maybe you run Smart-remove manually without using Settingsdialog? This way you bypassed the check and config.sshMaxArgLength was 0 -> no limit