Include file inside excluded path will include all files in excluded path

[Reefkeeper]

BackInTime 1.1.12 on Linux Mint 17.3
I want to backup my /home/myname directory but want to exclude the Dropbox directory.
Although serveral directories as "Downloads", "Encfs" are correctly excluded as they are marked so, BIT does backup the directory /home/myname/Dropbox which is also in the list of exclusions.
See screenshot below!
What makes this directory different that it cannot be excluded?
Thanks!
Friedhelm

[Germar]

AFAIK there is nothing special with Dropbox folder. Did you add it through Ornder hinzufügen? If not, please double-check that it is spelled correct. It is case sensitive, maybe your original folder is camel-case DropBox?

[Reefkeeper]

Hi Germar,
thank you for reassuring that there is nothing generally wrong with using the Dropbox folder.
Yes, I added it via "Ordner hinzufügen" (add folder), but I also tried manual adding, with and without trailing '/' and '*' with no change. Consulting the ~/.config/backintime file does not reveal anything wrong (excerpt):

profile2.snapshots.exclude.2.value=/home/friedhelm/Daten/_fotos/Lightroom/Backups
profile2.snapshots.exclude.20.value=.Private
profile2.snapshots.exclude.21.value=/home/friedhelm/Dropbox
profile2.snapshots.exclude.22.value=/home/friedhelm/Encfs
profile2.snapshots.exclude.23.value=/home/friedhelm/Downloads

As said, /home/friedhelm/Encfs and /home/friedhelm/Downloads are excluded, but Dropbox (spelled correctly) is always backed up.

[Germar]

How comes that you've profile2.snapshots.exclude.20 up to 23? This should normaly be profile2.snapshots.exclude.1 to 5 and you should have profile2.snapshots.exclude.size=5. Please make a backup of your config and manually fix this in an editor.

[Reefkeeper]

Sorry, I showed only an excerpt from the config file. I do not exclude only those 5 directories, but 23.
BIT automatically makes 2 digit numbers when it wants to keep a sort order, and sorts them according to their character values.

But to eliminate even this possible cause, I reduced the number of exclusions to 2 items, Dropbox and the Downloads folder:
Here is the config file, I tested with this and - Dropbox is still being backed up, while Downloads folder is not. I do not understand it:

profile2.name=2 TB Seagate
profile2.qt4.last_path=/home/friedhelm
profile2.qt4.places.SortColumn=1
profile2.qt4.places.SortOrder=0
profile2.qt4.settingsdialog.exclude.SortColumn=1
profile2.qt4.settingsdialog.exclude.SortOrder=0
profile2.qt4.settingsdialog.include.SortColumn=1
profile2.qt4.settingsdialog.include.SortOrder=0
profile2.snapshots.automatic_backup_anacron_period=1
profile2.snapshots.automatic_backup_anacron_unit=20
profile2.snapshots.automatic_backup_day=1
profile2.snapshots.automatic_backup_mode=0
profile2.snapshots.automatic_backup_time=0
profile2.snapshots.automatic_backup_weekday=7
profile2.snapshots.backup_on_restore.enabled=true
profile2.snapshots.bwlimit.enabled=false
profile2.snapshots.bwlimit.value=3000
profile2.snapshots.check_for_changes=true
profile2.snapshots.continue_on_errors=true
profile2.snapshots.copy_links=false
profile2.snapshots.copy_unsafe_links=false
profile2.snapshots.cron.ionice=true
profile2.snapshots.cron.nice=true
profile2.snapshots.cron.redirect_stderr=false
profile2.snapshots.cron.redirect_stdout=true
profile2.snapshots.custom_backup_time=8,12,18,23
profile2.snapshots.dont_remove_named_snapshots=true
profile2.snapshots.exclude.1.value=/home/friedhelm/Dropbox
profile2.snapshots.exclude.2.value=/home/friedhelm/Downloads
profile2.snapshots.exclude.bysize.enabled=false
profile2.snapshots.exclude.bysize.value=500
profile2.snapshots.exclude.size=2
profile2.snapshots.full_rsync=false
profile2.snapshots.full_rsync.take_snapshot_regardless_of_changes=false
profile2.snapshots.include.1.type=0
profile2.snapshots.include.1.value=/home/friedhelm
profile2.snapshots.include.size=1
profile2.snapshots.local.nocache=false
profile2.snapshots.local.password.save=false
profile2.snapshots.local.password.use_cache=true
profile2.snapshots.local_encfs.path=/media/friedhelm/Externe_HD3
profile2.snapshots.log_level=3
profile2.snapshots.min_free_inodes.enabled=true
profile2.snapshots.min_free_inodes.value=2
profile2.snapshots.min_free_space.enabled=true
profile2.snapshots.min_free_space.unit=20
profile2.snapshots.min_free_space.value=10
profile2.snapshots.mode=local
profile2.snapshots.no_on_battery=false
profile2.snapshots.notify.enabled=true
profile2.snapshots.path=/media/friedhelm/Externe_HD3
profile2.snapshots.path.host=linux
profile2.snapshots.path.profile=2
profile2.snapshots.path.user=friedhelm
profile2.snapshots.preserve_acl=false
profile2.snapshots.preserve_xattr=false
profile2.snapshots.remove_old_snapshots.enabled=true
profile2.snapshots.remove_old_snapshots.unit=80
profile2.snapshots.remove_old_snapshots.value=10
profile2.snapshots.rsync_options.enabled=false
profile2.snapshots.rsync_options.value=
profile2.snapshots.smart_remove=true
profile2.snapshots.smart_remove.keep_all=2
profile2.snapshots.smart_remove.keep_one_per_day=7
profile2.snapshots.smart_remove.keep_one_per_month=24
profile2.snapshots.smart_remove.keep_one_per_week=4
profile2.snapshots.smart_remove.run_remote_in_background=false
profile2.snapshots.ssh.cipher=default
profile2.snapshots.ssh.host=
profile2.snapshots.ssh.ionice=false
profile2.snapshots.ssh.nice=false
profile2.snapshots.ssh.nocache=false
profile2.snapshots.ssh.path=
profile2.snapshots.ssh.port=22
profile2.snapshots.ssh.prefix.enabled=false
profile2.snapshots.ssh.prefix.value=PATH=/opt/bin:/opt/sbin:\$PATH
profile2.snapshots.ssh.private_key_file=
profile2.snapshots.ssh.user=friedhelm
profile2.snapshots.use_checksum=false
profile2.snapshots.user_backup.ionice=false

As you can see, the snapshots.exclude.size=2 (/home/friedhelm/Dropbox and /home/friedhelm/Downloads) and the snapshots.include.size=1 (/home/friedhelm).

Dropbox is still backed up!!!! (Grrrrr..)

[Germar]

Hmm, that's strange. What's the output of ls -la /home/friedhelm/Dropbox/?

[Reefkeeper]

ls -la /home/friedhelm/Dropbox/
insgesamt 280
drwx------ 12 friedhelm friedhelm  4096 Mär 26 16:23 .
drwxr-xr-x 63 friedhelm friedhelm  4096 Mär 26 16:23 ..
drwxr-xr-x  3 friedhelm friedhelm  4096 Dez  7  2014 Apps
drwxr-xr-x  4 friedhelm friedhelm  4096 Feb 18 17:49 boxcryptor.bc
drwxr-xr-x  2 friedhelm friedhelm  4096 Sep  8  2014 .comments
-rwxr-xr-x  1 friedhelm friedhelm    35 Jul 30  2015 .dropbox
drwxr-xr-x  5 friedhelm friedhelm 81920 Mär 26 20:03 .dropbox.cache
drwxr-xr-x  2 friedhelm friedhelm  4096 Mär 26 17:46 ForMyEyesOnly
-rw-r--r--  1 friedhelm friedhelm 23294 Jan 10  2014 hangouts_message.ogg
drwxr-xr-x  3 friedhelm friedhelm 12288 Mär 25 18:59 Kamera-Uploads
drwxr-xr-x  2 friedhelm friedhelm  4096 Jun 22  2013 Kameravideos
drwxr-xr-x  4 friedhelm friedhelm  4096 Dez 25 12:28 PDF
drwxr-xr-x  3 friedhelm friedhelm  4096 Mär 25 13:09 Photos
drwxr-xr-x  2 friedhelm friedhelm  4096 Mär 28  2015 Public
-rw-r--r--  1 friedhelm friedhelm  2837 Dez 28  2014 schwarzer_hintergrund.png
-rw-r--r--  1 friedhelm friedhelm 62266 Dez  6  2014 Tux.png

[Germar]

Looks okay to me. Sorry, I'm out of ideas.

[colinl]

Could this happen if there were something like a link to the Dropbox folder from somewhere else in the heirarchy? Though I suppose then it would appear in the snapshot in a different place.

[Germar]

@colinl As long as you did not activate Copy links or Copy unsave links symlinks will be backed up as symlinks. So the linked folder should not be included.

[Reefkeeper]

Thank you Germar and Colinl for your ideas and Happy Easter!
Those options (copy links, copy unsafe links, symlinks) are not activated with me.
Seems like I have to live with it.
Anyway if someone should have spare time he could try out backing up his home directory without his Dropbox directory ;-)

Nevertheless let me say a big THANK YOU for your software!
Friedhelm

[colinl]

A couple of further ideas. If in the BIT GUI you select the latest snapshot does Dropbox appear in the folder list under Backup folders? If not then how do you browse to it in the snapshot? Also have you explicitly included Dropbox in the Include folders?

[Reefkeeper]

If in the BIT GUI you select the latest snapshot does Dropbox appear in the folder list under Backup folders?
Yes, it does! See screenshot, additionally I check the directories on the ext. HD.

As you can see "Downloads" for example, does not appear, which is correct.

Also have you explicitly included Dropbox in the Include folders?
No Dropbox is not included explicitly. Only /home/friedhelm is indicated (which includes Dropbox implicitly).
This is what I include:

[Germar]

On LinuxMintUsers I've seen you actually tried to include parts of the Dropbox folder before. Which is totally fine and SHOULD work. But maybe this previous attempts causing your troubles now.

So just to double-check, could you please rename your ~/.config/backintime/config into config.sic and create a new profile pointing to a new, clean directory?

I tried your config above yesterday and it worked as expected. My dummy Dropbox folder was excluded. So I guess this has something to do with your previous snapshots. BTW do you use Full rsync mode or not?

[Reefkeeper]

Hi Germar,
I made the test you proposed, and it shows the problem! I do not use full rsync mode. Do you suggest I should? The target file system type is EXT4.

Step 1: Completely new config file
included: /home/friedhelm
excluded: /home/friedhelm/Dropbox and several directories as proposed by BIT.
The config looks like this:

profile1.snapshots.exclude.1.value=.gvfs
profile1.snapshots.exclude.10.value=/dev/*
profile1.snapshots.exclude.11.value=/run/*
profile1.snapshots.exclude.12.value=/etc/mtab
profile1.snapshots.exclude.13.value=/var/cache/apt/archives/*.deb
profile1.snapshots.exclude.14.value=lost+found/*
profile1.snapshots.exclude.15.value=/tmp/*
profile1.snapshots.exclude.16.value=/var/tmp/*
profile1.snapshots.exclude.17.value=/var/backups/*
profile1.snapshots.exclude.18.value=.Private
profile1.snapshots.exclude.19.value=/home/friedhelm/_fotos
profile1.snapshots.exclude.2.value=.cache/*
profile1.snapshots.exclude.20.value=/home/friedhelm/Downloads
profile1.snapshots.exclude.21.value=/home/friedhelm/Encfs
profile1.snapshots.exclude.22.value=/home/friedhelm/VirtualBox VMs
profile1.snapshots.exclude.23.value=/home/friedhelm/Zeitung
profile1.snapshots.exclude.24.value=/home/friedhelm/Dropbox
profile1.snapshots.exclude.25.value=/home/friedhelm/Daten
profile1.snapshots.exclude.3.value=.thumbnails*
profile1.snapshots.exclude.4.value=[Tt]rash*
profile1.snapshots.exclude.5.value=*.backup*
profile1.snapshots.exclude.6.value=*~
profile1.snapshots.exclude.7.value=.dropbox*
profile1.snapshots.exclude.8.value=/proc/*
profile1.snapshots.exclude.9.value=/sys/*
profile1.snapshots.exclude.bysize.enabled=false
profile1.snapshots.exclude.bysize.value=500
profile1.snapshots.exclude.size=25
profile1.snapshots.full_rsync=false
profile1.snapshots.full_rsync.take_snapshot_regardless_of_changes=false
profile1.snapshots.include.1.type=0
profile1.snapshots.include.1.value=/home/friedhelm
profile1.snapshots.include.size=1

Result: Dropbox is not backed up, which is correct!

Step 2:
Now include 1 file within Dropbox folder: ../ForMyEyesOnly/test.txt.
The config now looks like this (these are the only differences to #1)

profile1.snapshots.include.2.type=1
profile1.snapshots.include.2.value=/home/friedhelm/Dropbox/ForMyEyesOnly/test.txt
profile1.snapshots.include.size=2

Now the complete Dropbox folder with everything in it is backed up!!! This is wrong.

Step 3:
Remove the include in step 2 within Dropbox folder: ../ForMyEyesOnly/test.txt.
Result:
The config file looks exactly as in Step 1 AND the Dropbox folder is NOT backed up, which is correct.
Although this is a contradiction to the previous runs where removing the inclusion of a specific Dropbox file did not change the behaviour and the Dropbox folder was backed up although it shouldn't. This may as you said have something to do with previous snapshots.

In total:
I do not manage to exclude a subdirectory within a folder which is to be backed up.
If at least one file is indicated for backup, all the folder contents is backed up and the exclusion is ignored. In my opinion this has to do with the way rsync treats the exclusions?

Thank you for your patience!

[Germar]

Okay, I was able to reproduce this here. I'll have a look why the whole excluded path is backed up when only a single file inside the excluded path is included.

[ColiZei]

Hey!
I have the same problem with my Nextcloud folder.
Are there any plans to fix this bug?
Thanks!

[protist]

I just came across this bug with my setup too. Is there any workaround? Can we add some path (perhaps with wildcards), or add some additional options to rsync?

[buhtz]

@ColiZei and @protist : Can you please specify your version of BackInTime and the OS your are using?

If this is a reproducible bug it will be fixed. But currently there is not concrete time plan.

Currently a new group of developers/maintainers is forming around @Germar . This will take its time.

[ColiZei]

Backintime 1.3.2-2
Manjaro Linux
Gnome 42.4
Kernel: 5.18.17-1-MANJARO

[protist]

This will take its time.

Of course, @Codeberg-AsGithubAlternative-buhtz! I have no sense of entitlement that any FOSS software should be rapidly updated. Totally understandable!

I was posting here more to see if there was some workaround, similar to #1247. I wasn't sure if it was an issue with rsync options.

I'm also using the latest backintime 1.3.2-2, with up-to-date Arch Linux client and server.

[Saroumane]

Hello, I also stumbled on this bug today, trying to backup a single folder /media/user/a/b/c/d (in my "include" settings) whereas I have /media/* on my "exclude" settings. I reproduce the fact that the excluding statement is not operating anymore, everything is saved.
As a workaround, I intend to create a new dedicated profile that will trigger daily, just to backup this specific folder.

[emtiu]

Okay, so this behavior has been confirmed with backintime, but how about testing with rsync alone? Is it fundamental to the operation of rsync (probably depending on the order of the --exclude arguments?), or something that happpens only for backintime?

[buhtz]

Sidenote: I found a comment in the code about that issue here. See common.snapshots.Snapshots.rsyncSuffix().

Not clear if this can help.

[emtiu]

Sidenote: I found a comment in the code about that issue here. See common.snapshots.Snapshots.rsyncSuffix().

Link to the code:

backintime/common/snapshots.py

Lines 1797 to 1800 in 88d19d4

	# TODO: fix bug #561:
	# after rsync_exclude we need to explicite include files inside excluded
	# folders, recursive exclude folder-content again and finally add the
	# rest from rsync_include2

[insubstudios]

Hi,

Adding a double star to the end of the folder path, /path/to/folder**, will exclude it and all of it's children. But because of #1420, it will also exclude any of it's children you've tried to include.

* - matches any character except slashes
** - matches any character including slashes

Because rsync needs to "see" anything to add it and a universal exclude (--exclude=*) is used, each parent folder of included files and folders needs to be included. BiT does this by looping thru each include's parent folders and adding a include rule for each. Hence your extra files. (i get into how rsync and BiT handle include/exclude ordering in a comment on the 1420.)

EDIT: nevermind. i no longer think this is a good solution. i think documentation and help text is the way to go.
i think the fix here would be to change the way the Exclude > Add Folder button works so that it adds ** the end of the folder path. And a note in the Add Exclude Pattern dialogue would be nice too.

Would switching btnExcludeFolderClicked() in settingsdialog.py from self.addExclude_('{}'.format(path)) to self.addExclude_('{}**'.format(path)) like this work?

[insubstudios]

i was playing around with this a little more and have some more nuance depending on what inside of your excluded folders you want to include. i also found a workaround to my problem so everything can work in BiT now.

1. exclude with no inclusions (default for "Add Folder")
   • dir/ or dir: Only recommended if no sub-items will be included. If any children are included, All of that children's parent folders and cousins will be included.
2. exclude with included files
   • dir/** or dir**: Will exclude all children and grandchildren. Only items explicitly names will be included. Folders will be empty. Only recommended to include files from excluded folders or patterns.
   • to workaround bug 1420, put your include parameters into "additional options" under the "Expert Options" tab in settings.
3. exclude with included shallow folder (subdirectories are ignored)
   • dir/**/ or dir**/: this pattern excludes dir/ and all of it's subdirectories. Items explicitly named in includes AND folder's direct children files (and symlinks (depending on settings??)) will be included.
4. exclude with included deep folder
   • To get ALL children of an included folder without all the cousins, add an exclusion for the children of each parent folder.
     • for folder path/to/exclude/with/included-folder/, exclude:
       • path/to/exclude/with/*
       • path/to/exclude/*
       • path/to/*
       • path/*

note: the trailing slash of a dir mostly doesn't matter. none at all if a full path from root is given. but patterns may match files too. with trailing slash, only folders will be matched.

[protist]

Thank you @insubstudios for the detailed testing and post!

I can confirm that number 3 works as suggested. I'll also say that I'm a bit worried about future updates breaking the ** usage, so I'm still going to explicitly list everything in the meantime.

[insubstudios]

thanks @protist

the ** is part of rysnc and was introduced in 2006 in v2.6.7. i think it'll stick around :-)

[aryoda]

May I suggest to add a help page somewhere in this dialog to support users applying the right exclusion pattern (as described above) and add the "feature request" tag to this issue:

PS: @insubstudios THX a lot for your excellent summary of the include/exclude variations!

[protist]

the ** is part of rysnc and was introduced in 2006 in v2.6.7. i think it'll stick around :-)

Ah yes. Thanks again @insubstudios . I just had a look at man rsync, which clarifies why this works.

   o      a '*' matches zero or more non-slash characters.

   o      a '**' matches zero or more characters, including slashes.

[buhtz]

May I suggest to add a help page somewhere in this dialog

I would love to have something like this. ❤️