Add support for Shared Google Drives

[rsanger]

Fixes issue #368

This adds support for shared google drives (previously called team drives) storage.
I'm not sure if this is the best way to specify shared drives, but it works and remains backwards compatible.

A shared drive can be accessed via
gcd://sharedDriveId@path/to/storage

sharedDriveId is optional and if omitted duplicacy stores to the user's drive.
This remains backwards compatible with existing drives. E.g.
gcd://path/to/storage

I have not included any documentation or tests, as docs and tests appear to be maintained outside of this repo.

[CLAassistant]

All committers have signed the CLA.

[draimundo]

@rsanger very nice! Tried to rebuild duplicacy based on a go get from this repo, then updated with your files, but I'm getting an error Failed to load the Google Drive storage at gcd://sharedDriveId@/: Failed to create directory 'sharedDriveId@': Parent directory '' does not exist. How did you get it to work? (Your request comment couldn't be clearer, but somehow I can't get it to work...)

[rsanger]

@draimundo This is a carryover of existing drive behaviour (which doesn't make much sense).
You cannot save to the root of the drive, you need to save to a folder.

So you need to use at least:
gcd://sharedDriveId@backupdir/
You can get the sharedDriveId if you open the shared drive in a browser and take the last part of the URL.
And you need to create a backupdir folder in the shared drive.

I will look into removing that limitation (or at least improving the error), as it doesn't make sense for shared drives.

[rsanger]

As per @draimundo's comment.
Updated to support writing to the root of a drive using:
gcd://sharedDriveId@/

[vaishraj]

any thoughts on when this will make it to a release? thanks

[RXWatcher]

I know its not in here and I'm not a golang programmer so I can't add it but the inclusion of using Service Accounts would make this even better. https://cloud.google.com/compute/docs/access/service-accounts for details on them. Google limits the Google drive uploads to 750GBs a day and 10TBs a day down. Each Service Account has it's own limits that act independently of the main accounts limits. I dedicate a service account to a Team Drive today for rclone and restic(which uses the rclone). This allows me to have dedicated transfer limits for my backups vs it being shared with my main account.

[vaishraj]

An update on this - I was able to pull the master and merge rsanger's changes. I have been backing up three different folders and it has been working wonderfully.

Only caveat is that gcd://sharedDriveId@backup/: creates a : sub-folder in backup folder. I tried to run it without the : in the URI and it didn't accept it.

For now, that's a minor annoyance but I can live with that. I will migrate to the official release when this is pulled in but wanted to provide an update on the real world experience from the change.

[rsanger]

An update on this - I was able to pull the master and merge rsanger's changes. I have been backing up three different folders and it has been working wonderfully.

Only caveat is that gcd://sharedDriveId@backup/: creates a : sub-folder in backup folder. I tried to run it without the : in the URI and it didn't accept it.

For now, that's a minor annoyance but I can live with that. I will migrate to the official release when this is pulled in but wanted to provide an update on the real world experience from the change.

Thanks for the feedback.

I think I have found the problem. The drive code won't automatically create a folder in the root of the drive (for all types of drives, both user or shared), but will automatically create subdirectories in all other cases.

So with an empty drive (shared or default), where 'my-backup' does not exist is see:

$ ./duplicacy_main.exe -d -v init testnoslash gcd://0AGpvP3A-2CNpUk9PVA@my-backup
Reading the environment variable DUPLICACY_GCD_TOKEN
Failed to load the Google Drive storage at gcd://0AGpvP3A-2CNpUk9PVA@my-backup: Failed to create directory 'my-backup': Parent directory '' does not exist

Create 'my-backup' through the Gdrive website. Then I see:

$ ./duplicacy_main.exe -d -v init testnoslash gcd://0AGpvP3A-2CNpUk9PVA@my-backup
Reading the environment variable DUPLICACY_GCD_TOKEN
Reading the environment variable DUPLICACY_GCD_TOKEN
Compression level: 100
Average chunk size: 4194304
Maximum chunk size: 16777216
Minimum chunk size: 1048576
Chunk seed: 6475706c6963616379
Hash key: 6475706c6963616379
ID key: 6475706c6963616379
File chunks are encrypted
Metadata chunks are encrypted
D:\Desktop\DuplicacyShared will be backed up to gcd://0AGpvP3A-2CNpUk9PVA@my-backup with id testnoslash

Now that my-backup exists, I can create a backup in any subdirectory whether it exists or not.

$ rm -r .duplicacy
$ ./duplicacy_main.exe -d -v init testnoslash gcd://0AGpvP3A-2CNpUk9PVA@my-backup/folder1/folder2
Reading the environment variable DUPLICACY_GCD_TOKEN
Reading the environment variable DUPLICACY_GCD_TOKEN
Compression level: 100
Average chunk size: 4194304
Maximum chunk size: 16777216
Minimum chunk size: 1048576
Chunk seed: 6475706c6963616379
Hash key: 6475706c6963616379
ID key: 6475706c6963616379
File chunks are encrypted
Metadata chunks are encrypted
D:\Desktop\DuplicacyShared will be backed up to gcd://0AGpvP3A-2CNpUk9PVA@my-backup/folder1/folder2 with id testnoslash

I see identical behaviour both with and without a trailing slash.
Can you confirm if this is the behaviour you are seeing?

[vaishraj]

Yes. That’s exactly what was happening in my case. Thanks for confirming.

…

On Jan 10, 2020, 6:03 PM -0800, Richard Sanger ***@***.***>, wrote: > An update on this - I was able to pull the master and merge rsanger's changes. I have been backing up three different folders and it has been working wonderfully. > Only caveat is that ***@***.***/: creates a : sub-folder in backup folder. I tried to run it without the : in the URI and it didn't accept it. > For now, that's a minor annoyance but I can live with that. I will migrate to the official release when this is pulled in but wanted to provide an update on the real world experience from the change. Thanks for the feedback. I think I have found the problem. The drive code won't automatically create a folder in the root of the drive (for all types of drives, both user or shared), but will automatically create subdirectories in all other cases. So with an empty drive (shared or default), where 'my-directory' does not exist is see: $ ./duplicacy_main.exe -d -v init testnoslash ***@***.*** Reading the environment variable DUPLICACY_GCD_TOKEN Failed to load the Google Drive storage at ***@***.***: Failed to create directory 'my-directory': Parent directory '' does not exist Create 'my-directory' through the Gdrive website. Then I see: $ ./duplicacy_main.exe -d -v init testnoslash ***@***.*** Reading the environment variable DUPLICACY_GCD_TOKEN Reading the environment variable DUPLICACY_GCD_TOKEN Compression level: 100 Average chunk size: 4194304 Maximum chunk size: 16777216 Minimum chunk size: 1048576 Chunk seed: 6475706c6963616379 Hash key: 6475706c6963616379 ID key: 6475706c6963616379 File chunks are encrypted Metadata chunks are encrypted D:\Desktop\DuplicacyShared will be backed up to ***@***.*** with id testnoslash Now that my-backup exists, I can create a backup in any subdirectory whether it exists or not. $ rm -r .duplicacy $ ./duplicacy_main.exe -d -v init testnoslash ***@***.***/folder1/folder2 Reading the environment variable DUPLICACY_GCD_TOKEN Reading the environment variable DUPLICACY_GCD_TOKEN Compression level: 100 Average chunk size: 4194304 Maximum chunk size: 16777216 Minimum chunk size: 1048576 Chunk seed: 6475706c6963616379 Hash key: 6475706c6963616379 ID key: 6475706c6963616379 File chunks are encrypted Metadata chunks are encrypted D:\Desktop\DuplicacyShared will be backed up to ***@***.***/folder1/folder2 with id testnoslash I see identical behaviour both with and without a trailing slash. Can you confirm if this is the behaviour you are seeing? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[rsanger]

Ok.

Looking at the init docs

Duplicacy will create the destination path on the storage if it does not already exist.

It should automatically create this directory, so I will work on a fix.

[rsanger]

@vaishraj I've pushed a fix aa07fee
Let me know if that has fixed the problem for you

[gilbertchen]

Thank you for your contribution. This looks great!

[gilbertchen]

This pull request has been mentioned on Duplicacy Forum. There might be relevant details there:

https://forum.duplicacy.com/t/cli-release-2-5-0/3448/1