-
Notifications
You must be signed in to change notification settings - Fork 394
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
s3: update info on boto3 methods and permissions required...
in `imoprt-url`, `get-url` as well as `remote` and `remote add` command refs. Updates also related guides (install and config).
- Loading branch information
1 parent
a7ee6aa
commit 9d0819d
Showing
6 changed files
with
72 additions
and
44 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -38,10 +38,15 @@ DVC supports several types of (local or) remote locations (protocols): | |
| `hdfs` | HDFS | `hdfs://[email protected]/path/to/data.csv` | | ||
| `http` | HTTP to file with _strong ETag_ (see explanation below) | `https://example.com/path/to/data.csv` | | ||
|
||
> Depending on the remote locations type you plan to download data from you | ||
> might need to specify one of the optional dependencies: `s3`, `gs`, `ssh` (or | ||
> `all_remotes` to include them all) when | ||
> [installing DVC](/doc/get-started/install) with `pip`. | ||
> `remote://myremote/path/to/file` notation just means that a DVC | ||
> [remote](/doc/commands-reference/remote) `myremote` is defined, and when DVC | ||
> is running it internally expands this URL into a regular S3, SSH, GS, etc URL | ||
> by appending `/path/to/file` to the `myremote`'s configured base path. | ||
> [remote](/doc/commands-reference/remote) `myremote` is defined and when DVC is | ||
> running. DVC automatically expands this URL into a regular S3, SSH, GS, etc | ||
> URL by appending `/path/to/file` to the `myremote`'s configured base path. | ||
Another way to understand the `dvc get-url` command is as a tool for downloading | ||
data files. | ||
|
@@ -79,6 +84,8 @@ The above command will copy the `/local/path/to/data` file or directory into | |
|
||
</details> | ||
|
||
<details> | ||
|
||
### Click for AWS S3 example | ||
|
||
This command will copy an S3 object into the current working directory with the | ||
|
@@ -93,23 +100,13 @@ By default DVC expects your AWS CLI is already | |
DVC will be using default AWS credentials file to access S3. To override some of | ||
these settings, you could the options described in `dvc remote modify`. | ||
|
||
We use `boto3` library to set up a client and communicate with AWS S3. The | ||
following API methods may be performed: | ||
|
||
- `list_objects_v2`, `list_objects` | ||
- `head_object` | ||
- `download_file` | ||
- `upload_file` | ||
- `delete_object` | ||
- `copy` | ||
|
||
So make sure you have the following permissions enabled to enable all the above | ||
operations: | ||
|
||
- `s3:ListBucket` | ||
- `s3:GetObject` | ||
- `s3:PutObject` | ||
- `s3:DeleteObject` | ||
> We use the `boto3` library to and communicate with AWS S3. The following API | ||
> methods may be performed: | ||
> | ||
> - `head_object` | ||
> - `download_file` | ||
> | ||
> So make sure you have the `s3:GetObject` permission enabled. | ||
</details> | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -45,11 +45,11 @@ path. DVC currently supports seven types of remotes: | |
|
||
> Depending on the [remote storage](/doc/commands-reference/remote) type you | ||
> plan to use to keep and share your data you might need to specify one of the | ||
> optional dependencies: `s3`, `gs`, `azure`, `ssh`. Or `all_remotes` to include | ||
> them all. The command should look like this: `pip install "dvc[s3]"` - it will | ||
> install `boto3` library along with DVC to support AWS S3 storage. This is | ||
> valid for `pip install` option only. Other ways to install DVC already include | ||
> support for all remotes. | ||
> optional dependencies: `s3`, `gs`, `azure`, `ssh` (or `all_remotes` to include | ||
This comment has been minimized.
Sorry, something went wrong.
This comment has been minimized.
Sorry, something went wrong.
jorgeorpinel
Author
Contributor
|
||
> them all) when installing DVC with `pip`. The command should look like this: | ||
> `pip install "dvc[s3]"`. That particular example will include the `boto3` | ||
> library along with the DVC installation in order to support AWS S3 storage. | ||
> Other methods to install DVC already include support for all remotes. | ||
For example, to setup an S3 remote we would use something like (make sure that | ||
`mybucket` exists): | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
is it all_remotes or just all?