blog: remote optimization post #1451
Conversation
|
Not sure if the initial draft is too in depth/technical. @andronovhopf I'd appreciate it if you can take a look at this and give some suggestions on how to make it more interesting/applicable for users from an ML perspective |
| @@ -0,0 +1,174 @@ | |||
| --- | |||
| title: Optimizing DVC Remotes | |||
| date: 2020-06-29 | |||
There was a problem hiding this comment.
I think this can be whatever you prefer.
| date: 2020-06-29 | ||
| description: | | ||
| An overview of how syncing data to and from remote storage is optimized in DVC. | ||
| picture: 2020-05-04/owl.png |
There was a problem hiding this comment.
I have the impression if you leave it blank it uses a default img BTW.
There was a problem hiding this comment.
I was having issues running the dev server (via yarn develop) when picture was unset, maybe that's just some problem with my local environment though?
There was a problem hiding this comment.
TBH I'm not sure exactly how the blog engine works! You can create a bug report though and Ivan or Roger will probably answer to that 🙂
| @@ -0,0 +1,174 @@ | |||
| --- | |||
| title: Optimizing DVC Remotes | |||
There was a problem hiding this comment.
probably needs a more interesting title
There was a problem hiding this comment.
from the intro think this post is more about "Optimization improvements in DVC 1.0"
skshetry
added
A: docs
| author: peter_rowlands | ||
| --- | ||
|
|
||
| One of the key features provided by DVC is the ability to efficiently sync | ||
| versioned datasets between a user's local machine and | ||
| [remote storage](https://dvc.org/doc/command-reference/remote), and version 1.0 | ||
| includes several performance optimizations related to syncing data with remotes. |
There was a problem hiding this comment.
I would start if possible with something like "Our users have presented the need for optimizing remotes blah blah" and give some examples e.g. Discord message screenshots.
There was a problem hiding this comment.
Also, minor: I personally prefer "synchronizing" or "syncing". The pronunciation of the latter is questionable, no?
| author: peter_rowlands | ||
| --- | ||
|
|
||
| One of the key features provided by DVC is the ability to efficiently sync | ||
| versioned datasets between a user's local machine and | ||
| [remote storage](https://dvc.org/doc/command-reference/remote), and version 1.0 | ||
| includes several performance optimizations related to syncing data with remotes. |
There was a problem hiding this comment.
Also, minor: I personally prefer "synchronizing" or "syncing". The pronunciation of the latter is questionable, no?
| 3. Determine the difference between the two sets of files | ||
|
|
||
| Commonly used cloud sync utilities, such as [rclone](https://rclone.org/), must | ||
| be generalized to support any arbitrary file structure, which can come at the |
There was a problem hiding this comment.
| be generalized to support any arbitrary file structure, which can come at the | |
| be generalized to support any file structure, which can come at the |
| operations (i.e. `status -c`, | ||
| [push](https://dvc.org/doc/command-reference/push), | ||
| [pull](https://dvc.org/doc/command-reference/pull), | ||
| [fetch](https://dvc.org/doc/command-reference/fetch)). In DVC version 1.0, these |
There was a problem hiding this comment.
| operations (i.e. `status -c`, | |
| [push](https://dvc.org/doc/command-reference/push), | |
| [pull](https://dvc.org/doc/command-reference/pull), | |
| [fetch](https://dvc.org/doc/command-reference/fetch)). In DVC version 1.0, these | |
| operations (i.e. `dvc status -c`, | |
| `dvc push`, | |
| `dvc pull`, | |
| `dvc fetch`). In DVC version 1.0, these |
|
@pmrowla very nice! Please note on this repo we don't mind if you push a branch directly to upstream, in fact that's usually better because it fires up a review app automatically. I created one manually for this PR, you can see your post here: https://dvc-landing-blog-remote-uhiudf.herokuapp.com/blog/optimizing-dvc-remotes Cheers |
❗ Please read the guidelines in the Contributing to the Documentation list if you make any substantial changes to the documentation or JS engine.
🐛 Please make sure to mention
Fix #issue(if applicable) in the description of the PR. This causes GitHub to close it automatically when the PR is merged.Please choose to allow us to edit your branch when creating the PR.
Thank you for the contribution - we'll try to review it as soon as possible. 🙏
Initial draft for the remote optimization write up
TODO