Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce batch updating. #14

Closed
wants to merge 1 commit into from

Conversation

DrewAPicture
Copy link

Note: This is basically a more robust version of what @iandunn proposed over in #13.

Introduces updating in batches. Technically it adds batching to both first-time pulls and subsequent updates, though the idea of the batching the former is kind of pointless since you need all the things as a starting point.

The upside of "batching" the initial pull is that it allows you to see where in the overall scheme of things you are in terms of batches completed out of the total number. Nice to have an indicator there at least.

Regardless, the default number of plugins to update per batch is 1,000, and can be modified via a new argument passed when starting the updater, like so:

./update all 2500

Following the completion of each batch of plugins, the last_revision file is updated and the loop starts over. The obvious benefit of batching is that if you need to interrupt the update for whatever reason, you only lose you place by the number of plugins in a given batch.

I've added some additional text output giving better indication of where in the overall scheme of things you are including which batch out of the total calculated batches you're in currently, as well as integrated the cleanup messaging. I've been using this modified version to stay updated for the last month or so without issues.

Feedback would be appreciated.

@nylen
Copy link
Collaborator

nylen commented Feb 20, 2017

@DrewAPicture have you looked at #12? I'd prefer to do something like that, including parallel requests. Also it might be possible to more intelligently update the current progress in the .last_revision file so that a failed update can restart from anywhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants