Version syncing: repositories with too many branches/tags fail on syncing

[humitos]

I saw there are repositories that have too many branches/tags failing when we try to re-sync their versions. An example of it is jupyterlab which has 61k versions:

▶ git ls-remote https://github.com/jupyterlab/jupyterlab.git | wc -l
61290

It fails on our side because we truncate the output of the command if it's too big: 'Output is too big. Truncated at 4718592 bytes.'. We could remove this restriction, but that will produce a problem in our database when trying to re-sync 61k versions. I'm not sure what's the right approach to fix this issue 😞

This project in particular has ~1300 versions registered in Read the Docs:

In [1]: p = Project.objects.get(slug="jupyterlab")

In [2]: p.versions.count()
Out[2]: 1321

and they have Skip syncing tags feature flag enabled.

In [6]: list(p.features.all())
Out[6]: 
...
 <Feature: Use remote listing in VCS (e.g. git ls-remote) if supported for sync versions feature>,
 <Feature: Skip syncing tags feature>,
...

Sentry issue: https://sentry.io/organizations/read-the-docs/issues/3058924977/

[aanm]

I think we are facing this exact issue in https://readthedocs.org/projects/cilium. I can't find a tag that was recently created in https://readthedocs.org/projects/cilium/versions. Any workaround in place?

The only workaround found was to delete some upstream non-used references with git push --delete origin refs/heads/branch_name

[stsewd]

Another option is to cut the output only when saving the command.

[humitos]

It may be worth checking if updating 65k versions on each sync won't destroy our API / database first. I think it could be a problem. In that case, we may want to be able to sort the branches by descending date and keep the last 200 created only or similar.

[humitos]

Another option is to cut the output only when saving the command.

In that case, we will be sending a lot of data over the API, which may take some time, and then it will be discarded. I think we could add an argument to the method that runs these particular commands telling them not to chunk the output's command.

[stsewd]

In that case, we will be sending a lot of data over the API

I meant before using the API to save that command.

[dojutsu-user]

Hi @humitos @stsewd,

I can take this up, but not entirely sure on what needs to be done here.

[humitos]

@dojutsu-user I'm not sure what is exactly the solution to this problem. I suggested:

we may want to be able to sort the branches by descending date and keep the last 200 created only or similar

So, a good starting point is to figure it out how to do this with Git and all the other supported VCS (not considered to be deprecated soon, see #8840)