Automatically add cargo-port URLs to recipes

[daler]

Once a recipe gets archived, it should be updated to reflect the new URL (at least as a backup).

Maybe a separate archive module, which could also check for and handle bioconductor URL degradation.

xref bioconda/bioconda-recipes#1015 (comment)

[epruesse]

The cargo-port URL(s) should be added to the meta.yamls source:url: section(s) automatically.

Rationale:

• Avoid failures if the source has gone away when the recipe is rebuild for pinning updates.
• Meet license requirements (e.g. GPL)

Status:

cargo-port uses its jenkins to do a nightly scan of bioconda-recipes master for new recipes / urls and adds those to its repository.

(incidentally - there is something going wrong there, see galaxyproject/cargo-port#158)

This means that the URLs become live only up to 1 day after a PR is merged.

Options:

We can add the URL(s) to the meta.yaml at various stages:

• before the PR is merged (by the bot)
• while the PR is built and uploaded (by the build system)
• when the URL becomes active (by cargo-port)
• in a cron job on the CI comparing our packages/urls to cargo-port's data

Alternatively, we could just keep the link in the docs (suggestion by @dpryan79).

If we want to be fancy, we can build any other workflow as well. E.g. having a @biocondabot backup source command that has the bot send a webhook to the cargo-port jenkins that in turn processes the upload and sends a webhook back to the bot which then adds the URL to the recipe.

@bgruening Cargo-Port is your baby. You get to having to decide. :)

[daler]

Is there a way to figure out how much the cargo-port URLs are being used by conda-build in practice? I love the idea of archiving the source, but not sure it needs to be in the recipe. If it's used rarely, we could simply have guidance in the docs that if original source is missing, check cargo-port.

[epruesse]

It's rare, but possible. I know of at least one package that was blacklisted because the source had gone away.

[bgruening]

I treat cargo-port as last resort backup and would not expose it in the recipe, the exception is bioconductor where we know packages will disappear. Adding it to the doc and highlighting it as some community feature is my preferred solution.

[epruesse]

What's the CF approach to this? GPL mandates that the source must be made available to all to whom the binaries are made available.

[bgruening]

I don't think they have a solution in place for that. Tarballs can disappear :(

[epruesse]

Tarballs can disappear :(

Or change.... See bioconda/bioconda-recipes#14373 (trimmomatic 0.39 got a fix, but neither URL nor version changed...).