Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support install directly from git repo #1162

Merged
merged 1 commit into from
Jun 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,204 changes: 1,130 additions & 74 deletions Cargo.lock

Large diffs are not rendered by default.

7 changes: 6 additions & 1 deletion crates/bin/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@ tracing-subscriber = { version = "0.3.17", features = ["fmt", "json", "ansi"], d
embed-resource = "2.1.1"

[features]
default = ["static", "rustls", "trust-dns", "fancy-no-backtrace", "zstd-thin"]
default = ["static", "rustls", "trust-dns", "fancy-no-backtrace", "zstd-thin", "git"]

git = ["binstalk/git"]

mimalloc = ["dep:mimalloc"]

Expand All @@ -74,3 +76,6 @@ log_max_level_debug = ["log/max_level_debug", "tracing/max_level_debug", "log_re

log_release_max_level_info = ["log/release_max_level_info", "tracing/release_max_level_info"]
log_release_max_level_debug = ["log/release_max_level_debug", "tracing/release_max_level_debug"]

[package.metadata.docs.rs]
rustdoc-args = ["--cfg", "docsrs"]
39 changes: 37 additions & 2 deletions crates/bin/src/args.rs
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ pub struct Args {
/// install. The version syntax is as with the --version option.
///
/// When multiple names are provided, the --version option and override option
/// `manifest_path` is unavailable due to ambiguity.
/// `--manifest-path` and `--git` are unavailable due to ambiguity.
///
/// If duplicate names are provided, the last one (and their version requirement)
/// is kept.
Expand Down Expand Up @@ -88,9 +88,21 @@ pub struct Args {
/// This skips searching crates.io for a manifest and uses the specified path directly, useful
/// for debugging and when adding Binstall support. This may be either the path to the folder
/// containing a Cargo.toml file, or the Cargo.toml file itself.
///
/// This option cannot be used with `--git`.
passcod marked this conversation as resolved.
Show resolved Hide resolved
#[clap(help_heading = "Overrides", long)]
pub manifest_path: Option<PathBuf>,

#[cfg(feature = "git")]
/// Override how to fetch Cargo.toml package manifest.
///
/// This skip searching crates.io and instead clone the repository specified and
/// runs as if `--manifest-path $cloned_repo` is passed to binstall.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be for a future improvement, but would be useful to consider supporting some syntax or options for:

  • Branches/refs other than default
  • Subpaths in the repo

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can support that by using fragment, e.g.

https://github.com/cargo-bins/cargo-binstall#branch=...#subpath=...

or maybe using query:

https://github.com/cargo-bins/cargo-binstall?branch=...?subpath=...

Not sure which one is more compatible with git url, I need more advice on this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think doing whatever cargo and/or npm does is a good default?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah and that will be compatible with cargo-install:

       --git url
           Git URL to install the specified crate from.

       --branch branch
           Branch to use when installing from git.

       --tag tag
           Tag to use when installing from git.

       --rev sha
           Specific commit to use when installing from git.

so we would have to at least support --branch/--tag/--rev just like cargo-install.

I think cargo-install finds the Cargo.toml by the package name, so we can also do that in cargo-binstall.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Byron I tried searching for how to checkout branch in gitoxide but only found this GitoxideLabs/gitoxide#358 which uses low-level APIs.

I also checked gix::Repository but I also failed to find one, seems like only way is to dive into the low-level APIs of gix to do this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, indeed, the combination of choosing a refspec that fetches the desired branch and checking out its tree afterwards is how --branch would be implemented. The question of how to deal with the implementation's limitations would still have to be answered though.

Though I didn't know there's a filter in gix.

I don't know what filter means in this context. Feel free to ignore if everything is clear to you though.

I might need to take a look at PRs linked with rust-lang/cargo#11813 to see how they accomplish this.

cargo still performs checkouts using git2, and it's what I will be working to change from now on.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, indeed, the combination of choosing a refspec that fetches the desired branch and checking out its tree afterwards is how --branch would be implemented.

Thanks for confirmation of this!

This goes by the big caveat that filters aren't applied and that it will panic on submodules (not implemented) - checking things out is generally not ready yet even though it works well enough for some cases.

Well that's unfortunate.
Our goal here is to have the same behavior as cargo-install when --git is passed.
I'm not sure whether it clones submodules though.

cargo still performs checkouts using git2, and it's what I will be working to change from now on.

Thanks for the info, I will keep track of the adoption of gitoxide in cargo

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried

                prepare_fetch =
                    set_fetch_refspec(prepare_fetch, format!("+refs/heads/main:refs/remotes/origin/{branch}"));

fn set_fetch_refspec(prepare_fetch: clone::PrepareFetch, refspec: String) -> clone::PrepareFetch {
    prepare_fetch.configure_remote(move |mut remote| {
        remote.replace_refspecs([refspec.as_bytes()], remote::Direction::Fetch)?;
        Ok(remote)
    })
}

but somehow it still clones main.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be too hard to get everything done in one PR, while gix doesn't support checking out specific branch/tag/rev at clone.

I will open a new tracking issue for this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened tracking issue #1165 for this.

///
/// This option cannot be used with `--manifest-path`.
#[clap(help_heading = "Overrides", long)]
pub git: Option<binstalk::helpers::git::GitUrl>,

/// Override Cargo.toml package manifest bin-dir.
#[clap(help_heading = "Overrides", long)]
pub bin_dir: Option<String>,
Expand Down Expand Up @@ -391,13 +403,36 @@ pub fn parse() -> Args {
// Ensure no conflict
let mut command = Args::command();

#[cfg(feature = "git")]
if opts.manifest_path.is_some() && opts.git.is_some() {
command
.error(
ErrorKind::ArgumentConflict,
format_args!(
r#"Multiple override options for Cargo.toml fetching.
You cannot use --manifest-path and --git. Do one or the other."#
),
)
.exit();
}

if opts.crate_names.len() > 1 {
let option = if opts.version_req.is_some() {
"version"
} else if opts.manifest_path.is_some() {
"manifest-path"
} else {
""
#[cfg(not(feature = "git"))]
{
""
}

#[cfg(feature = "git")]
if opts.git.is_some() {
"git"
} else {
""
}
};

if !option.is_empty() {
Expand Down
17 changes: 13 additions & 4 deletions crates/bin/src/entry.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ use binstalk::{
ops::{
self,
resolve::{CrateName, Resolution, ResolutionFetch, VersionReqExt},
Resolver,
CargoTomlFetchOverride, Options, Resolver,
},
};
use binstalk_manifests::cargo_toml_binstall::PkgOverride;
Expand Down Expand Up @@ -95,7 +95,7 @@ pub fn install_crates(
let gh_api_client = GhApiClient::new(client.clone(), args.github_token);

// Create binstall_opts
let binstall_opts = Arc::new(ops::Options {
let binstall_opts = Arc::new(Options {
no_symlinks: args.no_symlinks,
dry_run: args.dry_run,
force: args.force,
Expand All @@ -104,7 +104,16 @@ pub fn install_crates(
no_track: args.no_track,

version_req: args.version_req,
manifest_path: args.manifest_path,
#[cfg(feature = "git")]
cargo_toml_fetch_override: match (args.manifest_path, args.git) {
(Some(manifest_path), None) => Some(CargoTomlFetchOverride::Path(manifest_path)),
(None, Some(git_url)) => Some(CargoTomlFetchOverride::Git(git_url)),
(None, None) => None,
_ => unreachable!("manifest_path and git cannot be specified at the same time"),
},

#[cfg(not(feature = "git"))]
cargo_toml_fetch_override: args.manifest_path.map(CargoTomlFetchOverride::Path),
cli_overrides,

desired_targets,
Expand Down Expand Up @@ -326,7 +335,7 @@ fn do_install_fetches(
resolution_fetchs: Vec<Box<ResolutionFetch>>,
// Take manifests by value to drop the `FileLock`.
manifests: Option<Manifests>,
binstall_opts: &ops::Options,
binstall_opts: &Options,
dry_run: bool,
temp_dir: tempfile::TempDir,
no_cleanup: bool,
Expand Down
2 changes: 2 additions & 0 deletions crates/bin/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
#![cfg_attr(docsrs, feature(doc_auto_cfg))]

pub mod args;
pub mod bin_util;
pub mod entry;
Expand Down
9 changes: 8 additions & 1 deletion crates/binstalk/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ command-group = { version = "2.1.0", features = ["with-tokio"] }
compact_str = { version = "0.7.0", features = ["serde"] }
detect-targets = { version = "0.1.7", path = "../detect-targets" }
either = "1.8.1"
gix = { version = "0.47.0", features = ["blocking-http-transport-reqwest-rust-tls"], optional = true }
glob = "0.3.1"
home = "0.5.5"
itertools = "0.11.0"
jobslot = { version = "0.2.11", features = ["tokio"] }
Expand All @@ -43,7 +45,9 @@ xz2 = "0.1.7"
windows = { version = "0.48.0", features = ["Win32_Storage_FileSystem", "Win32_Foundation"] }

[features]
default = ["static", "rustls"]
default = ["static", "rustls", "git"]

git = ["dep:gix"]

static = ["binstalk-downloader/static"]
pkg-config = ["binstalk-downloader/pkg-config"]
Expand All @@ -57,3 +61,6 @@ trust-dns = ["binstalk-downloader/trust-dns"]

zstd-thin = ["binstalk-downloader/zstd-thin"]
cross-lang-fat-lto = ["binstalk-downloader/cross-lang-fat-lto"]

[package.metadata.docs.rs]
rustdoc-args = ["--cfg", "docsrs"]
22 changes: 22 additions & 0 deletions crates/binstalk/src/errors.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ use thiserror::Error;
use tokio::task;
use tracing::{error, warn};

use crate::helpers::cargo_toml_workspace::LoadManifestFromWSError;

#[derive(Debug, Error)]
#[error("crates.io API error for {crate_name}: {err}")]
pub struct CratesIoApiError {
Expand Down Expand Up @@ -323,6 +325,23 @@ pub enum BinstallError {
#[diagnostic(severity(error), code(binstall::target_triple_parse_error))]
TargetTripleParseError(#[source] Box<TargetTripleParseError>),

/// Failed to shallow clone git repository
///
/// - Code: `binstall::git`
/// - Exit: 98
#[cfg(feature = "git")]
#[error("Failed to shallow clone git repository: {0}")]
#[diagnostic(severity(error), code(binstall::git))]
GitError(#[from] crate::helpers::git::GitError),

/// Failed to load manifest from workspace
///
/// - Code: `binstall::load_manifest_from_workspace`
/// - Exit: 99
#[error(transparent)]
#[diagnostic(severity(error), code(binstall::load_manifest_from_workspace))]
LoadManifestFromWSError(#[from] Box<LoadManifestFromWSError>),

/// A wrapped error providing the context of which crate the error is about.
#[error(transparent)]
#[diagnostic(transparent)]
Expand Down Expand Up @@ -358,6 +377,9 @@ impl BinstallError {
InvalidPkgFmt(..) => 95,
GhApiErr(..) => 96,
TargetTripleParseError(..) => 97,
#[cfg(feature = "git")]
GitError(_) => 98,
LoadManifestFromWSError(_) => 99,
CrateContext(context) => context.err.exit_number(),
};

Expand Down
3 changes: 3 additions & 0 deletions crates/binstalk/src/helpers.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
pub mod cargo_toml_workspace;
pub mod futures_resolver;
#[cfg(feature = "git")]
pub mod git;
pub mod jobserver_client;
pub mod remote;
pub mod signal;
Expand Down
Loading