Modifying tool.uv.sources between production and development

[iBuitron]

Hi!

this is my pyproject.toml

[project]
name = "quickbio"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "pydantic>=2.9.2",
    "pydantic-settings>=2.5.2",
    "data-management-tools",
    "quickbio-books",
    "quickbio-diversity",
    "pandera>=0.20.4",
    "pydantic-extra-types>=2.9.0",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.uv.sources]
data-management-tools = { path = "../../data_management_tools", editable = true }
quickbio-books = { path = "../quickBio-books", editable = true }
quickbio-diversity = { path = "../quickbio_diversity", editable = true }

There is any way to use

data-management-tools
quickbio-books
quickbio-diversity 

with git urls at the same time as local path.

What I mean is that I want to specify both paths, but for example when I'm working locally I use ../path, but when I use docker I use the git paths.
Is there a way to do that?

[zanieb]

Are you developing on a different platform than your Docker container? Like developing on macOS but using Linux for Docker?

[iBuitron]

No, all in Ubuntu.

I have a script that changes dependencies when I'm on one branch or another, for example:
If I'm on the dev branch, it uses the local path (in the pyproject.toml)
If I run the script (specifying the "main" branch) it overwrites the pyproject.tom changing the local paths to git paths.

What I want to know is if there is a way to avoid using the script I have, and for example, reading an environment variable, and thus using a group of dependencies (Git) over others (Local)

dev branch:

data-management-tools = { path = "../../data_management_tools", editable = true }
quickbio-books = { path = "../quickBio-books", editable = true }
quickbio-diversity = { path = "../quickbio_diversity", editable = true }

run script (and change to main):

data-management-tools = { git= "GITPATH" }
quickbio-books = { git= "GITPATH" }
quickbio-diversity = { git= "GITPATH" }

[charliermarsh]

You might be able to do this with --no-sources? Like:

[project]
name = "quickbio"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "data-management-tools @ git+ GITPATH",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.uv.sources]
data-management-tools = { path = "../../data_management_tools", editable = true }

Then uv sync would install as editable, but uv sync --no-sources should install from Git.

[iBuitron]

@charliermarsh hi!
Doesn't work :/

[project]
name = "quickbio-books"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "dash>=2.18.1",
    "httpx>=0.27.2",
    "odmantic>=1.0.2",
    "pydantic>=2.9.2",
    "pydantic-settings>=2.5.2",
    "xmltodict>=0.13.0",
    "pandas>=2.2.3",
    "tabula>=1.0.5",
    "pyreadr>=0.5.2",
    "pyarrow>=17.0.0",
    "dev-tools @ git+https://github.com/iBuitron/dev_tools.git",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.metadata]
allow-direct-references = true

[tool.uv.sources]
dev-tools = { path = "../../dev_tools", editable = true }

reveur@italobuitron:~/python/environmental/quickbio_books$ uv sync
error: Failed to build: `quickbio-books @ file:///home/reveur/python/environmental/quickbio_books`
  Caused by: Failed to parse entry for: `dev-tools`
  Caused by: Can't combine URLs from both `project.dependencies` and `tool.uv.sources`

uv sync --no-sources Works

[charliermarsh]

\cc @konstin -- do you think we should allow this? I kind of do.

[iBuitron]

For example.
Right now I'm using my own libraries that I use locally and that I change/evolve according to my needs so as not to have all my code coupled.

quickbio_book depends on dev_tools
quickbio depends on both quickbio_book and dev_tools
and so on, all locally, all in --editable mode.

but right now the limitation I have is that if I wanted to test it in docker, I would have to manually change all the local paths to the github repositories, and once tested, if I wanted to change something I would have to change it back to the local paths and then (again) change it to github to test it again in docker (so, for each of my libraries).

I do all of this with a script, which is responsible for making all the changes and even uploading them to github, but I was wondering if there is a way for UV to take care of it.

Maybe by creating a group where the libraries are specified in editable mode (dev) and in production mode, or by allowing you to read an environment variable with which UV allows you to use one over the other (like the marker platform).

[project]
dependencies = [
  "httpx",
]


[tool.uv.sources]
httpx = [
  { path = "../.../httpx",  custom_env_var= "dev" },
  { git = "https://github.com/encode/httpx", tag = "0.24.1", custom_env_var = "prod" },
]

If I set the environment variable "custom_env_var" == "dev" it would use the local.
If I set the environment variable "custom_env_var" == "prod" it would use the git path.

Of course, this is thought from my point of view in the development of my application (I am not a programmer as such, I am a Biologist who learned to program, maybe I am missing some more efficient way of doing it according to good practices) but it is what I have seen that has limited me and that perhaps UV can handle it.

Thank you for your time :)

[konstin]

do you think we should allow this? I kind of do.

From a technical perspective, yes we can support this.

Maybe by creating a group where the libraries are specified in editable mode (dev) and in production mode, or by allowing you to read an environment variable with which UV allows you to use one over the other (like the marker platform).

I'm trying to understand how the path/git workflow works in practice, specifically keeping things in sync. E.g., how are you making sure that quickbio and data-management-tools are in sync when deploying? Let's say you add a function foo to data-management-tools in the repo, and in the same commit start using it in quickbio. When deploying to docker, you could have an older git version of data-management-tools without foo, while the path version of quickbio expects foo.

[iBuitron]

First I will explain the way I do it currently, which is a bit complicated since there are several steps (although I have it automated and it works well for me)

What I do is.

v.1.1.1
- branch "dev"
quickbio ( localpath --editable)
dmt (localpath --editable)

• commit to github

I have to create another branch called "prod" where I merge the "dev" branch to change the dependency paths here

v.1.1.1
- branch "prod"
quickbio (gitpath)
dmt (gitpath)

• commit to github

So far, I have two branches, "dev" and "prod" both, with the same code, the only difference is the path of each dependency.

In docker, I use the "prod" branch so that the dependencies are installed from github using the gitpath

Now, we added a new feature

v.1.1.2
- branch "dev"
quickbio ( localpath --editable)
dmt with FOO (localpath --editable)

• commit to github (again)

v.1.1.2
- branch "prod"
quickbio (gitpath)
dmt with FOO (gitpath)

• commit to github (again)

So, in docker, the dependencies would be installed from github since the gitpath is still the same (and is updated with the latest commit)

Now, how do I see that this is solved?

I specify the paths for the same dependency

dmt = [
{ path = "../../dmt, editable = true, custom_env_var="dev"},
{ git = "https://github.com/iBuitron/dmt.git", custom_env_var="prod"}
]

Note: The branch should be specified to avoid conflicts

By default, the custom_env_var would be "dev", but in docker, if I wanted to use the git path, I would set custom_env_var = "prod"

[zanieb]

I also tried to do this in an example for someone today

[project]
name = "example"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "httpx @ git+https://github.com/encode/httpx",
]

[tool.uv.sources]
httpx = { path = "../../httpx", editable = true }

There was a discussion in Discord in which someone wants to use a workspace source during development but a built wheel path in production. We don't have a dev / prod toggle, you'd have to hack extra to do this?

[choucavalier]

I second the need for this functionality. Having some libraries as editable during development and correctly packaging them from git sources for CI/CD pipelines would be great. Right now there's no solution for this.

[charliermarsh]

Another example here would be: using a workspace = true source in development, then a path source to a specific wheel in production. See: #9675.

[charliermarsh]

It doesn't solve the broader "prod vs. dev" problem, but I think we should allow conflicting URLs between project.dependencies and tool.uv.sources. It's legitimately useful and semantically reasonable (and costs us nothing).