Not able to specify python_binary for agent in clearml.conf #523
Comments
|
Hi @nieag , What is the clearml-agent version you're using? Can you verify this also happens with the latest 1.2.0rc0? |
|
Running |
|
Upgrading to clearml-agent==1.2.0rc0 doesn't seem to help, the agent config is set as agent.python_binary = /usr/bin/python3.8 However, the executables that are being used are Manually starting either of these I can verify that both are python version 2.7.17 And the following error trace is returned |
|
Hi @nieag , Sorry to taking so long to answer 🙁 Are you running the CleaRML Agent using docker mode (i.e. |
|
Hi, no worries. I'm running in standard venv mode, without the docker flag. |
|
I'm also running the clearml-agent from a venv I created manually, don't know if that might affect things? Wasn't able to set the agents' python-binary to the one existing in the venv, nor did it seem to pick the one that I called the "clearml-agent daemon --queue default --foreground" command with. Should be the case according to lines 109-113, file session.py, in the clearml-agent repo I think, # HACK make sure we have python version to execute,
# if nothing was specific, use the one that runs us
def_python = ConfigValue(self.config, "agent.default_python")
if not def_python.get():
def_python.set("{version.major}.{version.minor}".format(version=sys.version_info)) |
|
@nieag that's strange, it looks to me like there some kind of configuration mixup - any chance |
|
Hey, VIRTUAL_ENV=/home/company/nieage/clearml_venv Which is the venv where I installed the clearml-agent and am currently running from. It doesn't use that path for the python-binary however. |
|
Did you set it up manually? |
|
No, just created the venv normally with python, activated it and installed the clearml-agent. I assume that variable is set by the venv activation script. |
|
Can you try running the agent without the activation script using |
|
Tried doing that, gives me the same error as before. Added the log file below: |
|
Well, the agent will always use one of the system-installed python binaries to execute the task (i.e. to create the venv for the task) and never the venv binary it was executed from. However, I'm not sure why you see this: |
|
Perhaps you can try to clean up the |
|
No that doesn't seem to affect it. However, I tried running "python3.8 -m virtualenv " this gives me the same python2 error, so this is most likely not due to clearml. Seemed to maybe be caused by how virtualenv is installed which can affect the python version it defaults to. Will try and investigate a bit on my end, and update this thread if I find something! Thanks for the help so far! |
|
I'm having a similar issue with
$ which python
#~/.pyenv/shims/python$ python --version
#Python 3.9.10
python_binary: "/home/acalabourdin/.pyenv/shims/python"^a valid path, returns the package_manager: {
# supported options: pip, conda, poetry
# type: pip,
type: poetry,
python = "~3.9"Upon running clearml-agent daemon --queue default --foreground from system (i.e. not from any python virtual env.) I get the following: Poetry Enabled: Ignoring requested python packages, using repository poetry lock file!
The currently activated Python version 3.7.10 is not supported by the project (~3.9).
Trying to find and use a compatible version.
Using python3 (3.9.10)
...
Installing dependencies from lock file
SolverProblemError
The current project's Python requirement (3.7.10) is not compatible with some of the required packages Python requirement:
- numpy requires Python >=3.8, so it will not be satisfied for Python 3.7.10
Because numpy (1.22.1) requires Python >=3.8
and no versions of numpy match >=1.20.0,<1.22.1 || >1.22.1, numpy is forbidden.
So, because no versions of numpy match <1.20.0
and ds-614 depends on numpy (*), version solving failed.
at ~/.local/share/pypoetry/venv/lib64/python3.7/site-packages/poetry/puzzle/solver.py:241 in _solve
237│ packages = result.packages
238│ except OverrideNeeded as e:
239│ return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
240│ except SolveFailure as e:
→ 241│ raise SolverProblemError(e)
242│
243│ results = dict(
244│ depth_first_search(
245│ PackageNode(self._package, packages), aggregate_package_nodes
The |
You should probably also set ignore_requested_python_version: true Can you verify the issue still exists in the latest RC (I think there was a fix to a similar issue) |
|
HI @bmartinn I should report that the problem still persists, with clearml==1.8.0, poetry==1.2.2, pyenv==2.3.5. Even with ignore_requested_python_version: true ClearML agent keeps clinging to the wrong python and it's really blocking my work on ClearML stack UPD: and this definitely doesn't happen I even use super explicit commands like: I need at least a workaround |
|
Hi @ColdTeapot273K, I think what @bmartinn meant was you need the latest clearml-agent version... (try v1.5.0rc0) |
Hi!
Been trying to setup an clearml-agent on a linux server. Running into issues when the agent picks a task from a queue and attempts to create a venv on the server. On the server there's multiple python versions available using update-alternatives to manage the various sym-links. As per default the agent picks a python 2.7 version on /usr/bin/python2, maybe due to /usr/bin/python pointing at /etc/alternatives/python?
I tried specifying a different version in the clearml.conf file using python_binary: "/usr/bin/python3.8" but this doesn't seem to have any effect.
Any help would be appreciated!
The text was updated successfully, but these errors were encountered: