Add basic virtual env setup script. #10980
Conversation
|
Jenkins results:
|
deploy/deploy-centralvenv.sh
Outdated
| ### Usage: -v <wmcore_path> WMCore virtual environment target path to be used for this deployment [Default: ./WMCore.venv3] | ||
| ### Usage: -p <patches> List of PR numbers | ||
| ### Usage: (in double quotes and space separated e.g. "5906 5934 5922") | ||
| ### Usage: -s <service_names> List of service names to be patched |
There was a problem hiding this comment.
The above flag is very nice (about patches), but this one is confusing. How will I know which service should be patched? For that I need to know what my patches provide, etc. What if my patch is common for several services and in my list here I'll skip one? I rather prefer to avoid this option and just provide patches which should be applied uniformly to all services.
There was a problem hiding this comment.
it is a legacy, coming from the time (actually still the case) when because of the RPM based deployment every service was having its own area and branch in the deployed WMCore tree. So when it comes to patching a live service it is every time a single patch needs to be applied on all the deployed components (or at least the set of components you know the patch is affecting). Hence the list of services to be patched. And here is a script which I personally use very often when patching a single component [1]. Of course we are talking here about getting rid of slowly getting rid of this structure and pushing it all to run from a single code base and then the list of services to be patched is going to be a redundant option for sure. I do like the idea of running and patching a single code base for both development and runtime. (Although just as a note here - we may have issues if we want multiple users to be running the different services (as it was with cmsweb before), unless we decide to solve that in the K8 deployment scripts - we still have the option to dedicate a set of pods per service so the privileges and authentication related issues may be addressed there).
[1]
https://github.com/dmwm/WMCore/blob/master/bin/patchComponent.sh
There was a problem hiding this comment.
Just a note here @vkuznet .
I did remove the list of services to be patched as you suggested and left only the list of patches to be applied, which is indeed much cleaner, but I still need to implement the actual call to the patching script. Yet to be done.
|
@todor-ivanov , the both echo outputs produced the same results on Linux and MacOS if I run the full path of the script. |
|
yes, Would definitely work for the Full script path but wont work here: I'll think of something. |
|
Jenkins results:
|
|
Jenkins results:
|
|
@todor-ivanov what is the status of this development and can you provide time estimate when I'll be able to try it out? Another question, assuming that I can use your script to setup virtual environment. How easy to move from it to have working WMAgent, or MS? Do we have any procedure in place that after setup we can test and validate it? Here are few examples I have in mind:
If I have all code in place, and running DBs, I would presume that everything can be done through local host communication. How ready configuration files and tests for this? |
|
one month forward (!), @todor-ivanov can we move with this? What is the status? What else should be done here? |
|
Hi @vkuznet I only need to create the proper runtime environment for the services. Which needs little bit of digging [1]. The rest is simply putting it here, which shouldn't be a problem. As we said this was not a planned development, and I am fighting to finalize what was in the the plans for Q1, so I work on this one only when I have some extra time, hence the issue is floating between waiting and work in progress. I will return to this soon. |
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
…r execution in main.
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
…de the current virtual env.
|
Jenkins results:
|
|
Thanks @vkuznet for making another review and all your comments. I think I have addressed more of them. Few of them with some new commits other by comments. If you do not mind, let me also try to shed some more light and clarify few things here as well. As a start let me express my view on the need of this script, which may help. We are in the process of phasing out the
I tend to disagree here. None of the steps, besides the ones related to the virtual environment and deployment tree creation, are mandatory and can easily be skipped during manual deployment.
Yes you can. I did put some extra protections with my last two commits to be 100% sure this script does not touch developer's working area. Upon initial deployment and configuration to run from source you can easily create a soft link at
This I see as supportive to what I have tried to achieve.
This, as I already said in the comments inline, is:
I already did
The ability is already in place. I have added few extra protections on top of that, and mentioned that, as long as the source code is mounted at the place where the deployment is happening, it can be made accessible from inside the venv through a symlink.
Again this is not the case. As long as your git working tree is physically accessible from the place where the deployment has taken place, it is going to work. On the other hand, I can share a hint on what I am doing in my case (hope this can help someone). I am deploying somewhere remote (at any place where my central services will be able to access all external services they depend on - like MongoDB etc.) and then I never login to the machine I have deployed at for code editing. Instead, I work from my development environment at my local machine through emacs like: While at the same time I do have this same source code linked from inside the virtual environment as: This way I do not have delays from ssh lags for remote code development, while using the IDE from my own laptop. And I do have my source wherever I need it, provided the area is physically mountable inside the container or VM. |
There was a problem hiding this comment.
@todor-ivanov instead of reviewing changes I tried to use it. Here is my findings using macOS:
./deploy-centralvenv.sh
./deploy-centralvenv.sh: line 219: md5sum: command not found
./deploy-centralvenv.sh: line 280: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]
=======================================================
Deployment parameters:
-------------------------------------------------------
serviceList : wmcore
enabledList : reqmgr2 reqmgr2-tasks reqmgr2ms-transferor reqmgr2ms-monitor reqmgr2ms-output reqmgr2ms-rulecleaner reqmgr2ms-unmerged workqueue reqmon reqmon-tasks t0_reqmon t0_reqmon-tasks
venvPath : ./WMCore.venv3
wmCfgRepo : https://:@gitlab.cern.ch:8443/cmsweb-k8s/services_config.git
wmCfgBranch : test
wmSrcRepo : https://github.com/dmwm/WMCore.git
wmSrcBranch : master
wmTag :
serPatch :
pypi Index : prod
runFromSource : false
Cleanup Virtual Env : true
verboseMode : false
assumeYes : false
central services host:
secSring :
=======================================================
Continue? [y]:
...
You still have 5 sec. to cancel before we proceed.
=======================================================
Creating minimal virtual environment:
Continue? [y]:
...
=======================================================
Activate WMCore virtual env:
Continue? [y]:
...
Collecting wheel
Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.37.1
Requirement already satisfied: pip in ./lib/python3.9/site-packages (22.0.4)
=======================================================
Setup WMCore paths inside the virtual env:
Continue? [y]:
...
./deploy-centralvenv.sh: line 289: /Users/vk/tmp/WMCore/WMCore.venv3/bin:/opt/anaconda3/bin:/Users/vk/.cargo/bin:/opt/local/bin:/opt/local/sbin:/Users/vk/bin:/Users/vk/perl5/bin:/Users/vk/bin/OSX/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/vk/Work/Languages/Go/gopath/bin:/opt/local/Library/Frameworks/Python.framework/Versions/Current/bin:/opt/local/share/java/hadoop-1.1.1/bin:/Users/vk/.fzf/bin: syntax error: operand expected (error token is "/Users/vk/tmp/WMCore/WMCore.venv3/bin:/opt/anaconda3/bin:/Users/vk/.cargo/bin:/opt/local/bin:/opt/local/sbin:/Users/vk/bin:/Users/vk/perl5/bin:/Users/vk/bin/OSX/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/vk/Work/Languages/Go/gopath/bin:/opt/local/Library/Frameworks/Python.framework/Versions/Current/bin:/opt/local/share/java/hadoop-1.1.1/bin:/Users/vk/.fzf/bin")
Even though it fails with something here is my attempt to use it
# and now if I list my local area I see the following
ls
WMCore.venv3 deploy-centralvenv.sh
ls WMCore.venv3
WMCore.venv3 bin include lib pyvenv.cfg
# and it seems to be non-active
type python
python is /opt/local/bin/python
# env | grep PYTHON returns nothing too
Since I know python venv I tried the following
ls WMCore.venv3/bin
Activate.ps1 activate.csh pip pip3.9 python3 wheel
activate activate.fish pip3 python python3.9
# let's activate it
source WMCore.venv3/bin/activate
and I see that env | grep PYTHON is still empty but python location has changed:
type python
python is /Users/vk/tmp/WMCore/WMCore.venv3/bin/python
and since there is not PYTHONPATH I can't load WMCore
python
Python 3.9.12 (main, Mar 25 2022, 04:56:27)
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import WMCore
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'WMCore'
From this trial I think you should do the following:
- fix installation issues with
md5sumanddeclare - provide summary after installation of what is installed and where
- setup proper environment or print instructions how to do that
- provide dump of examples, and either abort if user didn't provide required flags or dump example of what user can do when installation is completed with specific flags.
…e install packages step not mandatory.
|
Jenkins results:
|
|
Thank you very much @vkuznet for testing it. This is all indeed useful info.
The problem with the This old bash version seems to be a common issue for OS X systems. See a proposed fix discussed at stackoverflow here[1], this comment in particular explains exactly what one can expect for MacOS and a solution proposed in this one. I am adding a required minimal bash version for the script to run with my latest commit. As of
It is already there. If your run was not to fail so dramatically you were supposed to see the following output as a summary of what has been set and where: I will work on making it more readable, though. Valentin Thanks for testing it. Please give it a try again and write back your findings. I am also interested to see what the outcome will be. Since I am not able to run it under MacOS I will rely on your feedback in order to fix issues related to porting it to MacOS. [1] |
|
@todor-ivanov , you are wrong about my shell: So, take this into account. I'm using zsh and it is up to date (aligned with master HEAD :) Since your script uses is also up to date: So, don't blame me :) You should change to to fix the issue. |
|
Jenkins results:
|
|
Jenkins results:
|
|
Jenkins results:
|
|
Hi @vkuznet with the latest commit I did fix the printout of the final environment variables to be more human readable. Here is an example of what it looks like in one of my setups. I hope this is good enough. Please give it a try and let me know. |
There was a problem hiding this comment.
Todor, macOS does not have md5sum tool, therefore you need to adjust your script to manage md5 in a different way. Quick google search reveals that on maxOS we have md5 tool which seems to me equivalent.
Bottom line, here is what I got:
./deploy-centralvenv.sh
./deploy-centralvenv.sh: line 219: md5sum: command not found
The current setup script requires: md5sum in order to continue. Please install it and rerun.
Interrupt execution due to step failure:
ERRORNO: 1
Now, regarding compatibility:
- On Linux we have
echo "Todor" | md5sum
91e53b1970dc62a271b70d24ab3c66e5 -
- on macOS we have
echo "Todor" | md5
91e53b1970dc62a271b70d24ab3c66e5
There was a problem hiding this comment.
Todor, I finally managed to have a first look at this script, sorry for the delay.
It was hard to "parse" everything and closer to the end I didn't pay too much attention. Some general comments are:
- maybe we don't need to ask for input confirmation for actions that are not really changing something in someone's machine (like git clones).
- I think Valentin mentioned it in the past, but I do think it would be extremely helpful (and more sustainable) to have a small set of scripts. For instance, for CMSWEB, we have
prep,swandpost, each in charge of different things. Here, we could easily have a script that does the initial virtual environment setup + python package requirements; separated from the actual WMCore services deployment; and maybe even separated from the WMCore service execution. - I also see you integrated some service "manage" actions in this script, replacing the standard manage scripts we already have in the services_config/deployment repositories. Is this really necessary or is it worth it the extra complexity on our side?
| $verboseMode && set -x | ||
|
|
||
| # Swap noVenvCleanup flag with venvCleanup to avoid double negation and confusion: | ||
| venvCleanup=true && $noVenvCleanup && venvCleanup=false |
There was a problem hiding this comment.
maybe refactor it to venvCleanup from the very beginning (as a command line arg)?
deploy/deploy-centralvenv.sh
Outdated
| # so the `pythonCmd' variable shouldn't be needed any more. | ||
|
|
||
| pythonCmd=python | ||
| [[ $(python -V 2>&1) =~ Python\ *2.* ]] && pythonCmd=python3 |
There was a problem hiding this comment.
It might be useful to print which python version we are using for the initial virtual env setup.
| enabledList="" | ||
| for service in $_enabledListTmp | ||
| do | ||
| # First cut all pypi packaging version suffixes |
There was a problem hiding this comment.
At this stage, didn't we only provide the service name? Reading the code, I understand it parses it as if it was a line from the requirements.txt file (thus service + logical operator + version in pypi).
|
|
||
| pipOpt="--no-cache-dir" | ||
| [[ $pipIndex == "test" ]] && { | ||
| pipOpt="$pipOpt --index-url $pipIndexTestUrl --extra-index $pipIndexProdUrl" ;} |
There was a problem hiding this comment.
What is the effect of the --extra-index option here? Can't it use only what is defined in the --index-url?
| -d <wmcore_path> WMCore virtual environment target path to be used for this deployment [Default: ./WMCore.venv3] | ||
| -p <patches> List of PR numbers [Default: None] | ||
| (in double quotes and space separated e.g. "5906 5934 5922") | ||
| -m <security string> The security string to be used during deployment. Will be needed at startup [Default: ""] |
There was a problem hiding this comment.
I sort of have an idea of what this security string is, but what is the goal here?
We do have it in CMSWEB services (legacy implementation from +10 years ago) and the result is that people simply copy/paste the "I did read ..." security message. In other words, to me it's more like a complication than a security step.
| # 100 - Success - skip step, consider it recoverable | ||
| # 101 - Success - skip step based on user choice | ||
| # 102 - Failure - interrupt execution based on user choice | ||
| # 1-255 - Failure - interrupt all posix return codes |
There was a problem hiding this comment.
I assume it to be a range, right? In that case, this range also cover the exit codes above, which you define as being success :-D
| echo -n "Should we try to reinstall them while releasing version constraint? [y]: " | ||
| $assumeYes || read x && [[ $x =~ (n|no|nO|N|No|NO) ]] && return 101 | ||
| echo "..." | ||
| echo "Retrying to satisfy dependency releasing version constraint:" |
There was a problem hiding this comment.
I guess I failed to understand the logic for resolving dependencies issues. Could you please clarify it?
|
Jenkins results:
|
… full pkg dependencies only when running from source.
|
Jenkins results:
|
|
Can one of the admins verify this patch? |
Fixes #10979
Status
IN TESTING
Description
With the current script a proper virtual environment is setup for the WMCore central services. Together with satisfying all the package dependencies inside it based on the
requirements_py3.txtcoming with the WMCore code base.NOTE: This is still a work in progress and it is meant to also provide a fully configured and properly linked runtime environment .
Is it backward compatible (if not, which system it affects?)
NO
Related PRs
None
External dependencies / deployment changes
In general this should depend on the service configuration repository:
https://gitlab.cern.ch/cmsweb-k8s/services_config/