Simplify end-to-end deployment

[vkuznet]

Impact of the new feature
In order to enable end-to-end deployment procedure developers should be able to setup any WMCore service on their laptop, VM, k8s or any other hardware. Then, with the help of CI/CD pipeline (discussed #10911) they should develop and run their code locally without any deployment procedure (RPM based or otherwise). Finally, they should tag their code and rely on CI/CD pipeline to build a release, run tests, create new image and deploy it on k8s.

To do that, we need to adjust configuration files to avoid hard-coded CMSWEB deployment directory structure and allow developers to use environment variables to define location of auxiliary files for WMCore services. For instance, all config files has the following:

ROOTDIR = __file__.rsplit('/', 3)[0]
...
sys.path.append(path.join(ROOTDIR, 'auth/reqmgr2ms'))
...
sec.key_file = "%s/auth/wmcore-auth/header-auth-key" % ROOTDIR
...
heartbeatMonitor.log_file = '%s/logs/reqmgr2ms/heartbeatMonitor_output-%s-%s.log' % (__file__.rsplit('/', 4)[0],
                                                                                     HOST.split('.', 1)[0],
                                                                                     time.strftime("%Y%m%d"))
...

This implies pre-defined directory structure is required by the configuration file itself, and there is no way to change it in local setup since all paths are bound to location of configuration file. Moreover, plenty of logic required to have specific paths, directories, files to run the service.

Is your feature request related to a problem? Please describe.
I should be able to setup and run any WMCore service on my laptop without enforcing me to define CMSWEB directory structure environment. For example, if I want to run MS service, I should only define few environment variables required for this service, I should be able to choose whatever flat or nested directory structure I want to use, via local or system wise directories, and I should be able to go. For instance, I may to choose my logs to go to /tmp area, and my auth key file to point to my $HOME/secrets/auth-file, etc.

Describe the solution you'd like
Instead of using pre-define directory structure in config files we should identify common set of environment variables and use them. For instance, the example of config structure above I'll change as following:

# instead of
# ROOTDIR = __file__.rsplit('/', 3)[0]
# rely on env variable
ROOTDIR=os.environ.get('WMCORE_ROOTDIR')
if ROOTDIR == "":
   raise Exception("WMCORE_ROOTDIR is not set")
if not os.path.isdir(ROOTDIR):
   raise Exception("WMCORE_ROOTDIR=%s directory does not exist" % ROOTDIR)
...
# this may stay or relaxed to use another WMCORE_AUTHDIR env
# sys.path.append(path.join(ROOTDIR, 'auth/reqmgr2ms'))
auth_path = os.environ.get("WMCORE_AUTHDIR")
if auth_path and os.path.isdir(auth_path):
   sys.path.append(auth_path)
else:
   raise Exception("WMCORE_AUTHDIR is not set")
if not os.path.isdir(auth_path):
   raise Exception("auth_path=%s does not exist" % auth_path)
...
# instead of hard-coded auth key file in pre-defined location
# sec.key_file = "%s/auth/wmcore-auth/header-auth-key" % ROOTDIR
# we may use
key_file = os.getenv("WMCORE_AUTH_KEY")
if not os.path.isfile(key_file):
    raise Exception("key_file=%s does not exist" % key_file)
...
# instead of hard-coded pre-define location
#heartbeatMonitor.log_file = '%s/logs/reqmgr2ms/heartbeatMonitor_output-%s-%s.log' % (__file__.rsplit('/', 4)[0],
#                                                                                     HOST.split('.', 1)[0],
#                                                                                     time.strftime("%Y%m%d"))
# we should use
log_file = os.environ.get("WMCORE_HEARTBEAT_LOGDIR")
if log_dir and os.path.isdir(log_dir):
    heartbeatMonitor.log_file = log_dir
else:
   raise Exception("WMCORE_HEARTBEAT_LOGDIR is not set")
if not os.path.isdir(log_dir):
   raise Exception("log_dir=%s does not exist" % log_dir)
...

Such definitions will support both CMSWEB and user defined environment, for CMSWEB we will define all env variable to point to existing dir structure, while for local environment user may define his/her own locations. As such, the config files will provide universal configuration which will work both on CMSWEB and local environments. Moreover, such flexibility and dependence on env variables will significantly simplify CI/CD pipepine and k8s deployment and will allow to decouple deployment of WMCore services from CMSWEB deployment procedure.

Describe alternatives you've considered
The config files should be adjusted to provide all details of configuration logic required by WMCore services. It can be achieved in a different way. For instance, you provide additional shell setup.sh file which will define all environment variables, etc.

Additional context
This proposal will simplify CI/CD pipeline development and local deployment procedure, see #10911

[vkuznet]

@klannon , @amaltaro , @todor-ivanov , @goughes I consider this very important step towards end-to-end deployment procedure and automation via CI/CD. Please consider it carefully. I tried to provide reason why current configuration significantly constrain our ability to use local setup (and therefore introduced large latencies in dev cycle). For instance, if I need to work on issue A, which only necessary to fix some corner part of WMCore web framework, I should not be forced to install RPMs on VM with cmsweb deployment tools. I should be able to run WMCore service locally, test my feature, tag my change and get my build automatically. I successfully applied such practice in many other services which does not inherit constraints defined by WMCore structure and configuration files. I propose to rethink how configuration files should be structure and first step is to decouple them from CMSWEB pre-defined structure (since this is no longer required within docker image) and will simplify working with local setup.