New spec for python3 gfal2

[amaltaro]

Fixes dmwm/WMCore#10828

This PR has some requirements for the build node (and also for the node/container running MSUnmerged), given that we do not build all the dependencies brought up with gfal2-python. We need to install/enable this DMC EL7 Production repository:

sudo yum-config-manager --add-repo https://dmc-repo.web.cern.ch/dmc-repo/dmc-el7.repo
sudo yum-config-manager --enable dmc-el7/x86_64

and on the build and service-host node, we need to install these 4 packages (glib2 is likely already available):

yum install glib2 glib2-devel gfal2 gfal2-devel

note that the gfal2 and gfal2-devel packages must have the version 2.20.0-1.el7.cern or higher.

Then, we also have to install the gfal2 plugins on the service-host node (container). This one is mandatory:

yum install gfal2-plugin-gridftp gfal2-plugin-file gfal2-plugin-http gfal2-plugin-srm gfal2-plugin-xrootd

but right now, this is the set of packages+plugins that we are deploying in the ms-unmerged image:
https://github.com/dmwm/CMSKubernetes/blob/master/docker/reqmgr2ms-unmerged/Dockerfile#L13-L21

Reference, discussion with gfal2 experts: https://cern.service-now.com/service-portal?id=ticket&table=u_request_fulfillment&n=RQF1903380

[amaltaro]

And I created this PR more to see if @smuzaffar could help us building this spec :-D
I got stuck with this error (and actually there is another boost error before this):

Python 3 installed! 3.8
Boost Python3 library not found
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.27.1") 
-- Checking for module 'glib-2.0'
--   No package 'glib-2.0' found
CMake Error at /build/amaltaro/w/slc7_amd64_gcc630/external/cmake/3.7.0-comp3/share/cmake-3.7/Modules/FindPackageHandleStandardArgs.cmake:138 (message):
  Could NOT find GLIB2 (missing: GLIB2_LIBRARIES GLIB2_INCLUDE_DIRS)

Shahzad, if you could spare some time looking into this, the build log can be found at:
amaltaro@vocms055:/build/amaltaro $ less /build/amaltaro/w/BUILD/slc7_amd64_gcc630/external/py3-gfal2/1.10.1/log

[muhammadimranfarooqi]

to 05b7e34 on slc7_amd64_gcc630.
Build log can be found here.

[smuzaffar]

@amaltaro , looks like it requires gfal2 and glib devel packages. Try to install these on the build machine e.g.

yum install  glib2-devel gfal2 gfal2-devel

also it depens on boost so add

export CMAKE_PREFIX_PATH=${BOOST175PY3_ROOT}

in the %build section so that cmake can find your boost installation.

[amaltaro]

Thanks for looking into this, Shahzad.

Here is my silly question, if I install these dependencies on the build node, shouldn't I also install it on every node that will be running the service that depends on this package?

[smuzaffar]

yes those should be installed on service node too. I think you do not need -devel packages on service nodes ( just the library should be enough)

[smuzaffar]

if servcies are running in a container then just make sure that gfal2 and glib2 are installed in container

[amaltaro]

Shahzad, just to avoid possible confusions, what you suggest is to deploy these packages in the build node (do we also need glib2?):

yum install  glib2-devel gfal2 gfal2-devel

and on the node running this service - we use docker images/container for production - we only need to install:

yum install  glib2 gfal2

is that correct?

[smuzaffar]

yum install  glib2 gfal2

yes that is correct for nodes running the service

[muhammadimranfarooqi]

-1 to 05b7e34 on slc7_amd64_gcc630.
Build log can be found here.

[muhammadimranfarooqi]

-1 to 3fd7e91 on slc7_amd64_gcc630.
Build log can be found here.

[amaltaro]

@smuzaffar Shahzad, thanks for your prompt review. I'm going to look at your comments and update the code soon.

First, I just wanted to see whether this build would indeed fail, since we do not have all the required packages installed in vocms055 AND they are not in their latest version (which has quite some critical bugfixes!).

From the GFAL2 documentation: https://dmc-docs.web.cern.ch/dmc-docs/gfal2-python/pip-install.html
we need to install/enable the EL7 DMC Production repository, something like:

sudo yum-config-manager --add-repo https://dmc-repo.web.cern.ch/dmc-repo/dmc-el7.repo
sudo yum-config-manager --enable dmc-el7/x86_64

then we need to install the following 4 packages, which are required to build gfal2-python package:

sudo yum install glib2 glib2-devel gfal2 gfal2-devel

note that gfal2/gfal2-devel might have to be upgraded. The version we get from standard repositories is:

gfal2-2.19.2-3.el7.x86_64
gfal2-devel-2.19.2-3.el7.x86_64

while DMC Production will give us

gfal2-devel-2.20.0-1.el7.cern.x86_64
gfal2-2.20.0-1.el7.cern.x86_64

Shahzad, would you see any reason not to perform these extra steps in the build node (vocms055)? Or @muhammadimranfarooqi Imran, but I know he's on vacation at this moment. If you believe it's fine, then I will contact the VoC to make these modifications on vocms055.

PS: our reqmgr2ms-unmerged is running even older RPMs e.g. gfal2-2.18.2-2.el7.x86_64
PPS.: I'm still checking what exactly are the required packages to have in the container/node running MSUnmerged.

[muhammadimranfarooqi]

to 206a09b on slc7_amd64_gcc630.
Build log can be found here.

[amaltaro]

@muhammadimranfarooqi @smuzaffar concerning my comment above, please let me know if you see anything missing or any problem with that procedure. If it looks good to you, would you be able to make these changes on vocms055? Otherwise I might contact Germano/VoC.

[muhammadimranfarooqi]

to 206a09b on slc7_amd64_gcc630.
Build log can be found here.

[muhammadimranfarooqi]

+1 to 206a09b on slc7_amd64_gcc630.
Build log can be found here.

[amaltaro]

Wonderful! Thanks, Imran!