MSOutput: perform custodial data placement

[amaltaro]

Impact of the new feature
ReqMgr2MS

Is your feature request related to a problem? Please describe.
MSOutput needs to have the ability to make custodial data placement as well, which means, make transfer requests against MSS/Tape endpoints.

Describe the solution you'd like
This development depends on:
#9841

then we should resume the data placement implementation and adapt whatever is needed to subscribe data against the long term storage (MSS/Tape).

For DDM data transfers, it seem we need to find out the final location in the MSOutput, thus requesting DDM to move data under a specific location.
For Rucio data transfers, it could be that we just need to provide a Tape expression, and let Rucio handle the final location. Needs to be discussed and investigated though.

The general policy we discussed during the Rucio transition meeting was as follows:
a. Use the API / Eric’s logic to pick the tape RSE in MSOutput as mentioned in this issue: #9842 (comment) . The MSOutput should decide where the final destination (the site to which the tape request is made) for the dataset based on the quota and usage for all those.
b. Create the Rucio rule / DDM transfer request for the output datasets (rucio container) before the workflow gets announced (thus, when it's in closed-out state).
c. The Rucio rule should be deleted manually (eventually by Nick)

Describe alternatives you've considered
none

Additional context
To be worked once this one has been implemented: #9841

[nsmith-]

I believe there will remain a need for a per-campaign custodial RSE expression in Rucio, the motivation being HI campaigns which often are to be only archived at FNAL. A default custodial RSE expression could be simply type=TAPE, letting Rucio balance the data as appropriate.
As @ericvaandering pointed out, the approval mechanism may create obstacles for using simple RSE expressions to match all tape sites.

[ericvaandering]

Today we decided that the best way to do this would be to write a little function to select a Tier1 for us based on the the quota. Here is a little piece of code which implements the algorithm. If you tell me where you want it and which API you want to call it with, I will make a PR for @todor-ivanov

It does need quotas as inputs which don't exist on Tape right now (and probably wants available space instead of quota), but that should be simple to adapt to. But the basics should be there.

https://github.com/ericvaandering/CMSRucio/blob/weighted_rucio_choice/src/pick_rse.py

[amaltaro]

One item that still needs to be decided is, whether we pick one Tape RSE per workflow or one Tape RSE per output dataset.
Nick/Eric, please let us know if you have any preference here.

[ericvaandering]

I have no opinion on this

[nsmith-]

Let's go with the current behavior, namely per-workflow tape destination.

[amaltaro]

For the tape locking, we likely want to check whether a rule against tape already exists (be it the same Tape or any other tape endpoint).. If it does, then we should definitely not make another one.