Initial manage/config changes for CouchDB 3.1.2

[amaltaro]

Fixes dmwm/WMCore#10829

Let me try to document all the changes and why they have been made:

• couchdb/default.ini, wmagentpy3/default.ini and tier0/default.ini configuration file is required to start up CouchDB. This is a blind copy of what is provided by CouchDB itself.
• local.ini configuration file contains our specific CouchDB configuration, overriding whatever is in the default.ini
  • chttpd/authentication_handlers define the Proxy Authentication mechanism
  • couchdb/single_node defined to true to get the admin users automatically created during deployment, in addition to some system-like databases
• manage script required a few changes, like:
  • database/management calls need to be made with user/pass, even if it's all localhost!
  • couchdb -p, couchdb -a, couchdb -o, couchdb -e and couchdb -b no longer exist, removed!
  • couchdb -A has been replaced by couchdb -couch_ini
  • adapted the start and stop code
  • CouchDB status now uses the _up unathenticated endpoint
  • pushing couchapps follow exactly the same logic, only difference is that it uses for loop now
• couchdb/monitoring.ini needs to use user/pass (set up during couchdb deployment)
• wmagentpy3/deploy position CouchDB config files under the couchdb configuration directory
• wmagentpy3/local.ini defines 2 accounts, the admin one and unittest users. The unittest user will be disabled when CouchDB gets deployed in production. These accounts are automatically created during start up.
• wmagentpy3/manage is a mess and it makes our CI jenkins hard to deal with.
• for central CouchDB deployment and manage script operations, read credentials from couch_creds file.
• define max_replication_retry_count to infinity for WMAgent/Tier0 backend
• define replication connection_timeout to 300 secs (I've seen timeouts in testbed replications!)

In addition to that, it also includes:

• changes to reqmgr2ms/manage such that MSOutput and MSUnmerged do not get started in our VMs (MongoDB setup is broken).

TODO: Imran, we need to define a file for the Couch credentials under /data/auth/couchdb. We can discuss this over Mattermost.

[amaltaro]

For my own reference, this logic to kill the couchdb process in my private VM does not work. It fails and it exits the script, thus never starting CouchDB. Here is the relevant terminal output:

+ sudo -H -u $u bashs -lc '/data/srv/current/config/couchdb/manage stop '\''I did read documentation'\'''
Stopping CouchDB service...
  killing CouchDB process... 
/data/srv/current/config/couchdb/manage: line 229: kill: (11496) - Operation not permitted
  killing CouchDB process... 
/data/cfg/admin/InstallDev: line 89: 11496 Killed                  sudo -H -u $u bashs -lc "$cmd"

[vkuznet]

Alan, I made few suggestions:

• there is a pattern of using http://$USERPASS@localhost:5984, therefore it is better to define it once and use across the code. For instance, if it was already there then you don't need to change so many places.
• you should replace all hard-coded couch credentials in all files since you'll push these files into github and we should avoid leakage of credentials (even for our backends)
• usage of perl is not required but leads to increase size of RPMs, therefore I suggest to replace it with sed.

[vkuznet]

Regarding security measurements, according to Couch docs we can define group members via simple PUT HTTP request:

curl http://localhost:5984/OUR_DB/_security \
-X PUT -H 'content-type: application/json' \
-H 'accept: application/json' \
-d '{"admins":{"names":["dmwm"],"roles":["admins"]},"members":{"names": ["user1","user2"],"roles": ["developers"]}}'

Here, we create new users for OUR_DB database. The dmwm user has role admins, while user1,user2 have role developers. I suggest to make separate shell script with this curl template and add it to repository. Then, during deployment phase we can use a script with provided parameters to add new members, e.g.

#!/usr/bin/env bash
db=$1
name=$2
role=$3
# perform update in couchdb to assign new member to a given database
curl http://localhost:5984/$db/_security \
   -X PUT -H 'content-type: application/json' \
   -H 'accept: application/json' \
   -d '{"admins":{"names":["$name"],"roles":["$role"]}}`

[amaltaro]

@muhammadimranfarooqi Imran, as discussed today, I separated the frontend changes (and a TODO monitoring.ini) in this PR: #1151

Could you please review and merge this one?

[amaltaro]

All right! I think I resolved all the questions and concerns in this PR, other than the migration from perl to sed that goes beyond the scope of these changes.

@muhammadimranfarooqi I have all these things tested in my VM and from my side they are ready to go. Note that it does not require any changes to the frontends!

Regarding testbed deployment, please do not proceed with it because we will have to run the first upgrade together. Actions need to be taken to migrate the databases to a CouchDB 3.x format. It would be great to have it merged sooner though.