Skip to content

Latest commit

 

History

History
106 lines (64 loc) · 6.58 KB

2.CMDSRV.md

File metadata and controls

106 lines (64 loc) · 6.58 KB
Raw

CMDSRV

The Command Server (CMDSRV) is the heart of the Admin Console and provides the following functions,:

  • Escalation of user privilege to root for a predetermined list of operations
  • Management of long-running Jobs in service of those operations
  • Status Reporting

As mentioned in the Overview, the code can be found in the git repo https://github.com/iiab/iiab-admin-console and it is deployed to /opt/admin/cmdsrv.

Architecture

CMDSRV is written principally in Python3 and has the following components:

  • iiab-cmdsrv3.py - This is the main executable that runs as a systemd service.
  • /etc/systemd/system/iiab-cmdsrv.service - the unit file for that service.
  • /opt/admin/console/cmd-service3.wsgi - a wrapper that provides a security gateway and supports proxying by the web server.
  • /etc/uwsgi/apps-enabled/admin-console.ini - the definition of the previous.
  • /opt/admin/console/server-info.php - a script to test whether the CMDSRV is running.
  • /usr/lib/python3/dist-packages/iiab/ - a directory containing python library modules.

At the top level CMDSRV listens on a ZeroMQ message queue over a unix IPC socket at /run/cmdsrv_sock. This socket is owned by the web server user www-data and may also be accessed by root.

In most cases the Command string is sent by the Console app via ajax to the web server, which passes it on the uswgi wrapper module, which performs authentication and then writes it to the socket.

During initialization CMDSRV starts a pool of Worker threads to handle processing of Command strings and starts the Job Minder thread that manages the job queue.

After initialization, any thread in the processing pool that is not busy will poll the message queue for a Command and when it finds one it removes it from the message queue, validates it and looks up the function that processes it. That function either runs the job immediately if the Command is a short running task or defines one or more Jobs and puts them into a queue to be started by the Job Minder thread. In either case a response is sent to the caller with the results of a short running Command or a report of a Job having been created.

The Job Minder thread watches for running Jobs to terminate due to success, failure, or cancellation, and starts Jobs from the queue up to the maximum number of concurrent Jobs.

Incoming Command strings and Jobs are stored in a sqlite3 database /opt/admin/cmdsrv/cmdsrv.0.3.db in its present version.

Unprocessed messages remain in the ZeroMQ message queue until a Worker thread is available to remove and process them.

Commands

Some Commands take a relatively short amount of time to complete and return results directly. Many of these report status. Other Commands are intended to be long running and are put onto a job queue. Downloads fall into this category.

The list of Commands and the functions that implement them may be found by opening https://github.com/iiab/iiab-admin-console/blob/master/roles/cmdsrv/files/iiab-cmdsrv3.py and searching for 'List of all Commands is Here'.

Commands are issued in the form of XXX-XXX {"key": "value"}; the Command verb is always uppercase and separated by dashes, and Command parameters must be properly formed json.

Such a Command string may be sent from the Console app or from the iiab-cmdsrv-ctl utility for testing.

There is a special Command STOP that is evaluated before any other and simply causes the CMDSRV to close threads and stop execution.

Commands are checked for duplication and for the presence of an internet connection if the Command requires one. It looks for duplicate Commands and rejects them.

Config

/opt/admin/cmdsrv/cmdsrv.conf contains runtime configuration parameters, mostly set by ansible variables at install time. The majority are paths and urls, the names of which should indicate their use.

The following parameters and default values need more explanation:

  • "cmdsrv_no_workers" : 5 - the number of Worker threads to start
  • "cmdsrv_job_poll_sleep_interval" : 1 - time in seconds to wait until next cycle of polling job queue
  • "cmdsrv_max_concurrent_jobs" : 7 - maximum number of Jobs that can be running at a given time
  • "cmdsrv_max_rpi_jobs" : 4 - maximum number of Jobs that can be running at a given time if the server is a Raspberry Pi
  • "cmdsrv_lower_job_priority_flag" : true, - can CMDSRV lower the priority of some Jobs so as not to overrun the server
  • "cmdsrv_lower_job_priority_str" : "/usr/bin/nice -n19 /usr/bin/ionice -c2 -n7 " - OS command string to lower priority

There are also constants in /usr/lib/python3/dist-packages/iiab/adm_const.py (and iiab_lib.py set during IIAB install) that are used by the python libraries.

Jobs

For longer running Commands, Worker threads add Jobs to a python dictionary for future processing. The Job Minder thread removes a Job from that dictionary, adds it to the running dictionary and starts the Job, whenever there are less than cmdsrv_max_concurrent_jobs running.

A Job's Status may have one of the following values:

  • SCHEDULED - in the Job queue but not started
  • STARTED - a running Job
  • RESTARTED - a Job that was running when the server was shut down and has been restarted
  • SUCCEEDED - a Job that terminated with success
  • FAILED - a Job that terminated with failure
  • CANCELLED - a Job that was terminated with a cancel Command

The output from a running Job is written to /tmp/job-<job_id>

The state of a Job is updated in the sqlite3 database whenever the status changes.

Security

The principal security mechanism is the unix IPC socket at /run/cmdsrv_sock to which no user has access except www-data and root. Since www-data has no shell permitted no other user can issue Commands; this is also the basis of web server security. The uwsgi wrapper acts as a security gateway in that it authenticates the user submitting a Command from the Console against the /etc/shadow file and only passes the Command to the CMDSRV if authentication succeeds.

Logging

CMDSRV logs to the system log, and these messages are most easily viewed by issuing the OS command:

journalctl -t IIAB-CMDSRV

Troubleshooting and Debugging

When running as a service CMDSRV traps all error, and this can mask problems that need troubleshooting. The best way to troubleshoot is to issue the following as root:

  • cd /opt/admin/cmdsrv
  • systemctl stop iiab-cmdsrv
  • ./iiab-cmdsrv3.py

This will start CMDSRV from the command line and allow python errors not to be trapped.

iiab-cmdsrv-ctl 'COMMAND {"key": "value"}' can be used to test a particular Command. (Note use of single quotes to make the Command and parameters a single argument.)

Document Version

This document was last maintained on December 29, 2022.