INSTALL

         C3 version 4.0:   Cluster Command & Control Suite
           Oak Ridge National Laboratory, Oak Ridge, TN,
     Authors: M.Brim, R.Flanery, G.A.Geist, B.Luethke, S.L.Scott
                 (C) 2001 All Rights Reserved
 
                             NOTICE

 Permission to use, copy, modify, and distribute this software and
 its documentation for any purpose and without fee is hereby granted
 provided that the above copyright notice appear in all copies and
 that both the copyright notice and this permission notice appear in
 supporting documentation.
 
 Neither the Oak Ridge National Laboratory nor the Authors make any
 representations about the suitability of this software for any
 purpose.  This software is provided "as is" without express or
 implied warranty.
 
 The C3 tools were funded by the U.S. Department of Energy.


I. REQUIRED SOFTWARE
--------------------

Before C3 can be installed on a system, you must ensure that the following 
software is installed on your system. The following seven software packages
are required:  the C3 tools suite, Rsync, SSH (or OpenSSH), Python, and Perl.
You must also configure that system to support host name resolution of the 
machines listed in the configuration file (either through DNS or /etc/hosts).
 Finally, if you wish to use the C3 pushimage command, which pushes system 
images across a cluster, you must install SystemImager.

Instructions for obtaining each of these software packages are given below.

  C3 tools may be obtained from http://www.csm.ornl.gov/torc/C3

  Rsync, Perl, SSH, and Python should be included with your distribution.
  If they are not then download the source or binaries from their respected
  web sites.

  Perl may be obtained from http://www.perl.com
	C3 requires 5.005 or greater

  Python may be obtained from 
  http://www.python.org/
	C3 version 3 requires Python 2.0 or greater

	additionally C3 requires either the binary or a link to the python 
	interpreter to 	be in your path (and that it be named python2). To 
	check it type "python2 -V" and make sure you get output (the current 
	version of python being run). If you do not get any output then you must 
	find where the python library is on your machine and create a link to
	the binary. Such as "ln -s /usr/bin/python /usr/bin/python2" if
	/usr/bin/python is where your python binary is located and /usr/bin
	is in your path.

  SystemImager may be obtained from 
  http://www.systemimager.org/


II. C3 INSTALLATION
----------------
NOTE: if you are wanting to use the scalable model of the C3 tolls then follow
steps A and B, read C as it still pertains to the scalable model, then see the 
README.scale file for the scalable instructions.

A. pre-install
	Begin by making sure that Rsync, OpenSSL, OpenSSH, PERL, and
	Python are installed.  Install Systemimager, if needed.  Install DNS
	or /etc/hosts as needed, and make sure that hostname resolution is 
	supported.

	Directions for downloading each of these packages are given in 
	Section	I above.  Perl, Python, Rsync, OpenSSH, and OpenSSL are included 
	with most distributions

	You will need root access to install these packages on your system. 
        Follow the instruction in each package if you need to install them.

B. C3 install
	After you complete the pre-install (step A), install the Cluster 
	Command & Control (C3) tools. Begin by untar'ring the C3 package 
	and running the install script.  The install script places the C3 
	scripts in /opt/c3-4 and the man pages in the appropriate directory.  

	The C3 install script installs the C3 command suite, but does not 
	configure the commands or any local clusters for operation.  
	Directions for the remaining tasks are given below.

C. C3 configuration
	Specific instances of C3 commands identify their compute nodes with 
	the help of **cluster configuration files**:  files that name a set
	of accessible clusters, and that list and describe the set of 
	machines in each accessible cluster.  Cluster configuration files 
	are accessed in one of two ways:

	-.  explicitly:  an instance of a C3 command names a specific 
	configuration file, using a command-line switch.

	-.  implicitly:  an instance of a C3 command fails to name a specific
	configuration file, and the command defaults to the list of cluster
	descriptions given in /etc/c3.conf.
	
	When you install C3, you should create a default configuration file 
	that is appropriate to the site.  This file, which should be named
	/etc/c3.conf, should consist of a list of **cluster descriptor
	blocks**:  syntactic objects that name and describe a single cluster 
	that is accessible to that system's users.  

	The following is an example of a default configuration file that 
	contains exactly one cluster descriptor block:  a block that 
	describes a cluster of 64 nodes:

		cluster local {
			htorc-00:node0  #head node
			node[1-64]	#compute nodes
		}

	Cluster description blocks consist of the following basic elements:

	-.  a **cluster tag**:  the word "cluster", followed by a label,
	    which assigns a name to the cluster.  This name--here, "local"--
	    can be supplied to C3 commands as a way of specifying the cluster
	    on which a command should execute.

	-.  an open curly brace, which signals the start of the cluster's
	    declaration proper.

	-.  a **head node descriptor**:  a line that names the interfaces
	    on the cluster's head node.  The head node descriptor shown here
	    has two parts:

	    -.  The string to the left of the colon identifies the head 
		node's **external** interface: a network card that links 
		the head node to computers outside the cluster.  This string
		can be the interface's IP address or DNS-style hostname.
	    -.	The string to the right of the colon identifies the head 
		node's **internal** interface: a network card that links the
		head node to nodes inside the cluster.  This string can be 
		the interface's IP address or DNS-style hostname.

	    Here, the head node descriptor names a head node with an external
	    interface named htorc-00, and an internal interface named node0.

	    A cluster that has no external interface--i.e., a cluster that is
	    on a closed system--can be specified by either

	    -.  making the internal and external name the same, or
	    -.  dropping the colon, and using one name in the specifier. 

	-.  a list of **compute node descriptors**:  a series of individual
	    descriptors that name the cluster's compute nodes.  

	    The example given here contains exactly one compute node 
	    descriptor.  This descriptor uses a **range qualifier** to 
	    specify a cluster that contains 64 compute nodes, named node1, 
	    node2, etc., up through node64.  A range qualifier consists of 
	    -.  a first, nonnegative integer, followed by 
	    -.  a dash, followed by 
	    -.  a second integer that is at least as large as the first.

	    In the current version of the C3 tools et, these range values are 
	    treated as numbers, with no leading zeroes.  A declaration like

		    cluster local {
			htorc-00:node0  #head node
			node[01-64]	#compute nodes
		    }

	    expands to the same 64 nodes as the declaration shown above.  To 
	    specify a set of nodes with names like node01, node09, node10, ...
	    node64, use declarations like

		    cluster local {
			htorc-00:node0  #head node
			node0[1-9]	#compute nodes node01..node09
			node[10-64]	#compute nodes node10..node64
		    }

 	-.  a final, closing curly brace.

 	Configuration files that specify multiple clusters are constituted as
	a list of cluster descriptor blocks--one per accessible cluster.
	The following example of a cluster configuration file contains three 
	blocks that specify configurations for clusters named local, torc, 
	and my-cluster, respectively:

		cluster local {
			htorc-00:node0	#head node
			node[1-64]	#compute nodes
			exclude 2
			exclude [55-60]
		}

		cluster torc {
			:orc-00b
		}

		cluster my-cluster {
			osiris:192.192.192.2
			woody
			dead riggs
		}

	The first cluster in the file has a special significance that is
	analogous to the special significance accorded to the first 
	declaration in a make file.  Any instance of a C3 command that fails
	to name the cluster on which it should run executes, by default, on 
	the first cluster in the configuration file.  Here, for example, any
	command that fails to name its target cluster would default to local.

	The cluster configuration file shown above illustrates three final
	features of the cluster definition language:  **exclude qualifiers**,
	**dead qualifiers**, and **indirect cluster** descriptors.

	**Exclude qualifiers** allow nodes to be excluded from a cluster's
	configuration: i.e., to be identified as offline for the purpose of
	a command execution.  Exclude qualifiers may only be applied to 
	range declarations, and must follow immediately after a range 
	declaration to which they are being applied.  A series of exclude 
	declarations is ended by a non-exclude declaration, or the final "}" 
	in a cluster declaration block.  
	
	An exclude qualifier can be written in one of three ways:
	-.   "exclude n", where n is the number of a node to exclude from the
	     cluster;
	-.   "exclude[m-n]", where m, m+1, m+2, ..., n-1, n is the range of 
	     nodes to exclude; or as
	-.   "exclude [m-n], which has the same effect as "exclude[m-n]".
	Note that a string like "exclude5" is parsed as a node name, rather 
	than as an exclude qualifier.

	In the above example, the two exclude qualifiers have the effect of
	causing node2, node55, node56 node57, node58, node59, and node60 to 
	be treated as offline for the purpose of computation.

	**Dead qualifiers** are similar to exclude qualifiers, but apply to
	individual machines.  In the example given above, the machine named
	"riggs" in the cluster named "my-cluster" is excluded from all
	computations.

	"Dead", like "exclude", is not a reserved word in the current version
	of the C3 suite.  A specification block like

		cluster my-cluster {
			alive:alive
			dead
		}

	for example, declares a two-machine cluster with a head node named 
	"alive" and a compute node named "dead".

	An **indirect cluster descriptor** is treated as a reference to 
	another cluster, rather than as a characterization of a cluster per 
	se.  In the example shown above, the descriptor

		cluster torc {
			:orc-00b
		}

	is an indirect cluster descriptor.  An indirect descriptor consists 
	of

	-.  a cluster tag, followed by, 
	-.  an **indirect head head node descriptor**, followed by
	-.  an empty list of compute node descriptors.

	An indirect head node descriptor consists of an initial colon, 
	followed by a string that names a **remote** system.  This name, 
	which can either be an IP address or a DNS-style hostname, is checked
	whenever a C3 command executes to verify that that the machine being
	referenced is **not** the machine on which that command is currently 
	executing.

	A command that is destined for an indirect cluster is executed by 

	-.  first forwarding that command to the remote cluster's head node
	-.  next, executing that command, relative to the remote machine's 
	    default configuration file.

	For this feature to work properly, the remote machine must also 
	support	a fully operational C3 suite (version 4.0) placed in the 
	/opt/c3-4 directory.  

	The indirect cluster descriptors can be used to construct **chains** 
	of remote references:  that is, multi-node configurations where an
	indirect cluster descriptor on a machine A references an indirect 
	cluster descriptor on a machine B.  Here, it is the system 
	administrator's responsibility to avoid circular references.

D. Post-install
	For the C3 ckill command to work properly, ckillnode must be copied 
	to a directory on each compute node on every supported cluster.  The 
	easy way to install ckillnode is to use cexec and cpush.  After 
	installing and configuring C3 (cf. steps A-C above), use the 
	following two commands to push ckillnode to each node in the default 
	cluster.

	cexec mkdir /opt/c3-4
	cpush /opt/c3-4/ckillnode

	For the scalable version a full C3 install is needed on each node.
	This can be accomplished by either installing the RPM on each node
	or pushing the tarball out and using cexec (non-scalable at this point)
	to run the install script on each node.
	
This completes the installation of the C3 tools.

E. Notes
	The relative positions of nodes in c3.conf files can be significant
	for C3 command execution.  Version 3 of the C3 suite allows the use 
	of node ranges on the command line.  The command line parameters used
	to specify the indices of compute nodes refer to relative node 
	positions in c3.conf.  

	Consider, for example, the semantics of node range parameters, 
	relative to the following c3.conf file:

	cluster local {
		htorc-00:node0  #head node
		node[1-64]	#compute nodes
		exclude 60
		node[129-256]
	}

	This cluster is made up of 192 nodes.  Here, 

	-.  the 64 nodes named node1 through node64 correspond to slots 0-63
	-.  the 128 nodes named node129 through node256 correspond to slots
	    64-191--and **not**, for example, to slots 129-256.

	Note also that the excluded node--node60--acts as a place holder in 
	the range of indices: node60 is a relative index of 59, which allows
	nodes node61, node62, node63, and node64 to correspond to 60, 61, 62,
	and 63, respectively.  This "place holder" effect is an important 
	reason for explicitly specifying that a node is dead or excluded--as
	opposed to simply dropping that line from the specification.

	Two new tools in version 3.1 of the C3 tools suite support the 
	management of node numbers.  The first, cname, inputs a node name,
	and outputs that node's relative position (slot number).  The second,
	cnum, inputs a range of slot numbers, and outputs the names of the
	corresponding compute nodes.


III  C3 SUITE DOCUMENTATION
---------------------------
	
C3 command documentation may be found in two locations.
  1. Quick Usage Info - enter "<command> --help" at the command line
  2. Full Man Page - enter "man <command>" at the command line