README-mamake.md

mamake and the MAM language

MAM (Make Abstract Machine) is a simple rule-based make language that is implemented in just eight four-letter commands and four attributes, yet allows unlimited flexibility as it can execute arbitrary shell code. The program implementing MAM, mamake, is a portable C90 program written in a single file, mamake.c. This allows ksh 93u+m, or other programs using this build system, to be built using only a standard C compiler and utilities installation without any other dependencies or complications.

MAM was originally designed by Glenn Fowler at AT&T and intended as an abstraction layer for make implementations such as AT&T nmake. The original documentation for MAM specified a more extensive and markedly different language than was actually implemented in mamake.c. This file documents the MAM implementation that is currently in use.

Since fixing and maintaining AT&T nmake proved impractical, mamake is used here as a full make replacement, with some features gradually added to the language to facilitate human maintenance of the Mamfiles.

Table of contents

• General overview of the MAM language
  • Strict and legacy modes
• MAM variables
  • Special expansion syntax
  • Automatic variables
• Commands
  • Comments
  • Rules
  • Simple prerequisite rules
  • Referencing previously defined rules
  • Setting MAM variables
  • Shell actions
    • Viewpathing
    • Execution
    • Strict level 2+ change
  • Declaring common code for shell actions
  • Binding libraries
    • ...while scanning and sorting leaf directories
    • ...while building the current directory
  • Repeatedly iterating through a block
  • Including a MAM file
    • Provided include files
• Parallel processing
• Debugging mamake
• Appendix: Main changes from the AT&T version

General overview of the MAM language

MAM is a simple declarative language, easy to parse for machines and easy to read for humans, in which targets are defined that correspond to files that need to be generated or updated, or that are prerequisites. mamake reads build scripts from a file, Mamfile by default, from start to finish. As it encounters make target...done blocks, it 'makes' (updates) the targets using the instructions within those blocks. Those instructions may declare dependencies and/or specify shell command actions.

If a target is specified on the command line, mamake will update the actions contained within that target and ignore the rest.

Strict and legacy modes

By default, mamake remains fully backward compatible with Mamfiles as originally generated by AT&T nmake. If the MAMAKE_STRICT variable is set, some backward incompatible changes and deprecation warnings are activated to ensure correct operation and to facilitate human maintenance of the Mamfiles. A numeric value assigned to MAMAKE_STRICT indicates the backward incompatibility level. The empty value is equivalent to level 1. These are called the "strict levels" and their absence the "legacy mode", also known as strict level 0.

Each time backward incompatible changes are introduced to mamake that would break previous Mamfiles, those are made subject to a MAMAKE_STRICT value of one higher than the previous highest one; details of those changes are documented throughout this file and listed in the appendix below. This makes it possible to test or backport old code using the current build system. Current Mamfiles should use the highest strict level available. The current highest available strict level is 5.

MAM variables

MAM variables are imported from the environment or set via setv (see below). They are referenced and expanded using the syntax %{variable_name}. Except for the special expansion syntax described below, it is an error to reference an undefined variable using a % expansion.

Obsolete: At strict levels 3 and below, expansions may start with ${ instead of %{, making the syntax similar to sh(1), though the braces are not optional. This syntax potentially conflicts with the expansion of shell variables in shell actions. If an undefined MAM variable is expanded with the $ syntax and the variable name is valid in sh(1) syntax, the expansion is left in place unexpanded, otherwise it is removed. At strict level 2 and 3, it is left unexpanded even if it is not a valid sh(1) variable name; this allows POSIX shell expansions like ${foo#*bar} in shell actions. Old $ and new % syntax may be mixed.

Variables always expand to their literal values, and variable references (including in setv values) only work for previously defined variables.

Obsolete: At strict levels 1 and 0, the expansion of MAM variable references is recursive, i.e., the value may itself contain other variable references. Beware: there is no reference loop detection; any variable referencing itself directly or indirectly will cause mamake to crash.

Note that, in shell actions (see exec below), MAM variables are expanded before the script ever reaches the shell. Consequently, the use of single shell quotes '...' does not stop their expansion as you might expect.

Special expansion syntax

In %{variable?str?x?y?}, if the string value of the variable is identical to str, then the value x is substituted, otherwise y. A str of * is treated specially: if the variable is set to a non-empty string, then the value x is substituted, otherwise y. The x and y values may result from nested variable references. The last ? is optional.

In %{variable-x}, the value of variable is substituted if it is defined and non-empty, otherwise x is substituted. This expansion type is disabled for variable names starting with -. In %{variable+x}, x is substituted if the value of variable is defined and non-empty, otherwise the reference is removed. In both cases, x may be empty. Note that, unlike in sh(1), no distinction is made between an undefined variable and a defined variable with an empty value.

In %{variable|s}, the value of variable is split into fields by whitespace and the one-line sh(1) script s is executed with each field written as a separate line to its standard input. The value of the expansion is the output of s with each newline changed back into a space, except that a terminating newline (if any) is discarded. This mechanism allows easy editing of variable values containing multiple pathnames using line-oriented utilities that read from standard input, such as sed(1) or grep(1). For example, if the automatic variable %{^} (see below) contains foo.c bar.c baz.c, then %{^|sed 's/\.c$//'} yields foo bar baz.

In %{variable@s}, the value of variable is split into fields by whitespace and each field becomes a positional parameter (starting at $1) in the one-line sh(1) script s. The value of the expansion is the output of s with each newline converted into a space, except that a terminating newline (if any) is discarded. This allows easy processing of one or more pathnames using shell commands that take arguments, such as basename(1) or printf(1), or shell loops. For example, if the value of SRC is /dir/path/foo.c, then %{SRC@basenanme -- "$1" .c}.o yields foo.o (any subsequent fields would be discarded in this case).

Automatic variables

The following variables are set and updated automatically. They are inspired by similar variables in make implementations, but since mamake is different, so are these variables.

%{@} is the name of the rule currently being made.

%{<} is the name of the prerequisite that was last processed (make...done or makp) or referenced (prev) within the current rule.

%{^} is a space-separated list of names of all the current rule's previously processed or referenced prerequisites.

%{?} is a space-separate list of the current rule's previously processed prerequisites that have been updated by a shell action (see exec below) during the current mamake run. Prerequisites that were already up to date, or prerequisites that do not contain a shell action, are not included.

Commands

MAM commands have the following basic form:

command [ argument [ operand string ] ]

The command name consists of four lower-case letters. Unrecognized commands or attributes are an error. The argument is a single word. The operand string is any arbitrary text until the end of the line.

Comments

note is the comment command and is ignored.

Obsolete: In the legacy mode, info and meta are also ignored.

Rules

make target [ attribute ... ]
done [ target ]

A make...done block defines the rule named target using the other commands described here. Unless the virtual attribute is used, target names the pathname of the file generated or referenced by the rule.

mamake processes the commands within the block if the target is out of date or if the rule has the virtual attribute (see below).

The target may be repeated as the operand to the done command. In that case, it is matched against the current make target and any mismatch will produce a "mismatched done statement" error. If it is omitted, the current make target is assumed.

Dependencies may be defined in two ways:

1. By nesting make...done blocks: the enclosing rule is the parent and the enclosed rules are the prerequisites.
2. By using the prev command (see Referencing previously defined rules below) to reference a previous make...done block. The dependency is defined as if that block were repeated at the prev command's location.

If the block contains one or more exec commands (see Shell actions below), the done command executes the shell script defined by them.

A make...done rule may lack any exec action, in which case it declares a dependency (a file that must be present and whose timestamp is propagated to potentially outdate parent rules, such as a source code file that comes with the distribution or a header file preinstalled by a previously run Mamfile). The rule may include its own dependencies. If it does not have any dependencies of its own, the make...done block is empty, with make immediately followed by done. The one-line makp command (see below) may be used as a shorthand for the latter case.

Making a prerequisite that is currently being made, or one that has already been made, produces a warning; at strict level 3 and up, this is an error.

One or more attributes may be specified by appending them to the make command. (At strict levels < 2, they may also be appended to the done command; the effect is the same either way. At strict level 1, this produces a deprecation warning.) Attributes apply to the current rule only and do not propagate down to nested rules. The following attributes are available:

• dontcare: Marks files that do not need to exist. If the file exists then its last-modified timestamp is checked and propagated, otherwise it is silently ignored.
• ignore: The timestamp associated with the target is ignored in dependency resolution.
• notrace: Disables echoing (xtrace) of shell action commands. This does not disable the trace header for the containing rule (see Shell actions below).
• virtual: Marks a rule that is not associated with any file. Its associated shell action (if any) is executed every time the rule is processed. It will not run in parallel with subsequent shell actions even if the -j option was given. By convention, a virtual rule with target install performs pre-installation.

Obsolete: At strict level 4 and up, the following attribute is not available.

• implicit: Equivalent to dontcare.

At strict level 1 and up, specifying the following attributes is deprecated and will produce a warning; at strict level 2 and up, specifying these is an error.

• archive: Ignored. Historically used to mark the generation of an ar(1) archive.
• generated: Marks rules that produce output files generated by a shell action. The explicit assignment of this attribute is ignored at strict level 1. The exec command implicitly assigns this attribute. If a rule has this attribute, other rules dependent on this rule will avoid applying viewpathing based on this rule.
• joint: Ignored. Historically used to mark one of a group of rules that are built by a single shell action.

Simple prerequisite rules

makp target [ attribute ... ]

makp creates a rule that declares a dependency on a prerequisite file named by target in a manner equivalent to an empty make target/done block, with the optional attributes applied to the new rule. A nonexistent prerequisite is an error unless a virtual or dontcare attribute is given. Declaring a dependency on a prerequisite that is currently being made (i.e.: directly or indirectly within that prerequisite's block) is an error.

Referencing previously defined rules

prev target

If target matches a previously defined rule, prev adds a dependency on that rule to the current rule. This is used to make a rule a prerequisite of multiple make...done blocks without repeating the rule. It is an error to specify attributes, because the attributes of the referenced rule are used. In the legacy mode, attributes are silently ignored.

Obsolete: If the strict level is < 4, and if target does not match a previously defined rule, then the following applies. In the legacy mode, prev creates an empty dummy rule and ignores the attributes; this is for backward compatibility. At strict levels 1 and up, prev in this context is equivalent to makp (see above). Declaring a dependency on a prerequisite that is currently being made produces a warning; at strict level 3, this is an error.

Setting MAM variables

setv variable [ defaultvalue ]

Defines a new MAM variable, optionally assigning the initial defaultvalue. If the variable already has a value, the defaultvalue is ignored; assigning a new value is not possible. When mamake starts, it imports all environment variables as MAM variables, so any variable's default value can be overridden by exporting an environment variable by its name. Any leading and trailing whitespace is stripped from the value. The value is otherwise taken entirely literally, with no parsing of quotes, etc.

Obsolete: If the strict level is less than 2 and the defaultvalue begins and ends with double quotes ("), those quotes are discarded, though double quotes elsewhere in the value are not treated specially.

When the variable is CC, mamake runs the mamprobe script to probe the C compiler for flags and features, or uses that script's stored results if not outdated. The results are stored as a series of setv commands in a file in the directory %{INSTALLROOT}/lib/probe/C/mam, the file name being a hash of full path to the compiler indicated by %{CC}. That results file is then read and included in the current Mamfile as if it followed the setv CC command.

Shell actions

exec - code

One or more exec commands within a make...done block define a shell script that is run to generate the target. The argument following exec is ignored; by convention it is -. Each exec command appends a line of code to the shell script for the current rule. It is customary for a rule's exec commands to be contiguous, but not necessary.

Before adding each line of code to the script, MAM variable references (see MAM variables above) are expanded; their literal values are inserted into the code line (beware: no quoting is applied!). Because variables are expanded when the line is encountered, the value of the automatic variables for any exec line depends on the position of the line in the rule.

Viewpathing

After MAM variable expansion, viewpathing is applied. The first colon-separated element of %{VPATH} is considered the object code directory and the second the source code directory; viewpathing provides the first with a view to the second. Viewpathing applies two transformations.

The first is prerequisite replacement. Each word (separated by whitespace, ;, (, ), `, |, & or =) is searched for in the current rule's prerequisites, and if it matches the name of a prerequisite that was not generated by a shell action, it is replaced by the canonical path to it in the source directory, ensuring that things like prerequisite headers are found.

The second is include flag duplication. After every argument that looks like a compiler include directory path (i.e., starting with -I) with a relative path name (i.e., a directory path that does not start with a /), another argument starting with -I is inserted with that path name prefixed by the path to the source directory. This mechanism ensures that headers are found both in the object directory and in the source directory. It is processed regardless of the command; for example, it also works for compiler flags passed to iffe(1).

Note that shell quotes are not treated specially. If an argument starting with -I ends in a shell quote without preceding whitespace, that trailing quote is repeated along with the prefixed path and causes a syntax error.

Execution

When mamake encounters the done command, the script is executed by the shell whose path is in the SHELL environment variable or, absent that, by sh(1). Each shell action is run in a new instance of the shell.

Before executing the script, an empty line followed by a trace header in the following format is written to standard error:

# path/to/Mamfile: startline-endline: rule

During script execution, shell action commands are traced using the shell's xtrace option, unless the rule has the notrace attribute.

Strict level 2+ change

At strict level 2 and up, mamake turns off global pathname expansion (globbing) using set -f to make safer MAM variable expansion and shell field splitting possible; this avoids unexpected pathname expansion if a value contains ?, * or [. A shell action can override this using set +f; this should only be done for individual commands in a (subshell).

Declaring common code for shell actions

shim - code

One or more shim commands declare a 'shim': a common section of sh(1) code that will be automatically inserted in front of subsequent shell actions upon execution. Like exec, shim combines multiple lines of code into one section, with MAM variables expanded at declaration time and viewpathing applied at execution time. The effect of shim is global.

One use case is defining a shell function that each shell action can call. For example, such a function might invoke the compiler with a series of compiler flags common to all compiler invocations, with other flags added via arguments to the function, so the common flags do not need to be repeated in every shell action.

Only one shim is active at a time, but it can be redefined. When the next exec command is encountered, the shim is marked ready for use. The next time a shim command is encountered after that, it starts a new shim from scratch that affects subsequently executed shell actions. A single shim - deactivates the shim.

Binding libraries

bind -llibraryname [ dontcare ]

These commands are scanned for while sorting leaf directories for recursive building, and executed as normal commands while building the current directory.

...while scanning and sorting leaf directories

Any leaf directories with names that start with INIT will always be built before all others. For all other leaf directories, the presence of any bind command of the form bind -lfoo anywhere in a leaf directory's Mamfile causes the leaf directory named libfoo (if it exists) to be a prerequisite of that leaf directory. The prerequisite leaf directory does not have to be in the same parent directory, as long as it is processed as part of the same scan. At this stage, attributes are ignored.

...while building the current directory

The bind command takes an argument of the form -llibraryname, which causes a MAM variable mam_liblibraryname to be defined (see MAM variables above). The variable will contain either the compiler argument for linking to the library libraryname (either the -llibraryname flag, or the full path in case of a static library) or, if the dontcare attribute is specified, possibly the empty string. Any library dependencies are also included (see below). This can be used both for AST libraries shipped with the distribution and for system libraries.

In addition, for each corresponding *.a library archive dependency built previously, its time stamp is checked and the current target is marked as outdated if it is newer, as if a prev had been executed for it. The variable set by bind is global, but the marking of the target as outdated applies to the current rule only, so it may be necessary to repeat a bind command when statically linking executables that depend on a library, otherwise they may not be relinked when the library changes. The mam_liblibraryname variable will not be regenerated when repeating a bind.

There is also a mechanism to communicate library dependency information across Mamfiles and mamake invocations. If a file named libraryname.req in the current directory or an %{INSTALLROOT}/lib/lib/libraryname file exists, mamake processes each of the words in the form -llibraryname in its contents as if they were arguments to bind commands and the resulting values are appended to the value of mam_liblibraryname as dependencies separated by spaces. mamake does not create these dependency files; they are expected to be generated by Mamfile shell actions (see Shell actions above). The INIT package preinstalls the mkreq and mkreq-maplib scripts for this purpose. If no such dependency file exists, and the dontcare attribute is added, then mamake compiles a small test program on the fly to check if the library exists; if this fails, the mam_liblibraryname variable will be emptied.

Cross-directory dependencies on AST library headers (preinstalled in $INSTALLROOT/include/ast) are similarly communicated via a file with the path %{INSTALLROOT}/lib/mam/libraryname. The bind command automatically includes this file as necessary. The INIT package preinstalls the mkdeps script that Mamfile shell actions should use to generate this file while building each library. The generated file is expected to:

1. set the INCLUDE_AST variable to %{INSTALLROOT}/include/ast;
2. define a single virtual rule by the name of _hdrdeps_liblibraryname_ which contains all the rules that define the library's public include/ast headers and how they depend on each other.

This way, Mamfiles can declare a dependency on a single header and all its dependencies using a simple prev %{INCLUDE_AST}/headername.h command. Any bind -llibraryname command will automatically apply the dependencies defined in the corresponding file to the context of the current rule. The non-existence of this file is not an error and is silently ignored.

Repeatedly iterating through a block

loop variable word [ word ... ]
done

loop reads the lines contained between it and the corresponding done repeatedly with a named variable set to each of the words. The lines are processed as part of the rule containing the loop. The variable is restored to its previous state after the loop completes.

Note that loop causes repeated reading and processing of Mamfile lines, not necessarily repeated execution. For instance, a loop can be used to consolidate repetitive make...done rules. However, each rule is only made once and subsequent rules by the same name are an error at strict level 3 and up, or skipped over at strict < 3. So it only makes sense to do this if the contained make target names are modified by the expansion of the iteration variable.

loop requires that the Mamfile be seekable (i.e.: not a pipe).

Including a MAM file

incl filename

The incl command reads a file with MAM commands from filename as if those commands had appeared in place of the incl command. If filename does not contain a /, mamake will look for the file in the current directory first, then in %{PACKAGEROOT}/src/cmd/INIT/include.

Provided include files

The following are provided in %{PACKAGEROOT}/src/cmd/INIT/include:

• link_ar.mam: A shell action (see exec above) for linking a static library archive (libname.a) based on the contents of the automatic variables, for including at the end of a make libname.a...done rule. All the child rules are expected to be *.o object files. The action also deletes object files that no longer have a matching source from the archive.

Parallel processing

As of strict level 5, mamake supports parallel building using the new -j maxjobs option (which can be passed via bin/package make). This can speed up the build on multiprocessor or multicore systems.

Each make...done command containing a shell action (i.e., one or more exec commands) may have its shell action processed in parallel, with mamake continuing to process subsequent shell actions at the same or deeper nesting levels before the current one has finished. There is one exception to this: shell actions belonging to rules with the attribute virtual will not run in parallel with subsequent shell actions.

Each done command corresponding to a make rule will block any further reading of the Mamfile until all shell actions belonging to rules nested within the current rule have finished. The prev command will similarly block until the shell action belonging to the declared dependency has finished. The bind command, when binding to a static library built within the same Mamfile, will block until the library has finished linking.

Mamfiles at strict level 5 or higher are expected to be compatible with parallel processing by declaring these blocking dependencies correctly and explicitly, without assuming sequential processing.

Debugging mamake

If the environment variable MAMAKE_DEBUG_PREFIX is exported, its value is split into whitespace-separated command arguments and inserted before every mamake invocation. This allows debugging mamake with tools like valgrind, strace, or dtruss, which are used by passing a command to them as a set of operands.

Appendix: Main changes from the AT&T version

Compared to the original AT&T version, ksh 93u+m made a number of changes to mamake that facilitate correct operation and make it easier to maintain Mamfiles by hand. The following lists the important changes.

• Introduced the notion of 'strict mode' levels that tidy things up by activating some backward incompatible changes and deprecation warnings.
• Indentation and word separators may use any whitespace (e.g. tabs), not only spaces.
• Fixed a bug that stopped a rule marked virtual (not associated with any file) from being executed if a file by that rule's name exists.
• Unrecognized commands and rule attributes throw an error instead of being silently ignored. This also applies to the bind command with an argument not starting with -l.
• It has been made optional to repeat the make target after done.
• The notrace attribute was added to disable xtrace for a rule's shell action.
• The automatic variables %{@}, %{<}, %{^} and %{?} have been added.
• An iteration block command, loop...done, has been added.
• A command to set common code for shell actions, shim, has been added.
• The bind command now reads library header dependency rules from a central rules file that is automatically generated for each library by the supplied mkdeps script.
• Attempting to make a rule that has already been made produces a warning.
• Attempting to declare a dependency on a rule currently being made produces a warning.
• MAM expansions may start with %{ instead of ${, avoiding conflicts and confusion with the shell's ${ syntax in shell actions.
• The makp command may be used instead of an empty make...done block to declare a simple prerequisite with optional attributes.
• The special expansion syntaxes %{variable|sh-script} and %{variable@sh-script} have been added, allowing the editing of variable value fields written to the script's standard input as lines or passed as positional parameters.
• At strict level 1 and up:
  • Appending attributes to done instead of make is deprecated and produces a warning.
  • The ignored archive and joint attributes are deprecated.
  • Explicitly specifying the generated attribute is deprecated.
  • The dummy info and meta commands are unavailable instead of ignored.
  • The prev may be used as an equivalent of makp.
  • When prev references a previously processed target, attributes are an error instead of being ignored.
  • The legacy silent and ignore command prefixes are unavailable.
• At strict level 2 and up:
  • Appending attributes to done instead of make is an error.
  • The archive and joint attributes are unavailable.
  • Explicitly specifying the generated attribute is an error.
  • All variable references are expanded to their literal values without scanning the values for recursive variable references.
  • setv does not remove leading and trailing " from the value.
  • Shell actions have pathname expansion (globbing) disabled by default.
• At strict level 3 and up:
  • Attempting to make a rule that has already been made is an error.
  • Attempting to declare a dependency on a rule currently being made is an error.
• At strict level 4 and up:
  • MAM expansions can no longer start with ${. Only %{ is recognized.
  • The prev command may no longer be used as an equivalent of makp.
  • The implicit attribute is not available.
• At strict level 5 and up:
  • The new -j option for parallel building is allowed to take effect. Mamfile dependency declarations (prev, nested make...done, bind) are expected to be compatible with parallel processing.