propositum | WINDOWS

A collection of free and open-source, portable tools to facilitate task & information management, automation, data manipulation and analytics.

Usage

Installation via the Releases for the latest binary distribution. Unzip, then ensure you run the Finish script.

The next few sections (and this repo) are dedicated to the Literate documentation of the orchestration script which installs the various components.

### --- NOTE: If you are reading from the PS1 script you will find documentation sparse, --- ###
### --- this script is accompanied by an org-mode file used to literately generate it.   --- ###
### --- Please see https://github.com/xeijin/propositum for the accompanying README.org  --- ###

Define

Some background on the literate functions used, as well as the tables used to maintain the user-defined Components and Variables

Literate Functions

The list of Components and their attributes, as well as user-defined Variables are both maintained in an org-mode tables. Using org-babel and some elisp the contents of these tables are actually executed by PowerShell for use in scripts.

This makes it easier to add new components and variables, as well as modify the existing attributes of these items. It also helps us segregate user-defined inputs from functions, making debugging easier and our code (hopefully) cleaner.

Require elisp dependencies

First, let’s ensure we get any required elisp dependencies

(require 'org)
;(require 'json)

Import from CSV

Now define an org-babel source code block to import variables from an existing CSV file, into an org-mode table. It takes a single argument, the path to the CSV file as a string.

(with-temp-buffer
  (org-table-import csv-path nil) ;; MEMO on `nil' arg is in the footnotes.
  (setq LST (org-table-to-lisp))
  ;; comment out or cut below one line if you don't have column names in CSV file.
  (append (list (car LST)) '(hline) (cdr (org-table-to-lisp))))

Export to CSV

We also need to define an org-babel source block to export an org-mode table back to a CSV.

It takes two arguments, the path to the CSV file, and also a name for the org-mode table it generates. Both should be a string.

(save-excursion
  (org-open-link-from-string (concat "[[" tbl-name "]]"))
  (while (not (org-table-p)) (forward-line))
  (org-table-export csv-path "orgtbl-to-csv"))

Components

Components to be installed are maintained in a org-mode table and csv file which holds the metadata:

• var short/variable name for the component
• component full name for the component (including link to home page)
• license license type and evidence url
• usage the intended usage of the componet
• categorisation categorisation for assisting with internal sign-offs
• status whether a component is enabled or disabled
• service the method through which the component is acquired
  • github-release download as a GitHub release artifact (will take care of finding the Url for the latest release)
  • github-clone clone directly from GitHub repo
  • apache-dir-dl download from an apache-style directory/file listing web page (e.g. GNU Download Page)
  • direct-dl direct download link (can be redirected url)
• user the GitHub user for clone / release artifact download
• repo the GitHub repo for clone / release artifact download
• regex a regular expression to find the latest version of the file
• dl direct download link (also used to store the download link in vairous PowerShell functions)
• source the source download link (usually where newest versions of files are placed) – some logic for handling component name/version folders
• comment any other noteworthy information on a component

Let’s ensure we’re in the script root first

cd $psScriptRoot

Now we use org-babel’s #+CALL: to import our variables defined in components.csv using the Literate Functions we defined earlier.

IMPORT IMPORT IMPORT

IMPORT IMPORT IMPORT

Next, within the table, define the environment variables and their desired values

component	license	usage	categorisation
Cmder	MIT	console emulator & cmd replacement	Standalone Tool
emacs & org-mode	GPL-3.0	task & information management, text editor, IDE, composing documentation	Loosely Coupled with internal code (e.g. internal REST APIs)
doom-emacs	MIT	configuration framework for emacs	Loosely Coupled with internal code (e.g. internal REST APIs)
AutoHotKey	GPL-2.0	general Windows automation, expanding commonly used text snippets	Standalone Tool
KNIME Analytics Platform	GPL-3.0	data pipelines, transformation, automation & reporting	Loosely Coupled with internal code (e.g. internal REST APIs)
RAWGraphs	Apache-2.0	data visualisation	Standalone Tool
Apache Superset	Apache-2.0	data exploration, dashboards & data visualisation	Standalone Tool
Pandoc	GPL-2.0	convert between many different document types	Standalone Tool
ImageMagick	ImageMagick (GPL-3.0 compatible)	convert between different image formats	Standalone Tool
Text Editor Anywhere	Freeware	use emacs to edit text in any text field	Standalone Tool
PlantUML	GPL-3.0	create diagrams using text descriptions	Standalone Tool

Then export to components.csv

EXPORT EXPORT EXPORT

EXPORT EXPORT EXPORT

Variables

Platform-specific variables & secrets

Use #+CALL: once again to import our variables defined in vars-platform.csv

IMPORT IMPORT IMPORT

IMPORT IMPORT IMPORT

Define the environment variables and their desired values in the table

• note that for AppVeyor some of these are defined in the UI as secrets, but when we run the script locally we will need to securely collect these from the user
• Remember not to include a $ before the variable name in the var column of the table. The New-Variable command will add this in upon execution
• Important to specify assign or execute values, otherwise iex can cause undesired behaviour (e.g. trying to evaluate a path that doesn’t exist instead of assigning)

Then populate with the variable names, which will be executed by Invoke-Expression (aka iex).

type	exec	var	appveyor	local	local-gs	testing	comment
normal	assign	env:propositumLocation	C:\propositum	C:\propositum	H:\propositum	C:\propositum-test	The git clone location of the propositum repo
normal	execute	env:propositumDrv	$env:propositumDrv	(& {if(($result = Read-Host ‘Please provide a letter for the Propositum root drive (default is ‘P’).’) -eq ‘’){‘P:’}else{$result.Trim(‘;’)+’:’}})	(& {if(($result = Read-Host ‘Please provide a letter for the Propositum root drive (default is ‘P’).’) -eq ‘’){‘P:’}else{$result.Trim(‘;’)+’:’}})	(& {if(($result = Read-Host ‘Please provide a letter for the Propositum root drive (default is ‘P’).’) -eq ‘’){‘P:’}else{$result.Trim(‘;’)+’:’}})	The drive letter $propositumLocation will map to
secure	execute	env:githubApiToken	$env:githubApiToken	(& {Read-Host -AsSecureString ‘Please provide your GitHub token.’})	(& {Read-Host -AsSecureString ‘Please provide your GitHub token.’})	(& {Read-Host -AsSecureString ‘Please provide your GitHub token.’})	API Token for interaction with GH (not currently used in non-AppVeyor builds)
secure	execute	env:supersetPassword	$env:supersetPassword	(& {Read-Host -AsSecureString ‘Please provide a password for the Superset user ‘Propositum’.’})	(& {Read-Host -AsSecureString ‘Please provide a password for the Superset user ‘Propositum’.’})	(& {Read-Host -AsSecureString ‘Please provide a password for the Superset user ‘Propositum’.’})	The password for the propositum user for the superset application

Then export to vars-platform.csv

EXPORT EXPORT EXPORT

EXPORT EXPORT EXPORT

We need to define a few key paths and other variables which will be referred to regularly throughout the coming scripts, but are not platform specific.

Let’s import these from vars-other.csv

IMPORT IMPORT IMPORT

IMPORT IMPORT IMPORT

Then lets define them in a simplified table

type	exec	var	value	comment
hsh-tbl	execute	propositum	@{}	Initialises the hash table
hsh-itm	execute	propositum.root	$env:propositumDrv+”"	Propositum root folder
hsh-itm	execute	propositum.apps	$env:propositumDrv+”\apps”	Propositum apps folder (scoop root)
hsh-itm	execute	propositum.home	$env:propositumDrv+”\home”	Propositum home folder (dotfiles & projects)
hsh-itm	execute	propositum.font	$env:propositumDrv+”\font”	Propositum fonts folder
env-var	execute	env:HOME	$propositum.home	Sets env-var home to propositum home
env-var	execute	env:SCOOP	$propositum.root	Sets scoop home to the propositum root (creates ‘apps’ folder)

Note: The type column here is important, particularly hsh-itm & env-var.

Finally, export the table back to csv

EXPORT EXPORT EXPORT

EXPORT EXPORT EXPORT

Set mode & determine build platform

Add a variable to allow us to switch to testing / development mode - this will use the variable assignments in the “testing” column when we come to our Variables.

$testing = $false

Figure out if the script is being run from a local machine, from gs machine or on appveyor, or if we’re testing/debugging

$buildPlatform = if ($env:APPVEYOR) {"appveyor"}
elseif ($testing) {"testing"} # For debugging locally
elseif ($env:computername -match "NDS.*") {"local-gs"} # Check for NDS
else {"local"}

Initialise Environment

Ensure the necessary tooling is in place & prepare the build environment.

Start in the Script Root

Make sure we start in the script root to avoid issues with executing in the wrong directory & to ensure we can access any scripts or data structures that we need to import.

cd $PSScriptRoot

Console formatting

Turn the PowerShell background color to Black to make blue output from commands easier to read

$Host.UI.RawUI.BackgroundColor = ($bckgrnd = 'Black')

Helper functions

Define helper functions to perform repetitive activities

Path-CheckOrCreate: Check for path and optionally create dir or symlink

Check if a dir exists, and if specified, create the directory (or symlink)

function Path-CheckOrCreate {

# Don't make parameters positionally-bound (unless explicitly stated) and make the Default set required with all
[CmdletBinding(PositionalBinding=$False,DefaultParameterSetName="Default")]

    # Define Parameters incl. defaults, types & validation
    Param(
        # Allow an array of strings (paths)
        [Parameter(Mandatory,ParameterSetName="Default")]
        [Parameter(Mandatory,ParameterSetName="CreateDir")]
        [Parameter(Mandatory,ParameterSetName="CreateSymLink")]
        [string[]]$paths,

        # Parameter sets to allow either/or but not both, of createDir and createSymLink. createSymLink is an array of strings to provide the option of matching with multiple paths.
        [Parameter(ParameterSetName="CreateDir",Mandatory=$false)][switch]$createDir,
        [Parameter(ParameterSetName="CreateSymLink",Mandatory=$false)][string[]]$createSymLink = @() # Default value is an empty array to prevent 'Cannot index into null array'
   )

    # Create Arrs to collect the directories that exist/don't exist
    $existing = @()
    $notExisting = @()
    $existingSymLink = @()
    $notExistingSymLink = @()
    $createdDir = @()
    $createdSymLink = @()

    # Loop through directories in $directory
    for ($i = 0; $i -ne $paths.Length; $i++)
    {

        # If exists, add to existing, else add to not existing
        if (Test-Path $paths[$i]) {$existing += , $paths[$i]}
        else {$notExisting += , $paths[$i]}

        # If any symlinks have been provided, also do a check to see if these exist
        if ( ($createSymLink[$i]) -and (Test-Path $createSymLink[$i]) )
        {$existingSymLink += , $createSymLink[$i]}
        else {$notExistingSymLink += , $createSymLink[$i]}

        # Next, check if valid path
        if (Test-Path -Path $paths[$i] -IsValid)
        {
            # If user wants to create the directory, do so
            if ($createDir)
            {
                if (mkdir $paths[$i]) {$createdDir += , $paths[$i]}
            }
            # If user wants to create a symbolic link, do so
            elseif ($createSymlink)
            {
            if(New-Item -ItemType SymbolicLink -Value $paths[$i] -Path $createSymLink[$i]) # Use the counter to select the right Symlink value
                {$createdSymLink += , $createSymLink[$i]}
            }
        }
        else {Throw "An error occurred. Check the path is valid."}

    }

    # Write summary of directory operations to console [Turned off as annoying to see each time the command is run]
    #Write-Host "`n==========`n"
    #Write-Host "`n[Summary of Directory Operations]`n"
    #Write-Host "`nDirectories already exist:`n$existing`n"
    #Write-Host "`nDirectories that do not exist:`n$notExisting`n"
    #Write-Host "`nDirectories created:`n$createdDir`n"
    #Write-Host "`nSymbolic Links created:`n$createdSymLink`n"
    #Write-Host "`n==========`n"
  
    # Create a hash table of arrs, to access a given entry: place e.g. ["existing"] at the end of the expression
    # to get the arr value within add an index ref. e.g. ["existing"][0] for the first value within existing dirs
    $result = [ordered]@{
        existing = $existing
        existingSymLinks = $existingSymLink
        notExisting = $notexisting
        notExistingSymLinks = $notExistingSymLink
        createdDirs = $createdDir
        createdSymLinks = $createdSymLink
    }
  
    # Write results to the console
    Write-Host "`n================================="
    Write-Host "[Summary of Directory Operations]"
    Write-Host "=================================`n"
    Write-Host ($result | Format-Table | Out-String)
  
    return $result

}

GitHub-CloneRepo: Clone GitHub repo

function Github-CloneRepo ($opts, $compValsArr, $cloneDir) {
Write-Host ("Cloning ... [ "+"~"+$compValsArr.user+"/"+$compValsArr.repo+" ]") -ForegroundColor Yellow -BackgroundColor Black
$cloneUrl = ("https://github.com/"+$compValsArr.user+"/"+$compValsArr.repo)
iex "git clone $opts $cloneUrl $cloneDir"
}

Import functions & variables

Import functions

Let’s import the helper functions we defined earlier. Using the . notation means they will be imported with access to the variables in the current script scope.

. ./propositum-helper-fns.ps1
  

Import platform-specific variables

We can now import vars-platform.csv we created earlier into PowerShell

Try
{
    $platformVars = Import-CSV "vars-platform.csv"
}
Catch
{
    Throw "Check the CSV file actually exists and is formatted correctly before proceeding."
    $error[0]|format-list -force
}
  

Finally, set each of the platform variables according to $buildPlatform

• Select is used to first narrow the PSObject to the column containing the variable name, and the column matching our buildPlatform
• iex ensures that the value of each variable gets executed upon assignment, rather than being stored as a string
• the if statement is used in conjunction with the exec column as mentioned earlier to avoid incorrectly executing a value that should be assigned

ForEach ($var in $platformVars | Select 'var', $buildPlatform, 'exec') { # Narrow to required columns & $buildPlatform
    if ($var.var -like "env:*") { # If variable name contains 'env:'
        if ($var.exec -eq 'execute') {Set-Item -Path $var.var -Value (iex $var.$buildPlatform)}  # If we need to 'execute'
        else {Set-Item -Path $var.var -Value $var.$buildPlatform} # Else just assign
    }
    else { # Logic for non-environment variables
        if ($var.exec -eq 'execute') {New-Variable $var.var (iex $var.$buildPlatform) -Force}
        else {New-Variable $var.var $var.$buildPlatform -Force}
    }
}

Import other variables

Let’s import the vars-other.csv into PowerShell

Try
{
    $otherVars = Import-CSV "vars-other.csv"
}
Catch
{
    Throw "Check the CSV file actually exists and is formatted correctly before proceeding."
    $error[0]|format-list -force
}

$env: or environment variables are set in a different way to regular variable, therefore we need some additional logic to handle those. Similarly for hsh-itm entries, we don’t want to try to assign as variables but actually add the value to the corresponding hash table.

ForEach ($var in $otherVars) {
    if (($var.var -like "env:*") -or ($var.type -eq 'env-var')) { # If variable name contains 'env:', or is type 'env-var'
        if ($var.exec -eq "execute") {Set-Item -Path $var.var -Value (iex $var.value)} # If we need to 'execute'
        else {Set-Item -Path $var.var -Value $var.value} # Else just assign
    }
    elseif ($var.type -eq 'hsh-itm') { # Logic for hash table items
        $hsh = $var.var -split '\.' # Split the hash table item into a two-member array (note all hash table items must follow a hashtbl.keyname format)
        $hshtbl = iex ('$' + $hsh[0]) # Add '$' & define as hash table
        if ($var.exec -eq 'execute') {$hshtbl.add($hsh[1], (iex $var.value))}  # Add the key-value entry top the hash table: The first array entry is the hash table name, the second the name of the key
        else {$hshtbl.add($hsh[1], $var.value)}  # Same as above, but assign rather than invoke/execute the $var.value
    }
    else { # Logic for everything else (i.e. a regular variable)
        if ($var.exec -eq 'execute') {New-Variable $var.var (iex $var.value) -Force} 
        else {New-Variable $var.var $var.value -Force}
    }
}

Calling the $propositum variable should now give us a hash table of paths

$propositum | Format-Table | Out-String | Write-Host

Clear testing directory

To save some time, let’s also delete the contents of the testing directory when in testing mode.

We also add an additional condition to ensure that $propositumLocation has been set, otherwise we could end up deleting the root drive..

Note there’s currently a powershell bug that prevents this from working if any symlinks are contained within the directories.

if ($testing -and $env:propositumLocation) {Remove-Item ($env:propositumLocation+"\*") -Recurse -Force}

Map propositum drive letter & create folder structure

Mapping the propositum folder to a drive letter creates a short, intuitive path to key directories

subst $env:propositumDrv $env:propositumLocation
  

Now let’s use the hash table we defined earlier in Other variables, and loop through the paths; creating the directories where they don’t already exist

$createdDirs = Path-CheckOrCreate -Paths $propositum.values -CreateDir
  

Using the hash table of paths, we can now navigate to a given folder in the following manner

cd $propositum.root
  

Set TLS / SSL versions

This stops WebClient and other processes that require a secure connection from complaining if the connection requires a version other than TLS v1.0

[Net.ServicePointManager]::SecurityProtocol = "Tls12, Tls11, Tls, Ssl3"

Install and configure scoop

Install scoop

scoop is a bit like chocolatey but focused more on open source tools, and importantly, allows you to install apps as self-contained ‘units’, as well as creating handy manifests for your own apps / customm installs.

We already set the $env:SCOOP earlier in Other Variables so we can go ahead and install scoop to that path

iex (new-object net.webclient).downloadstring('https://get.scoop.sh')

Add extras bucket

Add the extras bucket which contains some additional free or open source applications outside of the scope of the main scoop repo

scoop bucket add extras

Add propositum bucket

Add the scoop propositum bucket which contains the JSON manifest files for installing and configuring the different propositum components.

scoop bucket add propositum 'https://github.com/xeijin/propositum-bucket.git'

Clone propositum repo

A number of required or source-controlled artifacts, including fonts, scripts and configuration files are already located in the propositum Repo, let’s fetch those first

# Hash table with necessary details for the clone command
$propositumRepo = [ordered]@{
    user = "xeijin"
    repo = "propositum"
}

# Clone the repo (if not AppVeyor as it is already cloned for us)
if(-not $buildPlatform -eq "appveyor"){Github-CloneRepo "" $propositumRepo $env:propositumLocation}

Build

Bring together the different components & create the final build artifact.

Install components

Use scoop to manage the installation of all components, including any dependencies as defined in the component’s manifest JSON.

Anything suffixed with a -p (for propositum) indicates a customised manifest, likely doing something fairly specialised.

Use a powershell array to define the components to install (and for better readability)

$propositumComponents = @(
    'cmder',
    'lunacy',
    'autohotkey',
    'miniconda3',
    'imagemagick',
    'knime-p',
    'rawgraphs-p',
    'regfont-p',
    'emacs-p',
    'texteditoranywhere-p',
    'superset-p',
    'pandoc',
    'latex',
    'plantuml'
)

Let the user know which components are being installed

$componentsToInstall = $propositumComponents -join "`r`n=> " | Out-String
Write-Host "`r`nThe following components will be installed:`r`n`r`n=> $componentsToInstall" -ForegroundColor Black -BackgroundColor Yellow

And Invoke-Expression to call the scoop installer with the array

Invoke-Expression "scoop install $propositumComponents"

Install & setup doom-emacs

Save the current path & navigate to the $propositum.home folder

Push-Location $propositum.home

Clone the doom-emacs repo as our .emacs.d folder and switch to the develop branch (master is out-of-date)

git clone https://github.com/hlissner/doom-emacs .emacs.d; cd .emacs.d; git checkout develop

Add the doom-emacs binaries folder to path

$doomBin = $propositum.home + "\.emacs.d\bin"
$env:Path = $env:Path + ";" + $doomBin

Then doom quickstart to install packages for a basic configuration (at least until my custom one is ready).

-y accepts all prompts to prevent AppVeyor build from hanging.

doom -y quickstart

Return to the original path

Pop-Location

Create private doom-config

Clean-up

Post-installation clean-up, primarily to reduce the overall size of the final build artifact.

Delete scoop cache

scoop cache rm *

Summarise Installation

Summarise via Scoop

Provide the user a summary of what was installed (including any dependencies installed automatically)

scoop list | Write-Host

Generate a list of the applications & versions installed and store in a text file. This can be used as a reference of what was installed & also as an importable ‘install’ file for Scoop.

Push-Location $propositum.apps
scoop export | Out-String > install-info.txt
Pop-Location

Create build artifact

Create the 7zip’d build artifact for later deployment to GitHub - this is the file unzipped on systems which require an ‘offline’ install (i.e. no access to external package repositories).

We only need to do this if running on AppVeyor.

if ($buildPlatform -eq "appveyor")
{
    echo "Compressing files into release artifact..."
    cd $propositum.root # cd to root, as 7z -v switch does not support specifying end file and directory 
    echo "Creating TAR archive..."
    iex "7z a -ttar -snl propositum.tar P:\" # Create tar archive to preserve symlinks
    echo "Compressing TAR into 7z archive..."
    iex "7z a -t7z propositum.tar.7z propositum.tar -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on -v1500m" # Compress tar into 7z archive 
    
    # Workaround for AppVeyor BinTray issue (only accepts .zip archives)
    if ($bintrayDeploy)
    {
    iex -verbose "7z a propositum.zip propositum.tar.7z*"
    }

}

Due to limitations with BinTray uploads, if binTrayDeploy is set to Yes we should additionally put the artifact into a zip for upload.

Deploy

Deploy the latest propositum release to GitHub.

Only attempt to deploy if the $buildPlatform is AppVeyor

if ($buildPlatform -eq "appveyor") {$deploy = $true}
else {$deploy = $false}

Upgrade

Upgrade an existing instance of propositum

TODO List

• [ ] tangles as a separate file propositum-upgrade.ps1
• [ ] should include the propositum-helper-fns.ps1
• [ ] should be able to run as a local user (not an admin)
• [ ] should be able to take the latest propositum artifact release from GitHub as an input
• [ ] should have a separate function that just updates configs (or perhaps a separate github release that is just the config info? e.g. updated .doom.d config file)

Finish

General clean-up and post-installation activities.

Generate post-install script

These are variables or commands that need to be set again post-installation. Note that we use org-babel’s <<NOWEB>> syntax here to import the variables from wherever they are defined.

This section has a :PROPERTIES: section that tangles to propositum-post-install.ps1 allowing that file to be included e.g. as a script upon launch of cmder (or just run as a one-off).

<<set-build-platform>>
<<collect-platform-vars>>
<<set-platform-vars>>
<<collect-other-vars>>
<<set-other-vars>>
<<set-scoop-env-var>>
<<map-propositum-drv>>
reg add HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Run /f /v "Propositum" /d "subst $propositumDrv $propositumLocation" # Add registry entry to map on startup
<<propositum-components-list>> 
$env:Path = $env:Path + ";" + "$propositum.root\shims"  # Add shims to path again so scoop & other commands available on command line
<<doom-bin-to-path>>
iex "scoop reset *" # Re-enables all scoop apps

For completeness, here is a script to remove the reigstry key added for mapping the propositum drive on startup

reg delete HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Run /f /v "Propositum" # Removes the registry entry to map propositum drive on startup

Archived

The code & sub-sections below have been archived as they are no longer in-use.