PaSh: Light-touch Data-Parallel Shell Processing

A system for parallelizing POSIX shell scripts. Hosted by the Linux Foundation.

Service	Main	Develop
Tests
Build
Pages

Quick Jump: Running PaSh | Installation | Testing | Repo Structure | Community & More | Citing

Running PaSh

To parallelize, say, ./evaluation/intro/hello-world.sh with parallelization degree of 2× run:

./pa.sh ./evaluation/intro/hello-world.sh

Run ./pa.sh --help to get more information about the available commands. Jump to docs/tutorial for a longer tutorial.

Installation

On Ubuntu, Fedora, and Debian run the following to set up PaSh.

wget https://raw.githubusercontent.com/binpash/pash/main/scripts/up.sh
sh up.sh
export PASH_TOP="$PWD/pash/"
## Run PaSh with echo hi
"$PASH_TOP/pa.sh" -c "echo hi"

For more details, manual installation, or other platforms see installation instructions.

Repo Structure

This repo hosts the core pash development. The structure is as follows:

• compiler: Shell-dataflow translations and associated parallelization transformations.
• docs: Design documents, tutorials, installation instructions, etc.
• evaluation: Shell pipelines and example scripts used for the evaluation.
• runtime: Runtime component — e.g., eager, split, and associated combiners.
• scripts: Scripts related to continuous integration, deployment, and testing.

Community & More

Chat:

• Discord Server (Invite)

Mailing Lists:

• pash-devs: Join this mailing list for discussing all things pash
• pash-commits: Join this mailing list for commit notifications

Development/contributions:

• Contribution guide: docs/contributing
• Continuous Integration Server: ci.binpa.sh

Citing

If you used PaSh, consider citing the following paper:

@inproceedings{pash2021eurosys,
author = {Vasilakis, Nikos and Kallas, Konstantinos and Mamouras, Konstantinos and Benetopoulos, Achilles and Cvetkovi\'{c}, Lazar},
title = {PaSh: Light-Touch Data-Parallel Shell Processing},
year = {2021},
isbn = {9781450383349},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3447786.3456228},
doi = {10.1145/3447786.3456228},
pages = {49–66},
numpages = {18},
keywords = {POSIX, Unix, pipelines, automatic parallelization, source-to-source compiler, shell},
location = {Online Event, United Kingdom},
series = {EuroSys '21}
}