Apache Spark Standalone Cluster on Docker

The project was featured on an article at MongoDB official tech blog! 😱

The project just got its own article at Towards Data Science Medium blog! ✨

Introduction

This project gives you an Apache Spark cluster in standalone mode with a JupyterLab interface built on top of Docker. Learn Apache Spark through its Scala, Python (PySpark) and R (SparkR) API by running the Jupyter notebooks with examples on how to read, process and write data.

TL;DR

curl -LO https://raw.githubusercontent.com/cluster-apps-on-docker/spark-standalone-cluster-on-docker/master/docker-compose.yml
docker-compose up

Contents

• Quick Start
• Tech Stack
• Metrics
• Contributing
• Contributors
• Support

Quick Start

Cluster overview

Application	URL	Description
JupyterLab	localhost:8888	Cluster interface with built-in Jupyter notebooks
Spark Driver	localhost:4040	Spark Driver web ui
Spark Master	localhost:8080	Spark Master node
Spark Worker I	localhost:8081	Spark Worker node with 1 core and 512m of memory (default)
Spark Worker II	localhost:8082	Spark Worker node with 1 core and 512m of memory (default)

Prerequisites

• Install Docker and Docker Compose, check infra supported versions

Download from Docker Hub (easier)

1. Download the docker compose file;

curl -LO https://raw.githubusercontent.com/cluster-apps-on-docker/spark-standalone-cluster-on-docker/master/docker-compose.yml

2. Edit the docker compose file with your favorite tech stack version, check apps supported versions;
3. Start the cluster;

docker-compose up

4. Run Apache Spark code using the provided Jupyter notebooks with Scala, PySpark and SparkR examples;
5. Stop the cluster by typing ctrl+c on the terminal;
6. Run step 3 to restart the cluster.

Build from your local machine

Note: Local build is currently only supported on Linux OS distributions.

1. Download the source code or clone the repository;
2. Move to the build directory;

cd build

3. Edit the build.yml file with your favorite tech stack version;
4. Match those version on the docker compose file;
5. Build up the images;

chmod +x build.sh ; ./build.sh

6. Start the cluster;

docker-compose up

7. Run Apache Spark code using the provided Jupyter notebooks with Scala, PySpark and SparkR examples;
8. Stop the cluster by typing ctrl+c on the terminal;
9. Run step 6 to restart the cluster.

Tech Stack

• Infra

Component	Version
Docker Engine	1.13.0+
Docker Compose	1.10.0+

• Languages and Kernels

Spark	Hadoop	Scala	Scala Kernel	Python	Python Kernel	R	R Kernel
3.x	3.2	2.12.10	0.10.9	3.7.3	7.19.0	3.5.2	1.1.1
2.x	2.7	2.11.12	0.6.0	3.7.3	7.19.0	3.5.2	1.1.1

• Apps

Component	Version	Docker Tag
Apache Spark	2.4.0 | 2.4.4 | 3.0.0	<spark-version>
JupyterLab	2.1.4 | 3.0.0	<jupyterlab-version>-spark-<spark-version>

Metrics

Image	Size	Downloads
JupyterLab
Spark Master
Spark Worker

Contributing

We'd love some help. To contribute, please read this file.

Contributors

A list of amazing people that somehow contributed to the project can be found in this file. This project is maintained by:

André Perez - dekoperez - [email protected]

Support

Support us on GitHub by staring this project ⭐

Support us on Patreon. 💖