The Container Ecosystem Project

The ecosystem of awesome new technologies emerging around containers and microservices can be a little overwhelming, to say the least. We thought we might be able to help: welcome to the Container Ecosystem Project. The goals of this project are (1) to clearly lay out the different types technologies that make up the growing container ecosystem and the microservices technology stack – starting from the lowest levels of core container technology, and rising up through layers of abstraction to full-blown container platforms and support tools – and (2) to put forth the latest and greatest examples of each type of technology.

This project is a living document - please submit edits/pull requests for suggested changes.

View this project as a webpage here:
http://www.sysdig.com/the-container-ecosystem-project/

Table of Contents

• The Container Ecosystem
  • Core Container Technologies
    • Container specifications
    • Container runtimes
    • Container management
    • Container definition
    • Registries
    • Operating systems
    • VM management
  • Distributed Container Technologies
    • Scheduling
    • Cluster definition
    • Service discovery / Distributed configuration storage
    • Dynamic configuration management
  • Container Platform Technologies
    • Container orchestration platform
    • Hosted container platform
    • Container platform management
    • Container-based PaaS
  • Container-Native Support Technologies
    • Networking
    • Monitoring / Visibility
    • Data layer
    • CI/CD
    • Security
    • Getting started aides
• About the Container Ecosystem Project
• Further Reading

Core Container Technologies

Use these tools to run a small number of containers on a single host

Container specifications

An abstract definition of a standard "container", allowing an ecosystem of technologies to support a standard container with potentially multiple, interchangeable runtime implementations

• Docker open source

  • Open Container spec: open industry standard for container runtimes; supported by Docker, CoreOS, and most industry leaders; backed by the Open Container Initiative (OCI) (run by the Linux Foundation); currently absorbing CoreOS's AppC standard

• CoreOS open source

  • AppC (deprecated): CoreOS is now supporting the OCI

Container runtimes

This is your actual running container (essentially an abstraction of Linux kernel components like namespaces and cgroups that allow virtualization on top of a shared kernel)

• Docker open source

  • runc: Docker's container runtime, now donated to the OCI as the initial implementation of the standard; essentially a repackaging of libcontainer
  • libcontainer: a Linux container library; enables and abstracts interactions with Linux kernel components to create and control containers

• CoreOS open source

  • rkt: CoreOS's container runtime; initially an implementation of the AppC specification, which is now being rolled into the OCI spec

• Other open source

  • LXC: a Linux container library; originally utilized by runc until release of libcontainer
  • OpenVZ: a Linux container library

Container management

These tools abstract low level control of your container runtime adding further functionality and usability

• Docker open source

  • Docker Engine (aka "Docker"): the core of Docker and its primary interface; creates and runs Docker containers; includes:
    • Docker daemon: runs as a process on the host machine and provides an API that abstracts basic container control functions
    • Docker client: a CLI for interacting with the Docker daemon

• CoreOS open source

  • rkt CLI: rkt's container management functionality is delivered on-demand by a binary, rather than a daemon background process

• Other open source

  • LXD: daemon and UI for LXC
  • libvirt: container and virtualization mgmt library that supports LXC, OpenVZ, and a variety of hypervisor technologies

Container definition

These tools allow you to define specific containers, so they can be saved, shared and reproduced

• Docker open source

  • Docker image: a template representing a fully configured container; Docker container runtimes are created from these images; images are created with Dockerfiles and shared over registries
  • Dockerfile: text file containing all the commands needed to build a Docker image

• CoreOS open source

  • ACI (App Container Image): rkt's native container image format (note, rkt also supports Docker images)

Registries

Repositories for storing and sharing container images

• Docker open source

  • Docker Registry: open source Docker image registry that can be hosted in your own environment

• Commercial

  • Hosted
    • Amazon EC2 Container Registry (ECR): still in beta
    • Docker Hub: hosted registry with free paid tiers, private public repositories, and a collection of "official" images
    • Google Container Registry
    • Quay.io: CoreOS's hosted registry
  • On-premise
    • Docker Trusted Registry
    • CoreOS Enterprise Registry

Operating systems

OS's that are designed for hosting containers

• Docker open source

  • boot2docker (basically deprecated by Docker Machine): minimalist Linux for running Docker on PC and Mac in a VM; now used by Docker Machine in certain environments

• CoreOS open source

  • CoreOS: minimalist OS built for running distributed, containerized apps; includes etcd and fleet

• Other open source

  • RancherOS: minimalist, fully containerized OS
  • Project Atomic: minimalist Red Hat Linux; versions include RHEL Atomic, CentOS Atomic, and Fedora Atomic
  • Ubuntu Core "Snappy": minimalist Ubuntu
  • SmartOS: Solaris-based OS from Joyent that includes Zones (ie. Solaris containers)
  • Photon OS: minimalist OS from VMWare

VM management

These tools help you manage the host virtual environments in which you run your containers

• Docker open source

  • Docker Machine: creates and manages host VMs running Docker, including local VMs (eg. VirtualBox) and cloud VMs (eg. Amazon AWS, Google GCP)

• Other open source

  • Hashicorp Vagrant: creates pre-configured VMs for dev environments based on a variety of "Providers" (virtualization technologies) including Docker containers
  • Hashicorp Otto: extends Vagrant to deploy and manage VMs across many platforms

Distributed Container Technologies

Use these technologies to run applications on a distributed cluster of containers

Scheduling

These tools manage placement of new containers across abstracted underlying resources

• Docker open source

  • Docker Swarm: designed to extend Docker API to a cluster; includes scheduling and service discovery

• CoreOS open source

  • fleet: low level orchestration included in CoreOS; supports basic scheduling; can be used to bootstrap Kubernetes for higher level orchestration

• Other open source

  • Chronos: framework for scheduling on Mesos

Cluster definition

These tools allow you to define and manage a cluster of dependent containers as a single composable entity

• Docker open source

  • Docker Compose: text files used to define and configure a distributed application across a cluster of Docker containers

• CoreOS open source

  • fleet unit file: fleet uses a specialized version of systemd unit files to define a distributed application across containers

Service discovery / Distributed configuration storage

These tools allow applications within different containers to discover each other and share configuration information (eg. IP addresses or application settings); usually implemented as a globally distributed key-value store

• Docker open source

  • Docker Swarm comes with built in service discovery, but can also use etcd, Consul, Zookeeper

• CoreOS open source

  • etcd: globally distributed key-value store; included with CoreOS for service discovery

• Other open source

  • Marathon: framework for initializing long running jobs on Mesos; includes service discovery and cluster management functionality
  • Hashicorp Consul: service discovery, key/value store, and cluster health checking; uses Serf
  • Apache ZooKeeper: globally distributed key-value store

Dynamic configuration management

These tools let you dynamically update application settings based on changes to your distributed key-value store in applications that don't natively support this

• CoreOS open source

  • confd: originally built for etcd, but now supports Consul and ZooKeeper

• Other open source

  • Consul Template: built natively for Consul

Container Platform Technologies

Use these technologies as complete platforms for running distributed applications across clusters of containers

Container orchestration platforms

These platforms include or abstract away all of the core functionality (listed above) needed for container cluster management ("orchestration"), including container management, scheduling, cluster definition, and service discovery

• Docker open source

  • Docker Swarm, Compose, and Machine can all run together to create a complete orchestration platform (still beta); Docker Swarm can also support more advanced orchestration tools like Kubernetes

• Other open source

  • Apache Mesos: mature, highly scalable service that abstracts a pool of underlying resources and distributes "tasks" (including Docker images) from various application frameworks; uses Marathon and Chronos to add cluster management, scheduling, and service discovery; also can support Kubernetes
  • Kubernetes: orchestration platform designed specifically for running microservices on clusters of containers; includes scheduling, cluster management and service discovery through abstractions such as "pods", "replication controllers (RCs)", and "services"; originally from Google, now donated to the CNCF
  • Hashicorp Nomad: uses Consul

Hosted container platforms

These platforms offer container hosting and orchestration as a service

• Commercial
  • Amazon EC2 Container Service (ECS)
  • Google Container Engine: uses Kubernetes
  • Docker Tutum: acquired by Docker; still beta
  • Redhat Openshift: uses Kubernetes
  • Joyent – Triton
  • Giant Swarm: still beta
  • ProfitBricks: still beta
  • Modulus
  • Rackspace Carina: still beta

Container platform management

These technologies add further abstracted management and control layers to distributed container environments, often through GUIs

• Docker open source

  • Project Orca: opinionated management GUI built on top of full stack of Docker technologies; still alpha

• Other open source

  • Rancher: still beta
  • ContainerShip
  • Panamax
  • Shipyard
  • Joyent SmartDataCenter: uses SmartOS

• Commercial

  • Mesosphere DCOS: uses Mesos
  • CoreOS Tectonic: uses CoreOS+Kubernetes; still beta
  • Nirmata: multi-cloud container management; built in scheduling, policy-based orchestration, service discovery, dynamic load balancing, and infrastructure optimization.
  • ContainerShip Enterprise: still beta
  • StackEngine
  • AppFormix

Container-based PaaS

These platforms further abstract container-based infrastructures by managing application code deployment and offering PaaS-like user experiences

• Other open source
  • Deis: container based PaaS; uses CoreOS
  • Flynn: container based PaaS; uses etcd
  • RedHat Openshift Origin
  • Cisco Mantl: uses Mesos
  • Dokku: minimalist PaaS
  • Empire: PaaS built for Amazon's ECS

Container-Native Support Technologies

Use these additional container-native tools to support your container-based infrastructure

Networking

• Docker open source

  • Docker port expose: Docker feature that links a container port to a host port
  • Docker linking: Docker feature offering a basic connection between containers on the same host
  • libnetwork: advanced container networking library (still "under heavy development")

• CoreOS open source

  • flannel: overlay network built using etcd that gives each host a separate subnet for its containers

• Other open source

  • Weave: overlay network that puts all containers in a distributed system onto a single virtual network; also includes service discovery functionality
  • Calico: layer 3 virtual network that provides each container with an IP address

Monitoring / Visibility

• Docker open source

  • Docker ps/top/stats: runtime commands
  • Docker stats API: remote API for streaming basic container metrics; utilized by the Docker Ecosystem Technology Partners for Monitoring

• Other open source

  • sysdig: CLI for deep system/containers visibility; includes curses-based "csysdig" interface
  • cAdvisor: basic container metrics exporter from Google; includes web GUI; Heapster adds Kubernetes support
  • Weave Scope: container network topologies

• Commercial

  • Sysdig Cloud: uses sysdig; includes web-based UI, application topologies, and support for all major container formats and orchestration platforms

Data layer

• Other open source
  • CusterHQ Flocker: data volume manager for running stateful services like databases in containers

Log management

• Docker open source

  • Docker logs: runtime command

• Other open source

  • logspout: log router for Docker containers

CI/CD

• Commercial
  • Shippable
  • Wercker

Security

• Other open source

  • OpenSCAP

• Commercial

  • Twistlock
  • Scalock
  • Conjur
  • Lynis

Getting started aides

• Docker open source
  • Docker Kitematic: basic Docker GUI designed for getting started with Docker
  • Docker Toolbox: installer for a package of core Docker tools

About the Container Ecosystem Project

Here at Sysdig, the container-native visibility company, we talk to a lot of people in the container ecosystem: both consumers and producers of technology. And wow, there is a LOT of cool technology out there – and so much more coming out all the time. It can be hard to keep up with, even if you're a seasoned expert, much less as a curious newcomer just trying to figure out where to start. There are plenty of great guides out there for various container technologies and use cases (see below for some links). But we had yet to find a clearly organized survey of the different core technologies that make up the container ecosystem and the typical microservices stack. So we decided to make one: the Container Ecosystem Project.

For each type of technology, we've tried to provide a brief description, as well as list examples currently available for that technology. We've separated out open source solutions from commercial offerings. Two of the leading open source container technology producers, Docker and CoreOS, each get their own sections. Throughout the doc, we've tried to mark beta technologies and parent technologies accordingly.

This framework is not, of course, a perfect science, and this will be a work in progress. If you have any suggested edits, please tweet us or submit edits/pull requests. We'll do our best to keep this document up to date and prune off deprecated or abandoned technologies as the ecosystem evolves.

Further Reading

• Docker ecosystem introduction from Digital Ocean: https://www.digitalocean.com/community/tutorial_series/the-docker-ecosystem
• Lists of Docker ecosystem technologies
  • https://www.mindmeister.com/389671722/runc-open-container-ecosystem
  • https://github.com/weihanwang/docker-ecosystem-survey
  • https://github.com/veggiemonk/awesome-docker
• Docker docs: https://docs.docker.com/
• CoreOS docs: https://coreos.com/docs/

Project created and maintained by Sysdig, the container-native visibility company.
http://www.sysdig.com/
http://www.sysdig.org/