-
Notifications
You must be signed in to change notification settings - Fork 1
Fedora atomic
NOTE: manual pages are not available on RHEL Atomic, however they are available on RHEL Server systems.)
NOTE: Atomic is installed by the host provider on xfs filesystem.
The basis of the filesystem use in Docker is the storage backend abstraction. Each containers has two layers, one (called the "init" layer), which is based on an image layer and a child of that which contains the actual container content.
A lot of the stateful directories point to /var, while a lot of the non stateful directories point to /usr. This is by design as it is best to separate content that we don’t want to ever modify, with transient or runtime content.
1 - The Root partition is called root and is 3GB by default. root is a Logical Volume that contains the following:
-
The
/varand/etcdirectories. -
The
/ostree/repowhich contains the OSTree versions. -
The
/var/lib/docker/directory which contains container images data, such as temporary data or the docker volumes. A docker volume is a unit of storage that a running container can request from the host system. The unit of storage can be provided by another container but also by the host directly.
For the most part /var is read/write and /usr is read-only.
2 - A Container Image Partition called docker-pool. It is formatted as an LVM thin pool by the docker-storage-setup service. It is used to store the container images.
NOTE : Fedora Atomic Host comes bundled with a version of Docker based on this project atomic repo that moves no faster than the upstream Kubernetes project can abide (currently docker-1.13.1)
Docker uses /var/lib/docker as the default directory where all Docker related files, including the images, are stored. Atomic hosts however use direct LVM volumes via the devicemapper backend to store Docker images and metadata /dev/atomicos/docker-data and /dev/atomicos/docker-meta.
Docker suuports different storage drivers
We must be sure we are running in the direct-lvm mode and have LVM thin pools set up. This can be done using the docker-storage-setup utility.
LVM is a Logical Volume Manager for Linux Kernel.
Docker needs storage to store containers and container images. The default storage option for Docker on Fedora Atomic Host is an LVM thin pool.
We need to create thin pool volume. Thin Provisioning is used in lvm for creating virtual disks inside a thin pool. The docker-storage-setup service can assist you in setting up an LVM thin pool. When docker starts, it automatically starts docker-storage-setup.
By default, docker-storage-setup tries to find free space in the volume group backing the root volume and tries to setup a thin pool. If there is no free space in the volume group, docker-storage-setup will fail to set up an LVM thin pool and will fall back to using loopback devices.
Logical volumes can be thinly provisioned. This allows to create logical volumes that are larger than the available extents. Using thin provisioning, we can manage a storage pool of free space, known as a thin pool, which can be allocated to an arbitrary number of devices when needed by applications.
Below are steps to setup a new LVM thinpool storage.
NOTE :
-
here is the online manual for
container-storage-setup -
thin volumes are not supported across the nodes in a cluster. The thin pool and all its thin volumes must be exclusively activated on only one cluster node.
-
we can create our thin pool with LVM commands or use the helper script
container-storage-setupto configure the direct LVM storage. -
container-storage-setup without specifying a command defaults to using docker config files
/etc/sysconfig/docker-storage-setupfor input and/etc/sysconfig/docker-storagefor output. -
During boot, docker-storage-setup service reads the
/etc/sysconfig/docker-storagefile. We need to start and enable the service file.
1 - scan for already existing LVM partitions:
# lvmdiskscan
/dev/atomicos/root [ <19.00 GiB]
/dev/vda1 [ 1.00 GiB]
/dev/vda2 [ <19.00 GiB] LVM physical volume
/dev/vdb1 [ <100.00 GiB]
1 disk
2 partitions
0 LVM physical volume whole disks
1 LVM physical volume
# pvdisplay
--- Physical volume ---
PV Name /dev/vda2
VG Name atomicos
PV Size <19.00 GiB / not usable 1.98 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 4863
Free PE 0
Allocated PE 4863
PV UUID Ji8MSG-yiKd-ZSlF-cop0-siUg-SPSN-xMhEkx2 - create new physical volume
# pvcreate /dev/vdbNOTE: when running above command, you may be returned this error:
# pvcreate /dev/vdb
Device /dev/vdb excluded by a filter.It either means:
- there is a filter. Run
cat /etc/lvm/lvm.conf | grep filterand see if there are filetrs. - there is a partition or signature on the disk. Check with
gdiskand see if there is a MBR or a partition. In case of MBR, run# dd if=/dev/zero of=/dev/vdb bs=512 count=1. - run gdisk to remove everything
# gdisk /dev/vdb
p print
d delete partition if any
w write table
x expert mode
z destroy gpt data structure
y yes
q quit
# gdisk /dev/vdb
Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present- some extra commands:
vgextend atomicos /dev/vdb
vgreduce atomicos /dev/vdb1
pvremove /dev/vdb13 - create volume group
# vgcreate vg-docker /dev/vdb
# vgdisplay
--- Volume group ---
VG Name vg-docker
-------8<--------
# pvdisplay
--- Physical volume ---
PV Name /dev/vdb
VG Name vg-docker
PV Size <100.00 GiB / not usable 2.98 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 25599
Free PE 25599
Allocated PE 0
PV UUID Wblg0x-nsLd-1eyE-sZfA-jQfO-Ldvi-rUGdK1
-------8<--------4 - Edit the /etc/sysconfig/docker-storage-setup file. These settings will override default ones in /usr/share/container-storage-setup/container-storage-setup
# vim /etc/sysconfig/docker-storage-setup
VG=dockervg
GROWPART=enable
AUTO_EXTEND_POOL=enable
MIN_DATA_SIZE=8G
POOL_AUTOEXTEND_THRESHOLD=60
POOL_AUTOEXTEND_PERCENT=10
CONTAINER_ROOT_LV_NAME=dockerlv
CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker5 - stop docker
# systemctl stop docker6 - Remove any existing docker storage configuration file.
# rm -f /etc/sysconfig/docker-storage7 - Remove existing Docker location.
# rm -rf /var/lib/docker8 - Run the container-storage-setup configuration tool.
# container-storage-setup
INFO: Device /dev/vdb is already partitioned and is part of volume group atomicos
NOCHANGE: partition 2 is size 39843807. it cannot be grown
Physical volume "/dev/vda2" changed
1 physical volume(s) resized / 0 physical volume(s) not resized
NOCHANGE: partition 1 could only be grown by -33 [fudge=2048]
Physical volume "/dev/vdb1" changed
1 physical volume(s) resized / 0 physical volume(s) not resized
Rounding up size to full physical extent 124.00 MiB
Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.
Logical volume "docker-pool" created.
Logical volume atomicos/docker-pool changed.
# pvdisplay
--- Physical volume ---
PV Name /dev/vda2
VG Name atomicos
PV Size <19.00 GiB / not usable 1.98 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 4863
Free PE 0
Allocated PE 4863
PV UUID NQomBi-7W83-XyvX-bEYV-uhmz-Ot5o-UZpVSc
--- Physical volume ---
PV Name /dev/vdb1
VG Name atomicos
PV Size <100.00 GiB / not usable 2.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 25599
Free PE 0
Allocated PE 25599
PV UUID QhlEUm-MzPG-utLJ-jfUQ-exEK-IYaE-te41qN
# vgdisplay
--- Volume group ---
VG Name atomicos
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 62
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 1
Max PV 0
Cur PV 2
Act PV 2
VG Size 118.99 GiB
PE Size 4.00 MiB
Total PE 30462
Alloc PE / Size 30462 / 118.99 GiB
Free PE / Size 0 / 0
VG UUID ePxO4u-jNwJ-gUsa-MVJ1-1RQ1-JUxK-Lg0HLe
# mylsblk
NAME SIZE FSTYPE MOUNTPOINT LABEL
sr0 492K iso9660 config-2
vda 20G
├─vda1 1G ext4 /boot
└─vda2 19G LVM2_member
└─atomicos-root 19G xfs /sysroot
vdb 100G
└─vdb1 100G LVM2_member
├─atomicos-docker--pool_tmeta 124M
│ └─atomicos-docker--pool 99.8G
└─atomicos-docker--pool_tdata 99.8G
└─atomicos-docker--pool 99.8G 9 - Docker options. They can either be added in on file /etc/docker/daemon.json, or in many files in /etc/sysconfig
# ls -al /etc/sysconfig | grep docker
-rw-r--r--. 1 root root 875 Jul 4 15:29 docker
-rw-r--r--. 1 root root 56 Jul 4 15:29 docker-network
-rw-r--r--. 1 root root 230 Jul 6 11:03 docker-storage
-rw-r--r--. 1 root root 278 Jul 5 09:05 docker-storage-setup10 - restart docker
11 - verify
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 492K 0 rom
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 19G 0 part
└─atomicos-root 253:0 0 19G 0 lvm /sysroot
vdb 252:16 0 100G 0 disk
├─dockervg-docker--pool_tmeta 253:1 0 104M 0 lvm
│ └─dockervg-docker--pool 253:3 0 39.8G 0 lvm
├─dockervg-docker--pool_tdata 253:2 0 39.8G 0 lvm
│ └─dockervg-docker--pool 253:3 0 39.8G 0 lvm
└─dockervg-dockerlv 253:4 0 24G 0 lvm /var/lib/docker12 - Use the mount command to see the mount-point Docker is using:
# mount |grep devicemapper
36:/dev/mapper/atomicos-root on /var/lib/docker/devicemapper type xfs (rw,relatime,seclabel,attr2,inode64,noquota)Docker stores image and layer contents in the thinpool, and exposes them to containers by mounting them under subdirectories of /var/lib/docker/devicemapper/. The /var/lib/devicemapper/mnt/ directory contains a mount point for each image and container layer that exists.
NOTE :
- the
atomic storage modifycommand may be used for some operations described above.atomic storage resetwill clean all storages. - Docker documentation has a very good how-to for creating the thin pool volume without the helper script.
1- Under certain circumstances Docker may fail to start with the following error appearing in the logs:
Error initializing graphdriver: devmapper: Unable to take ownership of thin-pool (docker-pool) that already has used data blocks
This issue occurs when Docker tries to reuse a previously utilized LVM thin pool when using the devicemapper storage driver.
# systemctl stop docker.service
# rm -rf /var/lib/docker
# umount /var/lib/docker/containers
# lvremove /dev/atomicos/docker-pool
# gdisk /dev/vdb & remove everything2- Remove lost physical device from a group
If you see this message: Device for PV aQVolq-mbVY-PEf0-yig7-Hrny-gMfp-DkPoOb not found or rejected by a filter, we must remove the pv from the group.
# pvs
# vgs
# lvs -a -o +devices
# vgreduce --test vg_system --removemissing
# vgreduce vg_system --removemissing
check error is removed
# pvdisplay1 - Without this command, automatic extension does not occur even in the presence of the LVM profile.
# lvs -o+seg_monitor2 - monitor free space
root@control2➤➤ ~ # lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root atomicos -wi-ao---- <19.00g
[lvol0_pmspare] dockervg ewi------- 1020.00m
thinpool dockervg twi-a-t--- <95.00g 0.02 0.01
[thinpool_tdata] dockervg Twi-ao---- <95.00g
[thinpool_tmeta] dockervg ewi-ao---- 1020.00m 3 - LVM logs
% journalctl -fu dm-event.service4 - Set the storage option dm.min_free_space=10% to a value (representing a percentage) in /etc/docker.daemon.json. For instance, setting it to 10 ensures that operations fail with a warning when the free space is at or near 10%.
5 - If you reboot the host and find that the docker service failed to start, look for the error, “Non existing device”. You need to re-activate the logical volumes with this command:
# lvchange -ay docker/thinpool6 - Another feature of devicemapper is its use of snapshots (also sometimes called thin devices or virtual devices), which store the differences introduced in each layer as very small, lightweight thin pools.
!! don't work!!
When the Root Partition has run out of space, we can extend it to use free space in volume group.
# lvextend -r -L +3GB /dev/atomicos/rootAny kind of write-heavy load inside a container (such as databases or large logs) should generally be done to a volume. A volume is a plain directory from the host mounted into the container, which means it has none of the overhead that the storage backends may have. It also means you can easily access the data from a new container if you update the image, or if you want to access the same data from multiple concurrent containers.
The feature that allows for adding new packages to a system is known as package layering. Layering a package is achieved by executing the rpm-ostree install subcommand. It is necessary to reboot after you layer a package (required to boot into the new deployment).
To reinstall all uninstalled base packages, run:
# rpm-ostree override reset --allLocal packages are the ones installed from rpm files. To remove them, we need to specify the whole name, i.e package + version.
# rpm-ostree uninstall docker-ce docker-ce-selinux
error: Package/capability 'docker-ce' is not currently requested
# rpm-ostree uninstall docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch To workaround the need to reboot, an experimental feature known as livefs has been introduced. We tell rpm-ostree that we want to go ahead and apply the updates to the live running system with the following command run after adding a package:
# rpm-ostree ex livefsNOTE: ex is for experimental
Another experimental feature of rpm-ostree is the ability to remove packages that were delivered with the base OSTree.
# rpm-ostree override remove MyPackageError when running install command (below is example):
# rpm-ostree install MyPackage
-------8<--------
Resolving dependencies... Forbidden base package replacements:
selinux-policy-targeted 3.14.1-29.fc28 -> 3.14.1-30.fc28 (updates)
selinux-policy 3.14.1-29.fc28 -> 3.14.1-30.fc28 (updates)
failed
error: Some base packages would be replacedThis is basically because the content in the current deployment is a bit out of date with what is in the updates yum repos. In this case, we need to run:
# rpm-ostree rebase fedora/28/x86_64/updates/atomic-host- check for updates without applying the update:
rpm-ostree upgrade --check- list package installed components:
# dnf repoquery -l MyPackageWe can create a docker image containing whatever we want to run (even if it's the docker daemon itself) then run that container using atomic tool with ostree/trunc backend (without docker daemon):
# atomic run --storage ostree registry.fedoraproject.org/fedora-minimal /bin/bash
bash-4.4#
bash-4.4# exit
#Here is the list of registry.
atomic run attempts to start an existing container if a container name is
specified, or execute container image run method if an image name is
specified. Defaults to the following command, if image does not specify LABEL
run: docker run -i --name ${NAME} ${IMAGE}
When installing a package already installed, a message will always appear when installing/uninstalling/upgrading: Inactive requests: MyPackage (already provided by ...)
To remove this message, run rpm-ostree uninstall MyPackage. It will not uninstall but remove it from the list of packages to install (inactive request).
Pod management is done with the podman command. Libpodprovides a library for applications looking to use the Container Pod concept, popularized by Kubernetes. Libpod also contains the Pod Manager tool (Podman). Podman manages pods, containers, container images, and container volumes.
The podman command can run and manage containers and container images. It supports most of the same features and command options you find in the docker command, with the main differences being that podman doesn’t require the docker service or any other active container runtime for the command to work.
The podman command lets you run containers as standalone entities, without requiring that Kubernetes, the Docker runtime, or any other container runtime be involved. It is a tool that can act as a replacement for the docker command, implementing the same command-line syntax, while it adds even more container management features.
OSTree is an upgrade system for Linux-based operating systems that performs atomic upgrades of complete filesystem trees. It is not a package system; rather, it is intended to complement them. A primary model is composing packages on a server, and then replicating them to clients. The underlying architecture might be summarized as git for operating system binaries.
OSTree supports exactly two persistent writable directories that are preserved across upgrades: /etc and /var.
Directroies for the machine can be found in /sysroot/ostree/deploy/fedora-atomic.
Below command show last changes, with new tree showing first. We see that we added a local package: sshguard-2.1.0-1.el7.lux.x86_64sshguard
The layeredPackages are the ones we layered, or installed.
root@control2➤➤ ~ # rpm-ostree status
State: idle; auto updates disabled
Deployments:
● ostree://fedora-atomic:fedora/28/x86_64/atomic-host
Version: 28.20180527.0 (2018-05-27 19:05:29)
BaseCommit: 291ea90da29bc5abe757b5a50813b3de1396b08412939a89b3b671aba9856093
GPGSignature: Valid signature by 128CF232A9371991C8A65695E08E7E629DB62FB1
LayeredPackages: bind-utils byacc flex gcc gdisk git git-all git-subtree glibc-langpack-en jq kubernetes-client kubernetes-node libmodulemd libxcrypt-devel mcstrans nano nmap ntp ntpstat pam_ssh pass perl-MailTools python3-smartcols ranger rng-tools rpmconf rsyslog rxvt-unicode-256color selinux-policy-devel setools
-------8<--------
LocalPackages: sshguard-2.1.0-1.el7.lux.x86_64
ostree://fedora-atomic:fedora/28/x86_64/atomic-host
Version: 28.20180527.0 (2018-05-27 19:05:29)
BaseCommit: 291ea90da29bc5abe757b5a50813b3de1396b08412939a89b3b671aba9856093
GPGSignature: Valid signature by 128CF232A9371991C8A65695E08E7E629DB62FB1
LayeredPackages: bind-utils byacc flex gcc gdisk git git-all git-subtree glibc-langpack-en jq kubernetes-client kubernetes-node libmodulemd
libxcrypt-devel mcstrans nano
-------8<--------# ostree admin status
* fedora-atomic dced1fb28e15799c1e025c5110b51b5cb73d6124958953095d858a70bfa35210.0
Version: 28.20180527.0
origin: <unknown origin type>
fedora-atomic 943490d082982fa340acc3ab89774f24481f0606f2bf7766315a2fd9ed5fa607.0 (rollback)
Version: 28.20180527.0
origin: <unknown origin type># rpm-ostree db list 291ea90
ostree commit: 291ea90 (291ea90da29bc5abe757b5a50813b3de1396b08412939a89b3b671aba9856093)
GeoIP-1.6.12-3.fc28.x86_64
GeoIP-GeoLite-data-2018.04-1.fc28.noarch
NetworkManager-1:1.10.8-1.fc28.x86_64
NetworkManager-libnm-1:1.10.8-1.fc28.x86_64
NetworkManager-team-1:1.10.8-1.fc28.x86_64
acl-2.2.52-20.fc28.x86_64
-------8<-------- # ostree admin config-diff
M machine-id
M subgid
M subuid
M localtime
-------8<--------
A lvm/backup/atomicos
A sysconfig/network-scripts/ifcfg-eth0
A sysconfig/authconfig
A sysconfig/network
A sysconfig/anacondaThe M means Modified. An A would mean added, while a D would mean the file had been deleted.
Below command will show what has been modified in /etc/sysconfig
# ostree admin config-diff | grep sysconfigThe "canonical" way is to use cloud-init for setting up auth, and then ansible for configuration. The first step would be to convert customizations to an ansible playbook.
One of the primary benefits to Atomic Host and OSTree is the ability to configure once, deploy many times using custom OSTree images.
The procedure and playbook below enable to create our own Atomic Host OSTree image. This is the first step in creating our own distributions of Atomic Host to install on all cloud servers
1 - install packages
# rpm-ostree install git python1 - directory setup
We choose var/lib/docker/devicemapper (or dockerroot as configured in storage setup)
2 - download needed files
# cd var/lib/docker/devicemapper
# mkdir {myostree,repo,cache}
# cd myostree
# git clone https://pagure.io/fedora-atomic.git -b f283 - initiate a repo
# ostree --repo=repo init --mode=archive-z2
# ls repo
extensions/ objects/ refs/ state/ tmp/ config4 - compose a tree
rpm-ostree compose tree --cachedir=cache --repo=repo fedora-atomic/fedora-atomic-host.jsonThe treecompose process will now begin. First it parses all the input files and then downloads the required RPMs that will end up in the tree. It will then end up installing those RPMs into the tree and running a SELinux relabel on it.
# ostree fsck repo=repo
Validating refs...
Enumerating objects...
Verifying content integrity of 485 commit objects...
fsck objects (77647/77647) [=============] 100%
# ostree refs --repo=repo
fedora/28/x86_64/atomic-host
# ostree log --repo=repo fedora/28/x86_64/atomic-host
commit a759d4f33f5e6a32cf62eeb5a0da4dac4253b5c0e6407fa69574ded670bb079b
ContentChecksum: a52031211877e0b1c29674953feb2e0325c213ee7a60abc3b70537a56cd27c88
Date: 2018-06-07 07:51:10 +0000
Version: 28
(no subject)We’ll first create a systemd unit for a service that runs the compose, and a timer unit.
/etc/systemd/system/atomic-updates-rpm-ostree-compose.service
-------------------------------------------------------------
[Unit]
Description=run rpm-ostree compose for Fedora atomic
[Service]
Type=oneshot
ExecStart=/usr/bin/rpm-ostree compose tree --repo=${WORKING_DIR}/repo --cache=${WORKING_DIR}/cache ${WORKING_DIR}/fedora-atomic/fedora-atomic-host.json
[Install]
/etc/systemd/system/atomic-updates-rpm-ostree-compose.timer
-------------------------------------------------------------
[Unit]
Description=run rpm-ostree compose daily
[Timer]
OnCalendar=02:00
Persistent=true
Unit=atomic-updates-testing-rpm-ostree-compose.service
[Install]
WantedBy=timers.targetBy default, there always have a previous tree (an operating system snapshot) installed. So if something goes wrong, we can always fall back to the previous tree.
We will set up a build server and host the contents
- Fedora atomic tree matser
- Atomic 101 lab
- Atomic basic setup
- openstack HA guide
- red Hat Atomic documentation
- Atomic host Automatic build
- Fedora magazine build Fedora atomic
- Atomic host ansible automation
- Openshift HA cluster
- Custom OSTree
- Thin provisioning in centos & rhel7
- Red hat thin pool creation
- lvm thinpool
- setting up storage
- ATS garage
- Growroot
- Fedora magazine: build fedora atomic
- Projet atomic: build your own tree
- Red hat developper: custom atomic trees