Skip to content
This repository has been archived by the owner on Apr 25, 2023. It is now read-only.

Commit

Permalink
chore: Remove need for insecure registry
Browse files Browse the repository at this point in the history
  • Loading branch information
jimmidyson committed Oct 21, 2020
1 parent 5e7eb12 commit a6e6d4e
Show file tree
Hide file tree
Showing 7 changed files with 51 additions and 243 deletions.
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -166,4 +166,4 @@ controller-gen:
command -v controller-gen &> /dev/null || (cd tools && go install sigs.k8s.io/controller-tools/cmd/controller-gen)

deploy.kind:
DOCKER_PUSH=n KIND_LOAD_IMAGE=y FORCE_REDEPLOY=y ./scripts/deploy-kubefed.sh $(IMAGE_NAME)
KIND_LOAD_IMAGE=y FORCE_REDEPLOY=y ./scripts/deploy-kubefed.sh $(IMAGE_NAME)
79 changes: 0 additions & 79 deletions docs/environments/kind.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,7 @@
- [`kind` - `k`ubernetes `in` `d`ocker](#kind---kubernetes-in-docker)
- [Download and Install kind](#download-and-install-kind)
- [Create Clusters](#create-clusters)
- [Create Insecure Container Registry](#create-insecure-container-registry)
- [Configure Insecure Container Registry](#configure-insecure-container-registry)
- [Run Script](#run-script)
- [Delete Clusters](#delete-clusters)
- [Delete Insecure Container Registry](#delete-insecure-container-registry)
- [Run Script](#run-script-1)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

Expand All @@ -35,69 +30,6 @@ will be installed. Your `GOBIN` directory should be at `$(go env GOPATH)/bin`:

You can proceed to create clusters once you have `kind` available in your path.

### Create Insecure Container Registry

Please answer the following question to determine if you need to set up an
insecure container registry on your host:

1. Are you planning on creating container images locally without pushing to a
public container registry such as `quay.io`. For example, you can build your
own custom image e.g. `172.17.0.1:5000/<imagename>:<tag>`, as part of your
development workflow and push to this container registry . See the
[development guide](/docs/development.md#test-your-changes) for more
examples.

If you answered yes, then you will need to create an insecure container
registry. Creating a container registry is necessary if you want your kind
clusters to pull images that you built locally on your host without pushing
them to a public container registry. See the [docker
docs](https://docs.docker.com/registry) for more details.

In order to create an insecure container registry, you can pass the
`CREATE_INSECURE_REGISTRY` flag to `create-clusters.sh` as follows:

```bash
CREATE_INSECURE_REGISTRY=y ./scripts/create-clusters.sh
```

### Configure Insecure Container Registry

Please answer the following questions to determine if you need to configure an
insecure container registry on your host:

1. Is this the first time you're running the `create-clusters.sh` script?
2. Does your docker daemon need to be configured for an insecure container
registry?

If you answered yes to both of these questions, then you will need to configure
the docker daemon on your host for an insecure container registry. The reason
for an insecure registry is to simplify the container registry setup by not
enabling TLS. **This only needs to be done once for a particular host**.
See the [docker docs](https://docs.docker.com/registry) for more details.

In order to configure an insecure container registry, you can pass the
`CONFIGURE_INSECURE_REGISTRY_HOST` flag to `create-clusters.sh` as shown below. The
default container registry host is `172.17.0.1:5000` and needs to match
the IP address of the default docker bridge on your host, typically
`172.17.0.1`. If you would like to change this then set the
`CONTAINER_REGISTRY_HOST="<host>:<port>"` flag.

```bash
CONFIGURE_INSECURE_REGISTRY_HOST=y ./scripts/create-clusters.sh
```

This will automatically create the necessary dockerd daemon config and reload
the docker daemon for you. Keep in mind that it will **not** do this for you
if a config already exists, or your docker daemon is already configured with an
`--insecure-registry` command line option.

If you would like to manually make the changes to your docker daemon instead,
add `172.17.0.1:5000` as an insecure registry host and reload or restart your
docker daemon. See the [docker
docs](https://docs.docker.com/registry/insecure/) for more details.

### Run Script

Run the following command to create `2` `kind` clusters:

```bash
Expand All @@ -113,17 +45,6 @@ NUM_CLUSTERS=<num> ./scripts/create-clusters.sh

## Delete Clusters

### Delete Insecure Container Registry

Specify the `DELETE_INSECURE_REGISTRY` flag if you set up an insecure container
registry and would like to have it deleted.

```bash
DELETE_INSECURE_REGISTRY=y ./scripts/delete-clusters.sh
```

### Run Script

Run the following command to delete `2` `kind` clusters:

```bash
Expand Down
113 changes: 11 additions & 102 deletions scripts/create-clusters.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,90 +21,12 @@ set -o errexit
set -o nounset
set -o pipefail

source "$(dirname "${BASH_SOURCE}")/util.sh"
CREATE_INSECURE_REGISTRY="${CREATE_INSECURE_REGISTRY:-}"
CONFIGURE_INSECURE_REGISTRY_HOST="${CONFIGURE_INSECURE_REGISTRY_HOST:-}"
CONFIGURE_INSECURE_REGISTRY_CLUSTER="${CONFIGURE_INSECURE_REGISTRY_CLUSTER-y}"
CONTAINER_REGISTRY_HOST="${CONTAINER_REGISTRY_HOST:-172.17.0.1:5000}"
# shellcheck source=util.sh
source "${BASH_SOURCE%/*}/util.sh"
NUM_CLUSTERS="${NUM_CLUSTERS:-2}"
KIND_IMAGE="${KIND_IMAGE:-}"
KIND_TAG="${KIND_TAG:-}"
docker_daemon_config="/etc/docker/daemon.json"
containerd_config="/etc/containerd/config.toml"
OS=`uname`

function create-insecure-registry() {
# Run insecure registry as container
docker run -d -p 5000:5000 --restart=always --name registry registry:2
}

function configure-insecure-registry() {
local err=
if sudo test -f "${docker_daemon_config}"; then
if sudo grep -q "\"insecure-registries\": \[\"${CONTAINER_REGISTRY_HOST}\"\]" ${docker_daemon_config}; then
return 0
elif sudo grep -q "\"insecure-registries\": " ${docker_daemon_config}; then
echo <<EOF "Error: ${docker_daemon_config} exists and \
is already configured with an 'insecure-registries' entry but not set to ${CONTAINER_REGISTRY_HOST}. \
Please make sure it is removed and try again."
EOF
err=true
fi
elif pgrep -a dockerd | grep -q 'insecure-registry'; then
echo <<EOF "Error: CONFIGURE_INSECURE_REGISTRY_HOST=${CONFIGURE_INSECURE_REGISTRY_HOST} \
and about to write ${docker_daemon_config}, but dockerd is already configured with \
an 'insecure-registry' command line option. Please make the necessary changes or disable \
the command line option and try again."
EOF
err=true
fi

if [[ "${err}" ]]; then
if [[ "${CREATE_INSECURE_REGISTRY}" ]]; then
docker kill registry &> /dev/null
docker rm registry &> /dev/null
fi
return 1
fi

configure-insecure-registry-and-reload "sudo bash -c" $(pgrep dockerd) ${docker_daemon_config}
}

function configure-insecure-registry-and-reload() {
local cmd_context="${1}" # context to run command e.g. sudo, docker exec
local docker_pid="${2}"
local config_file="${3}"
${cmd_context} "$(insecure-registry-config-cmd ${config_file})"
${cmd_context} "$(reload-daemon-cmd "${docker_pid}")"
}

function insecure-registry-config-cmd() {
local config_file="${1}"
case $config_file in
$docker_daemon_config)
echo "cat <<EOF > ${docker_daemon_config}
{
\"insecure-registries\": [\"${CONTAINER_REGISTRY_HOST}\"]
}
EOF
"
;;
$containerd_config)
if [ "${CONFIGURE_INSECURE_REGISTRY_CLUSTER}" ]; then
echo -e "containerdConfigPatches:\n- |-\n [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"${CONTAINER_REGISTRY_HOST}\"]\n endpoint = [\"http://${CONTAINER_REGISTRY_HOST}\"]\n"
else
echo ""
fi
;;
*)
echo "Sorry, config insecure registy is not supported for $config_file"
;;
esac
}

function reload-daemon-cmd() {
echo "kill -s SIGHUP ${1}"
}
OS="$(uname)"

function create-clusters() {
local num_clusters=${1}
Expand All @@ -115,27 +37,24 @@ function create-clusters() {
elif [[ "${KIND_TAG}" ]]; then
image_arg="--image=kindest/node:${KIND_TAG}"
fi
for i in $(seq ${num_clusters}); do
kind create cluster --name "cluster${i}" --config - "${image_arg}" << EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
$(insecure-registry-config-cmd ${containerd_config})
EOF
fixup-cluster ${i}
for i in $(seq "${num_clusters}"); do
kind create cluster --name "cluster${i}" "${image_arg}"
fixup-cluster "${i}"
echo

done

echo "Waiting for clusters to be ready"
check-clusters-ready ${num_clusters}
check-clusters-ready "${num_clusters}"
}

function fixup-cluster() {
local i=${1} # cluster num

if [ "$OS" != "Darwin" ];then
# Set container IP address as kube API endpoint in order for clusters to reach kube API servers in other clusters.
local docker_ip=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "cluster${i}-control-plane")
local docker_ip
docker_ip=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "cluster${i}-control-plane")
kubectl config set-cluster "kind-cluster${i}" --server="https://${docker_ip}:6443"
fi

Expand All @@ -144,23 +63,13 @@ function fixup-cluster() {
}

function check-clusters-ready() {
for i in $(seq ${1}); do
for i in $(seq "${1}"); do
util::wait-for-condition 'ok' "kubectl --context cluster${i} get --raw=/healthz &> /dev/null" 120
done
}

if [[ "${CREATE_INSECURE_REGISTRY}" ]]; then
echo "Creating container registry on host"
create-insecure-registry
fi

if [[ "${CONFIGURE_INSECURE_REGISTRY_HOST}" ]]; then
echo "Configuring container registry on host"
configure-insecure-registry
fi

echo "Creating ${NUM_CLUSTERS} clusters"
create-clusters ${NUM_CLUSTERS}
create-clusters "${NUM_CLUSTERS}"
kubectl config use-context cluster1

echo "Complete"
17 changes: 3 additions & 14 deletions scripts/delete-clusters.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,33 +21,22 @@ set -o errexit
set -o nounset
set -o pipefail

DELETE_INSECURE_REGISTRY="${DELETE_INSECURE_REGISTRY:-}"
NUM_CLUSTERS="${NUM_CLUSTERS:-2}"

function delete-insecure-registry() {
docker kill registry &> /dev/null || return 0
docker rm registry &> /dev/null || true
}

function delete-clusters() {
local num_clusters=${1}

for i in $(seq ${num_clusters}); do
for i in $(seq "${num_clusters}"); do
# The context name has been changed when creating clusters by 'create-cluster.sh'.
# This will result in the context can't be removed by kind when deleting a cluster.
# So, we need to change context name back and let kind take care about it.
kubectl config rename-context "cluster${i}" "kind-cluster${i}"

kind delete cluster --name cluster${i}
kind delete cluster --name "cluster${i}"
done
}

if [[ "${DELETE_INSECURE_REGISTRY}" ]]; then
echo "Deleting container registry on host"
delete-insecure-registry
fi

echo "Deleting ${NUM_CLUSTERS} clusters"
delete-clusters ${NUM_CLUSTERS}
delete-clusters "${NUM_CLUSTERS}"

echo "Complete"
Loading

0 comments on commit a6e6d4e

Please sign in to comment.