Sharing a local registry with minikube

How do I get a locally built docker image into minikube with the least amount of pain, and the greatest amount of speed?

If you’re developing with minikube you need to use containers. Containers need images, and during development you may want to frequently modify images. To complete your development workflow you need to get the images you build into minikube so that kubernetes can run containers/pods built off the image.

For the rest of this post, we’ll use the more complicated case of having docker-machine to get our system running. I mean we entered the rabbit hole when we decided to roll with docker anyway, so why start making things easy now.

But:

• This will result in way too many image tags on your docker hub repo
• Also, uploading and downloading hundreds of floppy disks worth of data through ze internetz to just get software from one part of your computer to another doesn’t sound right.

Both docker push and kubectl run will fail because the registry is insecure.

First, let’s look at: docker push 192.168.99.100:5000/my-image

• Will fail because 192.168.99.100 is not a secure registry
• Use SSL certs and restart the docker daemon to get this to work, or set this as an insecure registry and restart the docker daemon to get this to work.
• The push command will change every time docker-machine’s ip changes (potentially whenever docker-machine starts)

Second, let’s look at: kubectl run --image=192.168.99.100:5000/my-image

• Will fail to pull because 192.168.99.100 is not a secure repository
• Use SSL certs, or set this as an insecure registry and restart minikube (this is a pain to get working on minikube)
• This command will also change every time the docker-machine ip changes

docker doesn’t complain about pushing to or pulling fromlocalhost:5000 even if that’s an insecure registry. Ooooh.

So: docker push should push to a registry named localhost:5000 and inside minikube, docker pull should pull from a registry named localhost:5000.

Let the port-mapping shenanigans begin!

The idea here is for the docker daemon on minikube to be able to pull from a registry called localhost:5000. This is achieved by actually running a registry on minikube and then setting up a proxy so that the minikube VM port 5000 maps to the registry’s 5000.

Now, we need to make sure that the docker daemon on docker-machine thinks that localhost:5000 is legit. So, we map 5000 of the docker-machine ⇢ 5000 of the host (via a reverse SSH tunnel) and 5000 of the host ⇢ 5000 of the registry running on minikube (via kubectl port-forward). Ofcourse, if you’re one of the enlightened, you’re already running a linux host and of your 99 problems this ain’t one.

So here are all the steps finally:

1. Create a registry on minikube

Create a registry (a replication-controller and a service) and create a proxy to make sure the minikube VM’s 5000 is proxied to the registry service’s 5000.

kubectl create -f kube-registry.yaml

(Grab kube-registry.yaml from this gist on github.)

At this point, minikube ssh && curl localhost:5000 should work and give you a response from the docker registry.

2. Map the host port 5000 to minikube registry pod

kubectl port-forward --namespace kube-system \ 
$(kubectl get po -n kube-system | grep kube-registry-v0 | \awk '{print $1;}') 5000:5000

After this, from the host curl localhost:5000 should return a valid response from the docker registry running on minikube

3. Non-linux peoples: Map docker-machine’s 5000 to the host’s 5000.

ssh -i ~/.docker/machine/machines/default/id_rsa \
-R 5000:localhost:5000 \docker@$(docker-machine ip)

After this, running docker-machine ssh && curl localhost:5000 should return a valid response from the docker registry running on minikube.

All of this put toghether, and finally:

It’s not elegant, but it works without fiddling around and restarting docker client, docker daemon, minikube flags, minikube’s config.json. All of which can change any time today, tomorrow or have probably already changed yesterday and you’re on an older version. Hah.

Also, just FYI, we’ve spent a spectacular amount of effort in moving code a nanometer distance or something. 2017 y’all.