DevTips: Attaching VinD to an existing docker network

After a while of hand mangling some initial local development environments to fit my needs, I’ve decided to start looking at some new tooling to help me make this a bit more streamlined. Like most folks who want to spend 0 dollars to work on half baked ideas, I’ve relied heavily on old faitherul, KinD, to get some clusters setup. But once I dig into an idea, that generally ends up with me veering pretty far from the standard config.yaml and needing to dig into further configurations or toss together some other form of automation to get all the things I need deployed.

More recently, I’ve been playing around with Vcluster, more specifically the VinD tooling/config to see if this gives me what I'm looking for. Still early days here, but I imagine I’ll write some words about the experience once I put this through some more thorough testing. But one thing I’ve run into (as with most tools) is the overloaded wording of config.

You see, in my development environment. I kind of want all the things I’m spinning up to be able to talk to each other. From my local machine, to (maybe) multiple clusters, to other standalone docker containers. And I’m not talking just hard coding some docker IPs. I want to be able to have name resolution, certificate handling. All the usual goodies.

So for me, being able to get all these things on a reachable network together is good. So when I first started digging into how/where to configure joining new vind clusters to an existing docker network. I thought this would be fairly straightforward. The problem was that my feeble mind read the documentation and interpreted “add this to your config” and assumed that everything is talking about the same config.

Problem One: Specifying Docker Networks

The first task of course was tracking down how to specify the docker network that I wanted my clusters to get created in. Conveniently, vcluster has some docs which point you in the right direction. Specifically for vind, you’ll be interested in the configurations available under experimental.docker . From there, we can see that we can add the following to attach to our network of choice:

experimental:
  docker:
    network: <your-network-here>

The only call out here is to make sure when you search for vcluster docs in your search engine of choice, that you make sure to pick the docs for the latest stable release of vcluster (0.32 at the time of writing). My initial problem was that google was sending me to some old documentation which didn’t contain the experimental.docker docs and I had to dig a bit further to find what I was looking for.

Actual Problem: What Config Are We Talking About

So finding out what to put in the config file was actually the easy part of this problem. My stumble actually showed up about how to apply that config. You see, when I just did a quick vcluster create --help I quickly saw the following and assumed config meant config:

vcluster create --help
<snip/>
Global Flags:
      --config string       The vcluster CLI config to use (will be created if it does not exist) (default "/Users/tylerauerbeck/.vcluster/config.json")

I saw the --config flag and (clearly) immediately stopped reading. Upon further inspection, it does call out vcluster CLI config. But I clearly interpreted that as “gimme all your configs”. So whenever I dumped our above snippet into a config.yaml and tried to create a cluster with vcluster create <cluster-name> --driver docker --config config.yaml , I see something that resembles:

vc create medium --driver docker --config config.yaml
12:15:45 info Ensuring environment for vCluster medium...
12:15:46 done Created network vcluster.medium
12:15:51 warn Failed to run command docker <snip/>

So we can see that it clearly told me to get lost because it’s creating its own network and ignoring my preferences. But with also the added benefit of breaking everything else! Why is everything broken? Well because I unknowingly gave it a faulty vcluster CLI config, which now leaves it unable to create it’s primary task of creating a cluster, with the added annoyance of making me feel like it’s ignoring me and telling me that it’s going to create its own network anyway.

So with a little poking and prodding, I discovered the -f flag (representing values)

vcluster create --help
<snip/>
  -f, --values stringArray                Path where to load extra helm values from

Here is where I realized that --config is only actually configuring the CLI, which was what was causing the cluster creation to blow up. To actually pass in the configuration for our CLUSTER — I needed to pass in the cluster configuration values. Apparently under the covers there is some helm’ing happening which is one of the things that threw me off. Since I was specifying a docker config (and running this in docker), I had assumed this would have been pre-cluster configuration that needed to happen. But, live and learn.

Solution

So with all of this knew knowledge in place — our command now looks something like this:

vcluster create medium --driver docker -f config.yaml
12:30:56 info Ensuring environment for vCluster medium...
12:31:01 info Starting vCluster standalone medium
12:31:02 done Successfully created virtual cluster medium
12:31:02 info Finding docker container vcluster.cp.medium...
12:31:02 info Waiting for vCluster kubeconfig to be available...
12:31:04 info Waiting for vCluster to become ready...
12:31:13 done vCluster is ready
12:31:13 done Switched active kube context to vcluster-docker_medium
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster

And now we’re ready to get rolling! As you can see, our message about creating its own network is gone. And we can actually see the cluster get created.

If nothing else, this can serve as a reminder to myself next time I run into this problem. But hopefully this helps to smooth out any sharp edges for anybody else running into the same problem.