What is the rabbit hole?

If you’re unfamiliar with the term, a rabbit hole is a slightly pejorative term used to describe going deep in something that’s adjacent to the problem you’re “supposed to” be solving.

Why do I feel qualified to opine on rabbit holes?

Well, yesterday I built output88.com.

Actually, yesterday morning, I made (what I thought was) a small change to my out-of-band network overlay, on amethyst, my personal kubernetes cluster. It was one of those “let’s start the day with a nice spring cleanup change,” after all it’s May already and the weather is quite nice.

So, I made the change on my switch and got to work on the “real work.”

And then

• Later that day, as I was deploying output88, I noticed intermittent connection failures.
  • I assumed it was something about DMARC and SPF records and external-dns, probably related to an unrelated (ha) change I made a few weeks ago because I couldn’t get external-dns to handle certain types of fastmail-related DNS records properly.
  • I also added two more physical nodes to my cluster last week (jade and onyx), and I thought I’d done everything correctly, but suddenly I remembered, oh, there might have been some sys.ipv4.forwarding OS setting I forgot to set.
  • Ask an LLM, “it’s because the rule that sets your iface names and mac addresses is not configured”

Confounding Variables

OK, so at this point, we have a couple of confounding variables, and three new things.

Also, up until yesterday I didn’t have a monitoring alert running for this site, so I wasn’t sure if it was a new issue or not. But nonetheless, there’s an issue. So three confounding variables and also we’re not sure if this is a new issue or not.

ARPAnet

I also do remember something about ARPing, because I choose to run MetalLB on amethyst because it’s awesome.

If I’m debugging something and I get to ARPing, either I’m .. not going to try and finish that sentence.

Oh also I started a long-running process to rebalance k8s node usage last week, so that’s 5 confounding variables and I have no idea what ARPAnet is.

But I’m a “startup founder.” In fact, I’m an “engineer-turned-CEO-so-i-should-be-careful-about-fun-things”

What here is “business critical”? Zero. How many users do I have? Well I just pivoted my company yesterday so I literally have no business, and absolutely no business to be running a self-hosted k8s cluster.

Why am I doing this?

OK, we’ve hit the ground. Feet feel dirt. It’s dusty. But we’re at the bottom of the rabbit hole.

The ground feels nice.

I’m here because I want to be.

There is no tether, there is no debt to be paid. I don’t need to prove myself to others. Everyone who I think would care is probably in their own rabbit hole.

I’m free to go elsewhere.

This is, in my opinion, the most important part of every rabbit hole I’ve been in. This is step 0 of finding your own bootstraps so you can pull yourself out of the hole. Decide:

• Yes, I will have the resolve and allocate this much time and budget to solving this problem, or
• No, I will not solve this problem right now.

The worst is when you’re in this ambiguous middle ground and it’s 3:00pm and you had a different goal yet you know that this is acute yet you’re not sure if you should be solving it.

Wondering whether you should be in a rabbit hole, while you may or may not be in the rabbit hole, is the fastest path to wasting your day/week/month/quarter/life.

Step 1: Decide. Will I pursue this rabbit hole.

Not “can I solve it” but “do I want to solve it?”

Step 2: Prepare

If you’re going to go down the rabbit hole, go boldly. There is no honor in fighting the war unless you equip yourself with the right tools; moreover, fighting a battle that you know will be challenging, without the right tools, isn’t honorable. Just like the YC demo day pitch “we’ve gotten 100 users and we haven’t even turned on paid on marketing yet” – that doesn’t mean you’re smart or that your product is good - it means you don’t know how to allocate your resources.

Even if you win the battle, there’s no glory in doing while being underprepared. That just means you have low executive functioning and aren’t properly allocating capital.

Clarity of mind will show up when you have resolve in your heart.

Either do it or don’t. I got places to be.

“I will not solve this problem” doesn’t say anything other than whether or not you choose to solve this problem.

Once you de-couple rabbit holes from your own self worth, a world of possibilities opens up.

Tempted to close this with a witty or ostensibly “the journey is the destination” but instead I’ll just say if you want to host a website, you could run your own kubernetes cluster on your own hardware, but it sure is a lot easier to just use render.com.