Sometimes it takes banging your head against a wall a couple times before you finally figure out a problem. This can apply to all sorts of things, and often we can become guilty of resorting to muscle memory, where our practice amalgamates into habit instead. Before we know it, we are coasting trying the same solution to the same problem, and as Einstein quotes, “Insanity is doing the same thing over and over and expecting different results.”
That is a similar song that was sung yesterday as I was the insane system administrator trying to figure out a peculiar issue with connectivity in my network.
The Problem
The problem was simple:
from my main computer, I could ping any other device. However, from any other computer, I couldn’t ping my main computer. I was able to use a VPN to talk to my main computer though, which was what was strange and prompted me to investigate.
Looking into Firewall rules
I looked into my firewall rules in my PfSense firewall. I figured that if I couldn’t ping, I should role that out by double-checking I wasn’t blocking connectivity between each of my networks that I had segmented. In fact, I ended up disabling every firewall rule temporarily to absolutely make sure there was no interference. Still, the pings returned void.
Making sure it’s not DNS…
After I ruled out the firewall rules, quite literally, I decided to see if there was some issue with my DNS resolver in PfSense. As my PfSense router is a wonder-router/firewall thing, it has an internal DNS server and I use PFBlocker to manage DNS for blocking certain ads and malicious websites. By this time, I was sifting carefully through each setting and ensuring that nothing was connected to DNS. Still, I was unable to ping.
My original thought for ensuring there was nothing wrong with DNS, despite the fact I was pinging the direct IP, is that I use a NAT rule to redirect DNS traffic through the firewall, so I just wanted to be absolutely sure, since I had been able to connect with the Tailscale “magic” DNS. In retrospect, the problem had little to do with DNS, which we’ll get to the solution soon, but I mention this to show you how I was sort of blindsided by the peculiar nature of this issue.
Checking into other potential misconfigurations
After I ruled out DNS, I was starting to get a little desperate, so I looked into my Proxmox Server, which is the main system that virtualizes everything on my network. My thought here was that maybe a certificate was invalid, a VLAN was misconfigured, or something!! (I was really just trying anything and everything to find things out, even if now I realize they really didn’t make much sense.) While this was a little painful of a route to go down and still not get any answer, I knew I was starting to get somewhere.
Looking for a pattern
As I assured myself that these things weren’t contributing or related to the issue, I conducted an NMAP ping scan across the network on each sub network that I use to see what I COULD ping. I noticed there were a few IP addresses missing, and that’s when I noticed that there wasn’t really a pattern to what I could connect to, or what I couldn’t.. or was there?
Back to the VPN
That’s when I started to get skeptical of Tailscale, my VPN client, and I began to check into my settings. I had read on various threads and forums, and I had found some hits for similar issues. I think this is the moment I started to think I was onto something, and some users even mentioned that they were able to ping devices within Tailscale, but not normal network pings. That’s when I tried it myself, and sure enough, I could ping the device within Tailscale, but not in my own network.
Problem Solved
I opened up my Windows Firewall for each Windows device that had the client, and I went into my Inbound firewall settings and added an exception for the appropriate local subnets, and resent the pings… and well, it worked. It worked seamlessly. I spent so long trying to figure out what was going wrong, and it was such a simple issue. However, I am native to spending longwinded amounts of time trying to solve simple issues, but, this, was something else.
Lessons Learned
The moral of this story is, I got it to work. Sure, I went through some roundabout ways of doing things, got a little disappointed, tried some things that made no sense, but I figured it out. The only reason I believe I got to the solution though is because I have a passion and drive to understand what’s happening, and I was so perplexed by the nature of such an issue, which motivated me because of how puzzled I was, and there was a hunger to learn more.
I’ll leave some documentation below on some various threads I visited and resources that might pertain to the world of troubleshooting. I try to leave a lesson at the end of each of my posts, and I’d say that this one was all about learning to be okay with dead-ends and understanding that having the right mindset transcends any misconfiguration or failure. You will learn from the mistake as long as you’re able to catch it and have the mentality that you can learn from anything, regardless of how much or how little you know.
Resources
Local network access not working after connecting to Tailscale
byu/dwellexity inTailscale