{"id":622,"date":"2018-12-08T07:44:53","date_gmt":"2018-12-08T15:44:53","guid":{"rendered":"https:\/\/porkrind.org\/missives\/?p=622"},"modified":"2019-02-17T21:07:32","modified_gmt":"2019-02-18T05:07:32","slug":"att-causes-mdns-on-linux-to-fail","status":"publish","type":"post","link":"https:\/\/porkrind.org\/missives\/att-causes-mdns-on-linux-to-fail\/","title":{"rendered":"AT&T causes mDNS on Linux To Fail"},"content":{"rendered":"

Today my Jenkins builds were not working because all of the build slaves were offline. Digging around in the logs showed that the couldn’t connect because of name resolution failures. I use mDNS on my network (the slaves are Mac OS X VMs running on a Mac Mini), and so they were named something like xxxx-1.local<\/code> and xxxx-2.local<\/code> I tried just pinging the machines and that refused to resolve the name, too.<\/p>\n

I verified that Avahi was running, and then used avahi-resolve --name xxxx-1.local<\/code> to check the mDNS name resolution. It worked just great.<\/p>\n

So why would mDNS be working fine network-wise, but no programs were resolving correctly? It struck me that I didn’t know (or couldn’t remember!) how mDNS tied in to the system. Who hooks in to the name resolution and knows to check for *.local<\/code> using mdns?<\/p>\n

It turns out it’s good old \/etc\/nsswitch.conf<\/code> (I should have remembered that)! There’s a line in there:<\/p>\n

hosts:          files mdns4_minimal [NOTFOUND=return] dns\n<\/code><\/pre>\n
That tells the libc resolver (that everything uses) that when it’s looking for a hostname, it should first look in the \/etc\/hosts<\/code> file, then it should check mDNS, then if mDNS didn’t handle it, check regular DNS. Wait, so mDNS is built right in to libc??<\/p>\n
Nope! On my Debian system there’s a package called libnss-mdns<\/code> that has a few files in it:<\/p>\n
\/lib\/x86_64-linux-gnu\/libnss_mdns.so.2\n\/lib\/x86_64-linux-gnu\/libnss_mdns4.so.2\n\/lib\/x86_64-linux-gnu\/libnss_mdns4_minimal.so.2\n\/lib\/x86_64-linux-gnu\/libnss_mdns6.so.2\n\/lib\/x86_64-linux-gnu\/libnss_mdns6_minimal.so.2\n\/lib\/x86_64-linux-gnu\/libnss_mdns_minimal.so.2\n<\/code><\/pre>\n
Those are plugins to the libc name resolver so that random stuff like mDNS doesn’t have to be compiled into libc all the time. In fact, there’s a whole bunch of other libnss plugins in Debian that I don’t even have installed.<\/p>\n
So my guess was that this nss-mdns plugin was causing the problem. There are no man pages in the package, but there are a couple README files. I poked around trying random things and reading and re-reading the READMEs many times before this snippet finally caught my eye:<\/p>\n
\n  If, during a request, the system-configured unicast DNS (specified in \/etc\/resolv.conf<\/code>) reports an SOA<\/code> record for the top-level local<\/code> name, the request is rejected. Example: host -t SOA local<\/code> returns something other than Host local not found: 3(NXDOMAIN)<\/code>.\u00a0 This is the unicast SOA heuristic.<\/em>\n<\/p><\/blockquote>\n
Ok. I doubted that was happening but I decided to try their test anyway:<\/p>\n
$ host -t SOA local\nlocal. has SOA record ns1-etm.att.net. nomail.etm.att.net. 1 604800 3600 2419200 900\n<\/code><\/pre>\n
Crap.<\/p>\n
Those bastards at AT&T set their DNS server up to hijack unknown domains! They happily give out an SOA<\/code> for the non-existant .local<\/code> TLD. So AT&T’s crappy DNS is killing my Jenkins jobs??? Grrrrr\u2026<\/p>\n
The worst part is that I tried<\/em> to use Cloudflare’s 1.0.0.1<\/code> DNS. My router was configured for it. But two things happened: 1. I got a new modem after having connection issues recently, 2. I enabled IPv6 on my router for fun. The new modem seems to have killed 1.0.0.1<\/code>. I can no longer connect to it at all. Enabling IPv6 gave me an AT&T DNS server through DHCP (or whatever the IPv6 equivalent is).<\/p>\n
So, straightening out my DNS (I had to revert back to Google’s 8.8.8.8<\/code>) caused NXDOMAIN<\/code> responses to the .local<\/code> SOA<\/code>, and that caused mDNS resolution to immediately work, and my Jenkins slaves came back online. Fwew.<\/p>\n","protected":false},"excerpt":{"rendered":"
Today my Jenkins builds were not working because all of the build slaves were offline. Digging around in the logs showed that the couldn’t connect because of name resolution failures. I use mDNS on my network (the slaves are Mac OS X VMs running on a Mac Mini), and so they were named something like … Continue reading AT&T causes mDNS on Linux To Fail<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-622","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/posts\/622","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/comments?post=622"}],"version-history":[{"count":10,"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/posts\/622\/revisions"}],"predecessor-version":[{"id":644,"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/posts\/622\/revisions\/644"}],"wp:attachment":[{"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/media?parent=622"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/categories?post=622"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/tags?post=622"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}