DNS Lookup Failure in Go part 2
I previously looked into a problem with DNS lookups in Go: DNS Lookup Failure in Go and The GNU C Library
As a result of a conversation I had the idea to try the native Go DNS library instead of the Go implementation that depends on glibc.
Using the native Go resolver
In order to do this one has to explicitly compile with the native Go DNS library:
go build -tags netgo -installsuffix netgo myprogram.go
(Note this changed to require -installsuffix netgo
as of Go 1.4 it seems).
Once you have done this you should see no further mention of libc when you run:
ldd myprogram
Instead you will see something such as
not a dynamic executable
I found that this resolved the issue with the invalid hostname.
1._spf.citigroup.com
can now be resolved just fine. The Go resolver is more lenient it seems about what is a valid host. In fact I did not find any checks for host validity.
A new problem
I encountered a different issue however: The resolver failed on TXT records with multiple character strings. For example, we can receive TXT record responses such as
"v=spf1 " "127.0.0.1"
Which while split apart into two strings is supposed to be decoded to a single string:
"v=spf1 127.0.0.1"
The resolver would only parse out the first string:
"v=spf1 "
I also discovered that this would lead to the resolver thinking there was in fact no answer at all. We would receive a "no such host" error in this case. These are cases that could be resolved just fine with the glibc resolver.
I traced this through as follows:
net/lookup.go
: This is whereLookupTXT()
is definednet/lookup_unix.go
:lookupTXT()
callslookup()
net/dnsclient_unix.go
:lookup()
callstryOneName()
net/dnsclient_unix.go
:tryOneName()
callsexchange()
net/dnsclient_unix.go
:exchange()
performs the DNS call over the network andtryOneName()
then callsreadDNSResponse()
which parses/unpacks the responsenet/dnsmsg.go
:Unpack()
callsunpackRR()
on each answernet/dnsmsg.go
:unpackRR()
callsunpackStruct()
which calls aWalk()
function for each record type. For TXT we end up in a block of code like this (comments added by myself):if off >= len(msg) || off+1+int(msg[off]) > len(msg) { return false } // find length from this offset n := int(msg[off]) // move into the message off++ // copy out the message b := make([]byte, n) for i := 0; i < n; i++ { b[i] = msg[off+i] } // skip past the message off += n // append the message as a string s = string(b)
The problem is that there are multiple offsets and strings to pull out
but this code only takes the first.
As a result, going back to unpackRR()
, we find that the whole
answer is not unpacked (offset vs. end of message) and we return only
the header portion with the rest of the answer ignored:
if off != end {
return &h, end, true
}
Then going back to tryOneName()
: It calls answer()
which finds that
only a header was received:
for _, rr := range dns.answer {
if _, justHeader := rr.(*dnsRR_Header); justHeader {
And we end up here:
if len(addrs) == 0 {
return "", nil, &DNSError{Err: noSuchHost, Name: name, Server: server}
}
Which tells us the host doesn't exist, though it does, but we did not fully parse it out.
My solution to see that this was indeed the problem was to
patch unpackStruct()
to loop and pull out strings and append them
until the end of the message is hit:
for off < len(msg) {
// find length from this offset
n := int(msg[off])
// move into the message
off++
// copy out the message
b := make([]byte, n)
for i := 0; i < n; i++ {
b[i] = msg[off+i]
}
// skip past the message
off += n
// append the message as a string
s += string(b)
}
Though this works I suspect it needs more safeties in place.
Resolution
I thought that this would be a nice little fix to pass upstream. I looked through Go's github issues and noticed one that was recently closed:
net: LookupTXT fails when TXT record contains multiple strings #10482
Which seems to be the exact issue here. It's already resulted in a patch in the master branch of the repository and closed. Oh well. A fun investigation!
Looking at the solution though it was a bit different than I had
shown above. The change was made to dnsRR_TXT
's Walk()
function instead
where it iterates until the reported answer length is hit. Similar,
but without touching the more abstract unpackStruct()
function.