The One and the Many

DNS Lookup Failure in Go part 2

I previously looked into a problem with DNS lookups in Go: DNS Lookup Failure in Go and The GNU C Library

As a result of a conversation I had the idea to try the native Go DNS library instead of the Go implementation that depends on glibc.

Using the native Go resolver

In order to do this one has to explicitly compile with the native Go DNS library:

go build -tags netgo -installsuffix netgo myprogram.go

(Note this changed to require -installsuffix netgo as of Go 1.4 it seems).

Once you have done this you should see no further mention of libc when you run:

ldd myprogram

Instead you will see something such as

not a dynamic executable

I found that this resolved the issue with the invalid hostname.

1._spf.citigroup.com

can now be resolved just fine. The Go resolver is more lenient it seems about what is a valid host. In fact I did not find any checks for host validity.

A new problem

I encountered a different issue however: The resolver failed on TXT records with multiple character strings. For example, we can receive TXT record responses such as

"v=spf1 " "127.0.0.1"

Which while split apart into two strings is supposed to be decoded to a single string:

"v=spf1 127.0.0.1"

The resolver would only parse out the first string:

"v=spf1 "

I also discovered that this would lead to the resolver thinking there was in fact no answer at all. We would receive a "no such host" error in this case. These are cases that could be resolved just fine with the glibc resolver.

I traced this through as follows:

The problem is that there are multiple offsets and strings to pull out but this code only takes the first. As a result, going back to unpackRR(), we find that the whole answer is not unpacked (offset vs. end of message) and we return only the header portion with the rest of the answer ignored:

if off != end {
  return &h, end, true
}

Then going back to tryOneName(): It calls answer() which finds that only a header was received:

for _, rr := range dns.answer {
  if _, justHeader := rr.(*dnsRR_Header); justHeader {

And we end up here:

if len(addrs) == 0 {
  return "", nil, &DNSError{Err: noSuchHost, Name: name, Server: server}
}

Which tells us the host doesn't exist, though it does, but we did not fully parse it out.

My solution to see that this was indeed the problem was to patch unpackStruct() to loop and pull out strings and append them until the end of the message is hit:

for off < len(msg) {
  // find length from this offset
  n := int(msg[off])

  // move into the message
  off++

  // copy out the message
  b := make([]byte, n)
  for i := 0; i < n; i++ {
    b[i] = msg[off+i]
  }

  // skip past the message
  off += n

  // append the message as a string
  s += string(b)
}

Though this works I suspect it needs more safeties in place.

Resolution

I thought that this would be a nice little fix to pass upstream. I looked through Go's github issues and noticed one that was recently closed:

net: LookupTXT fails when TXT record contains multiple strings #10482

Which seems to be the exact issue here. It's already resulted in a patch in the master branch of the repository and closed. Oh well. A fun investigation!

Looking at the solution though it was a bit different than I had shown above. The change was made to dnsRR_TXT's Walk() function instead where it iterates until the reported answer length is hit. Similar, but without touching the more abstract unpackStruct() function.

Comments