Lookup function not working?

[kingjerod]

Describe the bug
I've added a custom lookup function with a cache for the DNS but it doesn't appear to be working. This is my code:

interface IDNSEntry {
  address: string;
  family: number;
  expires: number;
}
const dnsCache: Map<string, IDNSEntry> = new Map();

function lookup(hostname, options, callback) {
  if (dnsCache.has(hostname)) {
    const entry: IDNSEntry = dnsCache.get(hostname);
    if (Date.now() < entry.expires) {
      return callback(null, entry.address, entry.family);
    }
  }
  dns.lookup(hostname, (error, address: string, family: number) => {
    if (error) {
      callback("Failed to lookup " + hostname);
    } else {
      dnsCache.set(hostname, {address, family, expires: (Date.now() + (60 * 1000))});
      return callback(null, address, family);
    }
  });
}
tracer.init({profiling: true, analytics: true, hostname: "dd-agent", lookup: lookup});

Maybe I'm not understanding what the lookup function does. When I go into my Datadog console, I see 100 req/s for "dd-agent", and 3 req/s for other services. I would expect these numbers to be < 1 req/s since they should be cached for 1 minute.

• Environment
  AWS C5.Large Docker Node image

• Operation system:
  Docker

• Node version:
  10.14.1

• Tracer version:
  0.23.2

• Agent version:
  Latest

[rochdev]

Does your code run at all or is it never called? Is this something you could reproduce with a small snippet?

[kingjerod]

It is called, because if there is an error in there it causes issues. I'm not sure what other code I can give you. My goal is to reduce the amount of DNS look ups done, most other services are less than 1 per second, but since adding APM I can see it's 10+ req/s for looking up the dd-agent hostname. Must be every time it wants to send APM stat? Since dd-agent is just another Docker container on the host it doesn't need to be looking it up 100 times per second.

[kingjerod]

Maybe this image will help to show what I'm seeing. I've erased the other host names below dd-agent. This is the dns service. I'm trying to reduce that 33 million DNS look ups for dd-agent.

[rochdev]

Since it's the dns module that calls the lookup function and not the tracer, if the above code is called it means it was properly passed. Is it possible that the code does not result in caching for some reason? If you use the above code directly with the http or dns module do you get the same behavior?

How many instances of the service are there? The default setting for calls to the agent is once every 2 seconds, so to get 100 reqs/s that would mean 200 instances, unless they get very high traffic which causes early flushes.

In any case, one thing that I don't understand is that we now use HTTP keep alive by default, which would mean that once the connection is established, there should be no further DNS lookups even without a cache.

[kingjerod]

There should be only one instance of the dd-tracer per running server process. The posted code is in a file by itself and imported before anything else in the index.ts file. There are multiple servers running, but at the most it should only be 8 instances total.

I have not tried directly with the http or dns module.

I put the above code into JSFiddle and the caching logic appears to be working properly: https://jsfiddle.net/qrtuv8x7/

In the console:

[rochdev]

Both keep-alive and the lookup option work for me locally. I'm afraid I will really need a reproduction snippet. Any way you can reproduce the issue in isolation?

[kingjerod]

Is there a way to reproduce it locally without sending it to Datadog? Rather not have to incur the APM costs to figure this out.

[rochdev]

One thing I've just realized is that the DNS lookups are to the agent but these should not show in the UI at all since we don't trace the tracer. Is it possible that something else unrelated to the tracer sends requests to the agent? For example a DogStatsD client? Also, internally the lookup function is cached but that would only work if the tracer is initialized before any other imports in the app. Can you validate that the tracer initialization is the first thing that happen in the entry point of the app?

Reproducing without sending to Datadog is definitely possible but it would be a bit involved since you'd either have to parse the log output or implement a mock agent to intercept and introspect the requests.

[kingjerod]

Hmm it could definitely be the DogStatsD client! I'll have to set that to use DNS cache. Next time I have to enable APM I will report back to see if that fixes things.

[rochdev]

@kingjerod Were you able to validate if the issue was caused by the DogStatsD client?

[rochdev]

Closing for now as from the above discussion it sounds like it's caused by the DogStatsD client and we were not able to reproduce with the tracer. Please feel free to re-open if that was not actually the cause.

[dackland]

@kingjerod Were you able to fix this issue? We're seeing the exact same problem, and using the DogStatsD client's cacheDns option has no effect.

[Raynos]

Seeing the same issue still happening.