DNS Lookups Failing After Updating to Latest Node 20 and 18 for IBM i

[DavidRusso]

Version

v20.15.1

Platform

IBM i 7.3:

OS400 idev 3 7 00780002A1F1 powerpc Os

Subsystem

DNS

What steps will reproduce the bug?

Any DNS lookup fails with connection timeout. Can run this code to reproduce:

require("dns").resolve("smtp.office365.com", function() { console.log(arguments); })

Fails with ETIMEOUT, even though the system DNS configuration is good and other software can perform DNS lookups.

How often does it reproduce? Is there a required condition?

Every time. This started happening when updating to Node 20.15.1 and Node 18.20.4 for IBM i. Everything was working fine before the update. Before, I was using Node 20.11.1 and Node 18.18.2.

What is the expected behavior? Why is that the expected behavior?

I expect a good DNS lookup, as the system DNS configuration is good and most other software can perform lookups.

What do you see instead?

ETIMEOUT on any DNS lookup.

Additional information

An interesting thing we noticed that may relate....

On the affected systems, the NSLOOKUP CL command can perform lookups with no problem, but the nslookup command in PASE fails with ETIMEOUT for any lookup.

[RedYetiDev]

(v22)
$ node -e 'require("dns").resolve("smtp.office365.com", function() { console.log(arguments); })'
[Arguments] {
  '0': null,
  '1': [ '52.96.184.50', '52.96.183.226', '52.96.165.210', '52.96.48.50' ]
}

(I also couldn't reproduce on v18, I got similar output to the one above)

I'm unable to reproduce, but I'm not on IBM i.

Have you tried on other computers/networks?

Additionally, is this reproducible on v22?

[richardlau]

CC @nodejs/platform-ibmi

[DavidRusso]

I should have clarified -- this is only happening on IBM i. The problem is not occurring on other platforms.

And only with Node 20.15.1 and Node 18.20.4. The problem started occurring after I updated those on my IBM i from the slightly older versions I mentioned above.

[DavidRusso]

The problem also happens in Node 22.4.1 for IBM i.

[DavidRusso]

The problem does not occur in Node 16 and Node 14 for IBM i, on the same system.

Based on this, and based on things working in the slightly older Node 18/20 versions I mentioned above, we suspect that something in the updated Node 18/20/22 versions for IBM i changes how it does DNS lookups.

I'm not sure if this is relevant or not, but on the affected system:

• The native IBM i CL command NSLOOKUP works: NSLOOKUP HOSTNAME(smtp.office365.com)
• But, the PASE command nslookup smtp.office365.com fails with this output:

;; connection timed out; no servers could be reached

We're not sure when the PASE command stopped working, or if it ever worked on this system. We never tried it before today, while trying to troubleshoot this issue. We're thinking that it probably never worked, as the system DNS configuration is good, works fine with all other software on the system, and has not been changed.

[kadler]

@DavidRusso what does Node think your configured servers are?

node -e 'console.log(require("dns").getServers())'

When I run this it shows 127.0.0.1 on Node 14, 16, 18, and 20. Prior to Node 20, I get ECONNREFUSED when I try to run your example, but on Node 20 it's now ETIMEOUT. I don't have a DNS server running locally, so either of those errors makes sense.

Also, I'm not sure where you got an nslookup PASE command as we don't ship one.

[DavidRusso]

@kadler ,

The PASE nslookup command I mentioned is /usr/bin/nslookup which is a symlink to /QOpenSys/QIBM/ProdData/OS400/DNS/bin/nslookup. I assumed that was provided by IBM based on the location.

• I also get 127.0.0.1 from dns.getServers() on all of those Node.js versions.
• I also get the same behavior you noted where the older versions throw ECONNREFUSED and newer throw ETIMEOUT.
• This actually explains why things appeared to work for me on Node 14 and 16, because I'm using the nodemailer package and it responds differently when dns.resolve() fails with ECONNREFUSED vs. ETIMEOUT. To be clear, dns.resolve() is actually not working for me with Node 14 or 16, either.
• Also, if my system was configured to use localhost for DNS then both errors would also be expected, because no DNS server running on local host.
• But, the thing is... my IBM i system is not configured that way. There are 2 external DNS servers configured under the CHGTCPDMN command. So, the question is... if Node doesn't get the DNS server addresses from there, where does it get them from?

[DavidRusso]

@kadler ,

Now that I understand more, it seems possible/likely that dns.resolve() never worked on IBM i to begin with. As you pointed out, it thinks the DNS server IP is 127.0.0.1, instead of using the DNS configuration from the OS CHGTCPDMN command.

The difference in behavior in the updated Node.js versions is just that now it throws ETIMEOUT instead of ECONNREFUSED when it can't connect to a DNS service on localhost.

This confused us at first, because the software we were using (nodemailer) reacts differently to the error message and for ECONNREFUSED it falls back to dns.lookup() which works. Apparently dns.lookup() calls an OS facility to do the lookup instead of trying to do it's own DNS comms over the network.

So, the root problem or question here is... why does dns.resolve() and dns.getServers() not use the system DNS config from CHGTCPDMN ? Is that a bug?

[kadler]

The PASE nslookup command I mentioned is /usr/bin/nslookup which is a symlink to /QOpenSys/QIBM/ProdData/OS400/DNS/bin/nslookup. I assumed that was provided by IBM based on the location.

Ohhh, I didn't know the DNS server option includes that.

So, the root problem or question here is... why does dns.resolve() and dns.getServers() not use the system DNS config from CHGTCPDMN ? Is that a bug?

My guess is that it's just following the AIX code path and probably trying to read /etc/resolv.conf and falling back? @abmusse can you investigate further there?

Once we know where this is coming from, I have some code somewhere that calls the ILE APIs to get the configured DNS servers and we can try to fix it that way.

[richardlau]

The difference in behavior in the updated Node.js versions is just that now it throws ETIMEOUT instead of ECONNREFUSED when it can't connect to a DNS service on localhost.

Potentially related is a c-ares change from 1.21.0+: #50743
Although for our own tests it was a change from EBADRESP to ETIMEOUT.

[abmusse]

My guess is that it's just following the AIX code path and probably trying to read /etc/resolv.conf and falling back? @abmusse can you investigate further there?

Once we know where this is coming from, I have some code somewhere that calls the ILE APIs to get the configured DNS servers and we can try to fix it that way.

@kadler, Yes I will need to investigate this further. I plan to look into this more next week and will provide an update of what I find.

[abmusse]

Quick update, I haven't had an opportunity to investigate more this week. This is still on my radar to investigate.

[cbeidel]

Hey @abmusse, we have the same issue here at some of our customers on IBM I (7.4). We also updated Node (NodeJs 20.16.0) and since then encountered the problem using nodemailer. We pasted the ip address manually to bypass this but this is just a temporary workaround of course.
We also tried to set the DNS manually like this

import dns from 'dns';
dns.setServers(['8.8.8.8', '8.8.4.4', '1.1.1.1', '1.0.0.1']); 

without any success.
The same Node.js is working without any problems on our Mac OS with the same Node 20.16.0 version.

Do you have any updates yet or are you still investigating?

Many thanks for your support

[richardschoen]

Similar issue when enabling the Ethernet packet filter on IBM I with appropriate rules. Name resolution fails even though ICMP and DNS are enabled.

[abmusse]

@cbeidel @richardschoen

Thanks for the additional reports. I have not forgot about this one. I've been busy putting out fires in other areas and this one is still on my radar.

Like @kadler mentioned in #53948 (comment)

I think we are following the AIX code path and probably trying to read /etc/resolv.conf but we need to add a patch to perform IBM i specific DNS changes to get things working properly.

[richardschoen]

@DavidRusso Did you ever make any progress on this on your end ?

[DavidRusso]

@richardschoen,

We just reverted to using an older Node (14) for our process that was running nodemailer until this is fixed in Node.js. That worked fine for the particular process we were having a problem with.

Until this is fixed in Node.js, the only way I could see to get nodemailer working on current versions of Node.js for IBM i would be to open a PR with the nodemailer authors to make it try dns.lookup() first, instead of dns.resolve().

That, or bypass DNS lookup altogether by using IP address. Obviously that could be problematic if the mail server IP is subject to change.