resolved: NXDOMAIN in cache after resolving local domain for the first time

[imwints]

systemd version the issue has been seen with

255

Used distribution

Gentoo

Linux kernel version used

gentoo-6.9.1

CPU architectures issue was seen on

x86_64

Component

systemd-resolved

Expected behaviour you didn't see

Resolved domain is put in cache

Unexpected behaviour you saw

Resolved domain gets NXDOMAIN entry in cache and isn't resolved afterwards

Steps to reproduce the problem

I have a custom DNS in my network at an address (10.0.0.1) with an entry xyz.internal pointing to a server (e.g. 10.0.0.2). All devices use the custom DNS, resolvectl shows DNS=10.0.0.1.

Both servers are up. Now when I ping the server ping xyz.internal it resolves the address once. After that there is a NXDOMAIN entry for xyz.internal and the domain isn't resolved any longer.

xyz.internal IN ANY NXDOMAIN

Additional program output to the terminal or log subsystem illustrating the issue

$ resolvectl
Global
         Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported
  resolv.conf mode: stub
Current DNS Server: 10.0.0.1
       DNS Servers: 10.0.0.1

Link 2 (dummy0)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported

Link 3 (eno1)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported
Current DNS Server: 10.0.0.1
       DNS Servers: 10.0.0.1

$ resolvectl flush-caches && ping -c 1 xyz.internal && ping -c 1 xyz.internal                                                                                                                                                                                                       ✘ 2
PING xyz.internal (10.0.0.2) 56(84) bytes of data.
64 bytes from aix.internal (10.0.0.2): icmp_seq=1 ttl=64 time=0.132 ms

--- xyz.internal ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.132/0.132/0.132/0.000 ms
ping: xyz.internal: Name or service not known

[rpigott]

Hm, can you provide a debug log? systemctl service-log-level systemd-resolved debug and try these two queries again.

[rpigott]

I feel like this could be a dupe of #20548. Hence, you might try with ping -4.

[imwints]

ping -4 doesn't work.

However I think

Your DNS server is simply broken. it returns NXDOMAIN for aaaa queries, while returning a successfully for A queries on the same name. That makes no sense.

is true since there is no AAAA entry in my DNS.

[rpigott]

Er, I forgot that ping always uses AF_UNSPEC, even though I wrote that...

Well, if your dns is really returning NXDOMAIN for some record types and not others, that's quite broken behavior. I don't think there's anything to be done on the resolver side.