Switching from ipv4ll to dhcp address loses nameserver (might be systemd bug)

[kaos-ocs]

ifup an Ethernet interface (eno1) defined as dhcp. The dhcp server is slow so there is no response to the initial discovery. After a few seconds, dhcpcd drops down to ipv4ll and assigns a 169.254/16 address. dhcpcd drops into background and retries discovery which now works. That generates a call to add the new IP and nameservers (dhcpcd-run-hooks BOUND with if_down=false) followed by a call to delete the ipv4ll address (dhcpcd-run-hooks IPV4LL with if_down=true). The first call sets the nameservers via resolvconf -a eno1.dhcp, the second call deletes the ipv4ll information with resolvconf -d eno1.ipv4ll -f. Unfortunately systemd-resolvconf treats the second call as RevertLink which wipes ALL information for eno1, resulting in no nameservers.

Not sure if this is a dhcpcd bug (should it delete ipv4ll before adding dhcp data?) or a systemd-resolvconf bug (confusing ipv4ll with link).

Workaround: add noipv4ll to /etc/dhcpcd.conf.

[rsmarples]

As you guessed a systemd bug: systemd/systemd#25032

They ship a "resvolvconf compatbile" shim which turns out isn't compatible with all other implementations and refuse to fix it.
Which is ironic as the only fix is to merge protocols into one interface file for resolvconf - and the job of resolvconf is to merge configuration data into /etc/resolv.conf. It's like an endless loop and I won't be fixing it here.

[DanielG]

refuse to fix it

Isn't systemd/systemd#25052 a fix?

[rsmarples]

@DanielG oooh, I didn't know they tried to fix it. @kaos-ocs does your systemd have that fix from 2022 in?

[DanielG]

I have to echo https://github.com/systemd/systemd/pull/25052/files#r999176445 though. The resolvconf interface design seems unfortunate.

[rsmarples]

I have to echo https://github.com/systemd/systemd/pull/25052/files#r999176445 though. The resolvconf interface design seems unfortunate.

It's unfortunate that he wants to map to a real interface as well. The original Debian init scripts had this in dnsmasq
echo 'nameserver 127.0.0.1 | resolvconf -a localhost

At the end of the day, it's just a label - and to ensure a unique key interface and protocol is the ideal combination.
Also, this is only a problem if 1) the system allows you to rename interfaces and 2) you're stupid enough to call an interface foobar.dhcp and another interface foobar and 3) you're running a system that needs to derive the real interface.

Now, the real question is why is 3) so damned important? Nothing in DNS is interface specific.

Once you realise this, you know why simply extending the interface to interface[.protocol] was such a no-brainer at the time because other people came up with this approach and I copied the idea.

[DanielG]

Nothing in DNS is interface specific.

I mean that ignores the fact that many sites have to use split horizon DNS, doesn't it? I think that's an (unfortunately necessary use-case) that has to be supported.

[rsmarples]

People put too much into an interface. Heh.
For example, how relevant is it in a bridge?

Anyway, split horizon is achieved using the source address of where the request came from, not which interface it came from which the DNS server cannot know.

Likewise doing the reverse, we configure resolvers to send to specific nameservers for a domain via addressing and not via any interface. I'm not even sure that's technically possible either, but it's been a few years since I dug into this.

[DanielG]

Hi Roy,

People put too much into an interface. Heh. For example, how relevant is it in a bridge?

I dont get what you're saying. So what if it's a bridge it's still an interface *shrug*. My workstation has a bridge br0 as it's main interface, so? I still want per-interface DNS request routing to work and I don't see how systemd-resolved could do that reliably without assigning meaning to the interfaces as far as resolvectl is concerned?

Anyway, split horizon is achieved using the source address of where the request came from, not which interface it came from which the DNS server cannot know.

Look at it from the client side. One may want to ensure private DNS traffic doesn't end up at one's employers DNS server because of an always-on VPN just used to reach internal resources (say intranet.corp.domain) or conversely corp internal queries unencrypted to the ISP's DNS recursor.

So it's not (mainly) a matter of the server figuring out where the request came from, but my machine routing the request to the right server in the first place (based on the advertised search domain(s)).

There's this old systemd issue by Jason (wg author), I found it helpful to understand the use case. I think systemd/systemd#17529 is the one I'm thinking of.

Likewise doing the reverse, we configure resolvers to send to specific nameservers for a domain via addressing and not via any interface. I'm not even sure that's technically possible either, but it's been a few years since I dug into this.

I don't follow. How does addressing help here?

[kaos-ocs]

@DanielG oooh, I didn't know they tried to fix it. @kaos-ocs does your systemd have that fix from 2022 in?

Ubuntu 24.04 LTS. NetworkManager and netplan have been removed, using ifup instead.
systemd 255.4-1ubuntu8.1 which contains commit 7d4e850323b4270979ac55dd0e6893a7c348c7e2 from 2022.

Since this box has a fixed location and the DNS servers will never change (Yeah, right!), I have just added my DNS servers to /etc/systemd/resolved.conf.d and turned off the nameserver option in dhcpcd.conf. That avoids the ipv4ll vs. dhcp confusion, although it does generate an extra message from resolvconf "No DNS servers specified, refusing operation"

[rsmarples]

So it's not (mainly) a matter of the server figuring out where the request came from, but my machine routing the request to the right server in the first place (based on the advertised search domain(s)).

Routing is done from source address to target address. The kernel picks the interface to send the data out on based on this information. For example given this config:

# Pretty standard
eth0: 192.168.1.10/24
DHCP search domain foobar.com
DNS server 192.168.1.1

# Not so standard
eth1: 10.1.1.10/24
DHCP search domain deadbeef.org
DNS server 192.168.1.1

Which interface do you think a lookup for thishost.deadbeef.org will go through?

Look at it from the client side. One may want to ensure private DNS traffic doesn't end up at one's employers DNS server because of an always-on VPN just used to reach internal resources (say intranet.corp.domain) or conversely corp internal queries unencrypted to the ISP's DNS recursor.

So it's not (mainly) a matter of the server figuring out where the request came from, but my machine routing the request to the right server in the first place (based on the advertised search domain(s)).

This is achievable with openresolv + a decent resolver such as unbound.
printf 'nameserver 10.1.1.1\ndomain corp.intranet\n' | resolvconf -p -a some_label

The -p marking will instruct openresolv to configure unbound to forward requests for *.corp.intranet to 10.1.1.1
10.1.1.1 will not be configured as a global forwarder. No leakage occurs.

Please note that the systemd resolvconf compat does not support -p, which is unfortunate.

There's this old systemd issue by Jason (wg author), I found it helpful to understand the use case. I think systemd/systemd#17529 is the one I'm thinking of.

I found it an interesting read. My opinion is this: The systemd-resolved interface design seems unfortunate.

[rsmarples]

@DanielG oooh, I didn't know they tried to fix it. @kaos-ocs does your systemd have that fix from 2022 in?

Ubuntu 24.04 LTS. NetworkManager and netplan have been removed, using ifup instead. systemd 255.4-1ubuntu8.1 which contains commit 7d4e850323b4270979ac55dd0e6893a7c348c7e2 from 2022.

Maybe this is a new bug about resolvconf -d? The commit referenced only has test cases for the -a command.

Since this box has a fixed location and the DNS servers will never change (Yeah, right!), I have just added my DNS servers to /etc/systemd/resolved.conf.d and turned off the nameserver option in dhcpcd.conf. That avoids the ipv4ll vs. dhcp confusion, although it does generate an extra message from resolvconf "No DNS servers specified, refusing operation"

Turn of the search and domain options as well.
But I do find is amusing that you must have dns servers for search and/or domains as I know it "works as designed" without.

EDIT: Works as designed in you're allowed a domain and/or search domains without name servers in /etc/resolv.conf