tldr; try this:
sudo iptables -D INPUT -j nixos-fw
Nix firewall chain drops connections from the kubedns Pod's attempt to contact api server.
This results in timeouts:
E0303 20:18:26.598781 1 reflector.go:199] pkg/dns/dns.go:145: Failed to list *api.Endpoints: Get https://10.0.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.0.0.1:443: i/o timeout
dmsg shows packet drops:
[ 7976.189453] rejected connection: IN=cbr0 OUT= PHYSIN=veth2bf58c98 MAC=0a:58:0a:01:00:01:0a:58:0a:01:00:08:08:00 SRC=10.1.0.8 DST=192.168.0.100 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=37407 DF PROTO=TCP SPT=41690 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0
The chains causing this behavior:
-A INPUT -j nixos-fw
-A nixos-fw-refuse -j DROP
Removing the fw rule from INPUT chain fixes
sudo iptables -D INPUT -j nixos-fw