Hi,
I noticed that Knot Resolver 6.5.0 drops UDP queries when the source port
is lower than 1024.
This is caused by min_udp_source_port = 1024 added in version 6.0.7 (!1507).
There is no config option to change this, and there are no logs unless
debug mode is enabled.
Environment
- Knot Resolver 6.5.0-cznic.1~bookworm
- Debian 12
What happens
Some of our clients send DNS queries from low source ports:
- Port 53: home routers asking for AFTR and NTP server names
- Port 753: online fiscal cash registers
- Ports 1, 565, and others
In just 60 seconds, we saw about 600 client IPs that did not get an answer.
How to reproduce
dig -b 127.0.0.2#53 @127.0.0.1 example.com
# Timeout
dig -b 127.0.0.2#1023 @127.0.0.1 example.com
# Timeout
dig -b 127.0.0.2#1024 @127.0.0.1 example.com
# Answer
RFC
RFC 5452, section 9.2, allows source ports "53, or 1024 and above".
RFC 1035 only defines the server port, so at least source port 53 should be
accepted.
Workaround
I found that this works in config.yaml:
lua:
script: |
require('ffi').C.the_network.min_udp_source_port = 0
But it uses an internal structure, so it may not be safe for future
versions.
Question
Could Knot Resolver accept queries from source port 53 by default, as RFC
5452 allows ?
Maybe other low ports could also be accepted, while blocking only ports
that may cause UDP loops, like echo (7) and chargen (19).
It would also be useful to have an official config option to change the
minimum UDP source port.
Best regards,
Mateusz Masłowski
Hi,
I found a performance issue in Knot Resolver 6.5.0 when reloading an RPZ
file.
Every time the RPZ file changes, the watchdog starts the policy-loader
after about 5 seconds. During this reload, some workers stop answering
queries for up to 0.9 seconds.
The problem seems to be related to cache.open(), which calls
posix_fallocate() on the whole cache file. With our 32 GB cache on tmpfs,
this takes about 0.85 seconds, even on an existing cache file.
We use Knot Resolver 6 at a local ISP with around 30k users. Our RPZ
blocklist is updated about once per hour, usually with only 100-200 names
changed.
* Environment *
- Knot Resolver 6.5.0-cznic.1~bookworm
- libknot16 3.5.8 (lab) / 3.5.9 (production)
- liblmdb0 0.9.24-1
- python3-watchdog 2.2.1-1
- Debian 12, kernel 6.1
- 30 workers, 64 CPU cores
- Cache on tmpfs, size-max 32G
- RPZ file: ~825k records, ~326k names, watchdog enabled
* What happens *
The generated policy-loader script includes:
cache.open(34359738368, 'lmdb:///var/cache/knot-resolver')
I checked the policy-loader with perf record -g. For about 0.85 seconds,
almost all CPU time is spent in:
shmem_fallocate vfs_fallocate __x64_sys_fallocate posix_fallocate64
(libc.so.6) [unknown] (libkres.so.9) kr_cache_open+0x8f (libkres.so.9)
I also sampled /proc/<pid>/{stat,wchan,syscall} for all workers every 10 ms.
During this operation, workers trying to write to the cache appear to be
blocked in futex(). Their DNS responses are delayed until posix_fallocate()
finishes.
On our lab host, 2 out of 30 workers were affected. On production, with
higher traffic, almost all workers were affected.
We measured the following maximum response times during normal RPZ updates:
Host Before update After update
Production 1 39 ms 874 ms
Production 2 53 ms 887 ms
There were no timeouts or SERVFAIL responses. All delayed answers were sent
almost at the same moment.
I also tested repeated fallocate() on an existing 8 GB file:
Filesystem Time
tmpfs 488 ms
tmpfs (huge=always) 348 ms
ext4 7 ms
On tmpfs, the time increases almost linearly with file size (around 60
ms/GB).
* How to reproduce *
Example configuration (cache on tmpfs):
cache: storage: /var/cache/knot-resolver size-max: 32G local-data: rpz: -
file: /etc/knot-resolver/hazard.rpz watchdog: true
Send a constant stream of DNS queries and append one record to the RPZ file:
echo "test.example IN A 192.0.2.1" >> /etc/knot-resolver/hazard.rpz
About 5-6 seconds later, query latency increases.
On our lab host, I reproduced this twice, with maximum response times of
781 ms and 615 ms. In both tests, delayed responses were sent at +5.96
seconds after the RPZ change.
* Workaround *
I used lua.policy-script-only with a script generated by kresctl convert
--type policy-loader, but removed the cache.open() line.
The RPZ file remains configured in local-data.rpz, so the watchdog
continues to reload the rules.
Results on the same lab host:
Default Without cache.open()
Max response time 615-781 ms 5 ms
Queries over 50 ms 8-17 0
Policy-loader runtime ~1.4-1.5 s 0.45 s
Workers blocked in futex 2 0
New RPZ rules applied yes yes
I also tested existing and new RPZ records, local addresses, PTR records,
NXDOMAIN responses, DNSSEC validation and normal resolution. Everything
worked as expected.
However, this workaround has two drawbacks:
1. With lua.policy-script-only, the policy-loader ignores local-data,
views and forward from the config. We have to generate these rules
ourselves using Ansible templates.
2. Changes in the lua section cause a rolling restart of all workers,
even when only lua.policy-script changes. I suspect this is because
config.lua is included in the worker-related config nodes in manager.py.
* Question *
Does the policy-loader actually need to open the cache? Could some features
depend on it, for example the cache “top” file or other rule types ?
In our setup, removing cache.open() fixes the latency spikes, and
everything we tested still works.
Would it make sense to skip cache.open() when it is not needed ?
Thanks!
Best regards,
Mateusz Masłowski
If Knot Resolver has no directive about trust anchors, it stills works
and validates automatically, great.
Now, I wanted to check the trust anchors in preparation for sunday's
rollover and I have questions:
* when you do not specify a trust anchors file, where is it stored?
I've checked many places on the disk without a result. (More general
question: how to display the entire configuration, with the default values?)
* when you do not specify a trust anchors file, how to check that you
indeed got the new root key?
% knot-resolver --version
6.4.0
Dear Knot Resolver users,
Knot Resolver 6.5.0 has been released!
Bugfixes:
- views: fix "unused tags" error when repeating a tag in views (!1883)
- watchdog: detect file-changes done via rename*() syscall (!1895)
This is relevant for RPZ files and TLS certs+keys.
- DoH fixes for sharing connections among different users (!1844)
- TCP-based connections used Nagle's algorithm since 6.2.0 (!1903)
This mistake significantly increased latencies of such transports.
- local-data: process upper-case names correctly (!1915)
Packaging:
- deb: migrate to sysusers and update packaging (!1857)
- rpm: fix source build against libknot >= 3.6 (!1907)
- python: 3.9+ (!1886), migrated to setuptools (!1889), removed
typing-extensions (!1910)
- Debian 11 is no longer supported
- Debian,Ubuntu: i386 and armhf architectures no longer supported
Improvements:
- decrease RAM usage by a lot (!1901)
- local-data: increase capacity to 63 GiB (logged MDB_MAP_FULL, !1885)
- systemd: use unit's {Runtime,State,Cache}Directory directives instead
of tmpfiles.d (!1853)
- controller: reimplement the notify socket in Python (!1875)
- python: improve Knot Resolver startup procedure (!1893)
Some startup actions previously handled by the manager have been
moved to a separate startup program.
- python: refactoring: utils (!1894), client (!1908)
- allow running under an unknown user, e.g. in containers (!1914)
Incompatible changes:
- remove legacy systemd unit files (kresd@.service,
kres-cache-gc.service) (!1853)
- remove $KRES_LOGGING_TARGET environment variable (!1893)
You can override the default by passing `--logtarget` to
`knot-resolver` command. (!1912)
- docker: remove cross-platform build for linux/arm/v7 (!1904)
Full changelog:
https://gitlab.nic.cz/knot/knot-resolver/raw/v6.5.0/NEWS
Sources:
https://knot-resolver.nic.cz/release/knot-resolver-6.5.0.tar.xz
GPG signature:
https://knot-resolver.nic.cz/release/knot-resolver-6.5.0.tar.xz.asc
Documentation:
https://www.knot-resolver.cz/documentation/v6.5.0/
--
Ales Mrazek
PGP: 3057 EE9A 448F 362D 7420 5A77 9AB1 20DA 0A76 F6DE
Hello,
I would like to report what appears to be a regression in Knot Resolver
6.4.2.
An RPZ rule loaded via local-data.rpz does not match queries if the rule's
owner name contains uppercase ASCII letters. Rules whose owner names are
entirely lowercase work correctly and match queries regardless of query
case.
Ordinary DNS lookups on the same resolver behave case-insensitively as
expected; for example, mixed-case queries such as GoGle.pl and WP.pl
resolve normally.
Affected version
Tested with:
-
knot-resolver6 6.4.2-cznic.1~bookworm
-
knot-resolver6-module-dnstap 6.4.2-cznic.1~bookworm
-
libknot16 3.5.8
-
libdnssec10 3.5.8
-
Debian 12 (bookworm)
Not affected
-
Knot Resolver 5.7.6 (cznic.1), with the same RPZ data loaded using Lua
policy.rpz()
Summary
DNS name matching is case-insensitive for ASCII letters. RFC 1034 §3.1 and
RFC 4343 specify that DNS name comparisons ignore ASCII case.
With local-data.rpz in Knot Resolver 6.4.2, however, a rule whose owner
name contains uppercase letters is never applied.
It does not match when the query:
-
is entirely lowercase,
-
uses exactly the same case as the RPZ owner name, or
-
is entirely uppercase.
Real-world example
Our RPZ blocklist contains eight entries with mixed-case owner names, for
example:
rabOna-7681.com IN A 10.15.20.254www.rabOna-7681.com IN A
10.15.20.254Italy9.cabaretclub.com IN A 10.15.20.254
totaIcasino.nl IN A 10.15.20.254
The same RPZ data is deployed on servers running Knot Resolver 5.7.6 and
6.4.2.
With 5.7.6 and policy.rpz():
$ dig www.rabona-7681.com @127.0.0.1
;; flags: qr aa rd ra;www.rabona-7681.com. 7200 IN A 10.15.20.254
With 6.4.2 and local-data.rpz:
$ dig www.rabona-7681.com @127.0.0.1
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN
$ dig rabOna-7681.com @127.0.0.1
(no RPZ answer)
None of the eight mixed-case entries receives the configured RPZ answer on
6.4.2. Lowercase entries from the same file work correctly.
Minimal reproducer
/etc/knot-resolver/case-test.rpz:
$TTL 300
@ IN SOA localhost. root.localhost. 1 3600 600 86400 300
@ IN NS localhost.
TeSt-case.example IN A 192.0.2.1
lower-case.example IN A 192.0.2.2
Relevant part of config.yaml:
local-data:
rpz:
- file: /etc/knot-resolver/case-test.rpz
kresctl validate --strict succeeds. I then run kresctl reload.
Results:
query status answer
lower-case.example NOERROR A 192.0.2.2
LOWER-CASE.example NOERROR A 192.0.2.2
test-case.example SERVFAIL none
TeSt-case.example SERVFAIL none
TEST-CASE.EXAMPLE SERVFAIL none
other.example SERVFAIL none
WP.pl NOERROR normal DNS answer
GoGle.pl NOERROR normal DNS answer
other.example, WP.pl, and GoGle.pl are control queries and are not present
in the RPZ file. WP.pl and GoGle.pl demonstrate that ordinary DNS
resolution works correctly and remains case-insensitive, while the
mixed-case RPZ owner name fails to match.
The lowercase rule matches queries regardless of case. The mixed-case rule
does not match any query, including a query using exactly the same case as
the owner name in the RPZ file.
Expected behavior
This rule:
TeSt-case.example IN A 192.0.2.1
should match:
test-case.example
TeSt-case.example
TEST-CASE.EXAMPLE
in the same way that the lowercase rule does, and in the same way that Knot
Resolver 5.7.6 with policy.rpz() behaves.
Actual behavior
An RPZ rule loaded through local-data.rpz is effectively ignored if its
owner name contains uppercase ASCII letters.
Workaround
Convert all owner names in the RPZ file to lowercase before loading the
file.
--
Best regards,
Mateusz Masłowski