Hi Vladimir, thanks for your reply. I'll do some more tests then (like decreasing the cache.size until it no longer fills up the tmpfs partition). Either way the documentation is a bit confusing to me because it says it sets the maximum but the maximum is just a hint that might get ignored. https://knot-resolver.readthedocs.io/en/stable/daemon.html#envvar-cache.size thanks, Christoph

On 5/9/19 11:04 PM, Christoph wrote: Yes, that's been "normal" behavior of kresd, at least so far.  I don't think it's documented, but when the cache is full (i.e. fails a write due to being full), the only coping is cache.clear() - in that state typically lmdb isn't able to commit *any* kind of changes (even clearing), so there's the "falling back" that removes the files and starts a new cache. We have a WIP on a "garbage-collecting" daemon that tries to remove data that are estimated as less useful when the cache is getting large, but so far typical deployments can afford setting the limit so large that it only fills up very rarely. I think it's actually typical to use tmpfs, at least for more "serious" deployments.  We certainly do that on our Turris routers (Omnia and MOX), as the writes could kill the flash storage soon.  I suppose persisting cache across system restarts isn't too useful :-)  and tmpfs is enough for persisting across daemon restarts. --Vladimir

On 5/10/19 9:27 PM, Christoph wrote: Yes, correct.  I agree it's surprising. I certainly haven't heard of anyone doing something similar.  It sounds possible, but I don't think it's worth it. I personally can't promise anything.  In any case, you can watch the issue: https://gitlab.labs.nic.cz/knot/knot-resolver/issues/257 Yes, the few seconds after clearing will be noticeably slower (I believe).  Records of the same name and type get updated in-place (with a short overlap, sometimes), so for each GB you'll typically need millions of *different* records to fill it - and that's not as fast as one might expect, because people mostly visit a not-too-huge set of sites, and these sets tend to overlap mostly.  You could try with your traffic and see how fast it grows (`du` command should be accurate until we have the GC). Now that I think of it, zram might be usable to extend the capacity in exchange for bearable latency hit on rarely used records, but I haven't tried to use it this way.  Swapping to an SSD also, I guess; directly placing the cache on an SSD would be more likely to cause too many writes, though it's possible almost all will be sequential writes and thus not too bad. --Vladimir

Thanks for the reference we enabled notifications on this issue so we get automatically updated if something happens there. I'm wondering why just a "few seconds". It will take much longer than a few seconds to restore the cache after a complete flush, no? (basically the same time it used to take to fill it to the point before the flush happened). We got such graphs out of the box since tmpfs usage is monitored as just another partition by system usage monitoring. It shows the mostly linear growth until it drops to 0 when the cache clear occurs. The resulting graph looks like a sawtooth wave with an interval of 3-4 days. https://en.wikipedia.org/wiki/Sawtooth_wave#/media/File:Waveforms.svg We could decrease the interval by increasing cache.size to maybe 2 weeks but that isn't a solution either since we would still start with an empty cache regularly - just less often. We don't know how big our user population is but I assume that public resolvers (like us) have a much more diverse user-base than typical ISP based resolvers that offer services for their customers in a geographically limited setting where users are more likely to visit overlapping destinations since most of them might share a single native language (less cache entries required). We are getting queries from over 50 countries (if we ignore countries with limited usage). https://twitter.com/applied_privacy/status/1127151981632606208 I assume that is a bad combination with a DNS resolver that does not support deleting cache entries without deleting all of them. Yes, we agree. So for the time being we switched back to unbound where we get around 50% cache hit rate with less than 1GB of cache size, but we are looking forward to test your future version that comes with a garbage collector. yes, we can test if there is a debian repository for it (otherwise we would wait for the releases reaching your stable repo) . your support via this mailing list is exceptional, thank you! Christoph

Thanks for providing these packages. We started directing some workload towards this version and will share our experiences once we see what happens when we are about to reach cache.size. In the meantime, here two minor suggestion for improvement with regards to your repo setup instructions at [1]: - I would find it more trustworthy if the signing key would be fetched from knot-resolver.cz (instead of opensuse.org). (when installing the key we went through the steps to find the same key on your domain which took a bit of time) - since the repo supports https maybe replace http with https URLs [1] https://software.opensuse.org//download.html?project=home%3ACZ-NIC%3Aknot-r…

Tomas Krizek: Your package does not appear to include a systemd service file for this daemon. Will this do fine? -- [Unit] Description=Knot Resolver Garbage Collector daemon [Service] Type=notify WorkingDirectory=/var/cache/knot-resolver ExecStart=/usr/sbin/kr_cache_gc -c /var/cache/knot-resolver -d 60000 User=knot-resolver Restart=on-abnormal [Install] WantedBy=kresd.target --

Thanks, since the gc daemon runs the cache size stopped growing, so I assume it works as intended so far. Actual current cache size was above 90% of cache.size when starting the daemon. There are kr_cache_gc syslog messages that say it deleted records but the cache size did not actually decrease when looking at the cache filesize.

Vladimír Čunát: Thanks for confirming. I somewhat assumed it because the documentation says you can not reduce the LMDB size.