Hi Daniel
I've made the suggested changes, a slight improvement I think, but load is still fairly high:
Any further suggestions greatly appreciated.
Thanks
Jonathan
Hi Jonathan
Try increasing the number or transmit channels:
ethtool -L eno1 rx 4 tx 4
And reduce the number of udp-workers
Daniel
On 10/8/20 2:12 AM, Jonathan Hewlett wrote:
> Hi Danel,
> I'm sorry for the late reply; I've been on holiday/vacation and playing catch-up! ;)
>
> ethtool:
> Channel parameters for eno1:
> Pre-set maximums:
> RX: 4
> TX: 4
> Other: 0
> Combined: 0
> Current hardware settings:
> RX: 4
> TX: 1
> Other: 0
> Combined: 0
>
> knotc output:
>
> Knot DNS 2.9.5
>
> Target: linux-gnu x86_64
> Compiler: gcc
> CFLAGS: -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches
> -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -DNDEBUG
> -Wno-unused -Wall -Wshadow -Werror=format-security -Werror=implicit -Wstrict-prototypes
> LIBS: -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld
> LibURCU: -lurcu
> GnuTLS: -lgnutls -I/usr/include/p11-kit-1
> Libedit: -ledit -I/usr/include/editline
> LMDB: shared -llmdb
> Config: 500 MiB default mapsize
>
> Prefix: /usr
> Run dir: /run/knot
> Storage dir: /var/lib/knot
> Config dir: /etc/knot
> Module dir:
>
> Static modules: cookies dnsproxy dnstap geoip noudp onlinesign queryacl rrl stats synthrecordwhoami
> Shared modules:
>
> Knot DNS libraries: yes
> Knot DNS daemon: yes
> Knot DNS utilities: yes
> Knot DNS documentation: yes
>
> Use recvmmsg: yes
> Use SO_REUSEPORT(_LB): yes
> Memory allocator: auto
> Fast zone parser: yes
> Utilities with IDN: libidn2
> Utilities with Dnstap: yes
> MaxMind DB support: yes
> Systemd integration: yes
> POSIX capabilities: yes
> PKCS #11 support: yes
> Ed25519 support: yes
> Ed448 support: no
> Code coverage: no
> Sanitizer: no
> LibFuzzer: no
> OSS-Fuzz: no
>
> Continued thanks! ;)
>
> Jonathan
>
> On Thu, 24 Sep 2020 at 11:14, Daniel Salzman <daniel.salzman@nic.cz <mailto:daniel.salzman@nic.cz>> wrote:
>
> Hi Jonathan,
>
> If you have just one 1GbE port, I would recommend reducing the number of udp-workers. What about 12 or less?
>
> How do you install Knot DNS (repository or source code)?
>
> What do say these commands?
> # sudo ethtool -l eno1
> # knotc status configure
>
> Daniel
>
> On 9/23/20 5:45 PM, Jonathan Hewlett wrote:
> > Hi Daniel
> >
> > * Server listens: 967 (multiple v4 and v6 networks)
> > * Physical server
> > * NIC details
> > *-network:0
> > description: Ethernet interface
> > product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
> > vendor: Broadcom Inc. and subsidiaries
> > physical id: 0
> > bus info: pci@0000:02:00.0
> > logical name: eno1
> > version: 00
> > serial: b0:83:fe:c8:f3:ec
> > size: 1Gbit/s
> > capacity: 1Gbit/s
> > width: 64 bits
> > clock: 33MHz
> > capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
> > configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=3.137 duplex=full firmware=FFV21.60.2 bc 5720-v1.39 ip=38.88.50.50 latency=0 link=yes multicast=yes port=twisted pair
> > speed=1Gbit/s
> > resources: irq:16 memory:d80a0000-d80affff memory:d80b0000-d80bffff memory:d80c0000-d80cffff memory:dd800000-dd83ffff
> >
> > Thanks for the continuing support
> > Jonathan
> >
> >
> > On Wed, 23 Sep 2020 at 12:47, Daniel Salzman <daniel.salzman@nic.cz <mailto:daniel.salzman@nic.cz> <mailto:daniel.salzman@nic.cz <mailto:daniel.salzman@nic.cz>>> wrote:
> >
> > Jonathan,
> >
> > there must be something wrong with the networking. How many `server.listen` configuration values
> > do yo have? Is it a physical server or a virtual one? What is the network card used?
> >
> > Daneil
> >
> > On 9/23/20 1:18 PM, Jonathan Hewlett wrote:
> > > Daniel
> > > Thank you for your time
> > >
> > > # knotc status workers
> > > UDP workers: 24, TCP workers 24, background workers: 10 (running: 0, pending: 0)
> > >
> > > image.png
> > >
> > > Kind regards
> > > Jonathan
> > >
> > >
> > > On Wed, 23 Sep 2020 at 07:59, <daniel.salzman@nic.cz <mailto:daniel.salzman@nic.cz> <mailto:daniel.salzman@nic.cz <mailto:daniel.salzman@nic.cz>> <mailto:daniel.salzman@nic.cz
> <mailto:daniel.salzman@nic.cz> <mailto:daniel.salzman@nic.cz <mailto:daniel.salzman@nic.cz>>>> wrote:
> > >
> > > Hi Jonathan,
> > >
> > > On 2020-09-22 23:11, Jonathan Hewlett wrote:
> > > > Hi,
> > > >
> > > > We run around 50 knot servers around the world, with great success -
> > > > thanks for the software!
> > >
> > > Thanks for this feedback :-)
> > >
> > > >
> > > > We are just a little curious about seemingly high CPU usage, and how
> > > > best to interpret the results, if someone could advise? Specifically,
> > > > we are seeing the majority of the cpu time in 'system' and not 'user',
> > > > what would that suggest is happening perhaps?
> > >
> > > Could you provide me the output from `knotc status workers`?
> > >
> > > I like htop. Are you able to send me a screenshot when running htop? :-)
> > > [Don't forget to configure:
> > > F2
> > > Display options->Detailed CPU time
> > > (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
> > > F10
> > > ]
> > >
> > > Best,
> > > Daniel
> > >
> > > >
> > > > Any tuning help gratefully received!
> > > >
> > > > Our platform (sample)
> > > >
> > > > 24 Core CPU E5-2440 0 @ 2.40
> > > > 64Gb RAM
> > > > SSD
> > > >
> > > > knot-2.9.5, running on Centos 8.2
> > > >
> > > > Currently receiving around 4k pps
> > > >
> > > > Server is authoritative for a 278 zones, the largest zone file being
> > > > around 600Mb
> > > >
> > > > Output of top
> > > >
> > > > top - 20:50:38 up 6 days, 4:49, 2 users, load average: 13.47,
> > > > 12.34, 11.83
> > > > Tasks: 321 total, 2 running, 318 sleeping, 0 stopped, 1 zombie
> > > > %Cpu0 : 4.0 us, 34.1 sy, 0.0 ni, 60.3 id, 0.3 wa, 0.7 hi, 0.7
> > > > si, 0.0 st
> > > > %Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id, 0.0 wa, 0.3 hi, 0.7
> > > > si, 0.0 st
> > > > %Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id, 0.0 wa, 0.7 hi, 0.3
> > > > si, 0.0 st
> > > > %Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id, 0.0 wa, 0.3 hi, 0.3
> > > > si, 0.0 st
> > > > %Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id, 0.0 wa, 0.3 hi, 0.0
> > > > si, 0.0 st
> > > > %Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id, 0.0 wa, 0.3 hi, 0.3
> > > > si, 0.0 st
> > > > %Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.3 hi, 0.0
> > > > si, 0.0 st
> > > > %Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id, 0.0 wa, 0.3 hi, 0.0
> > > > si, 0.0 st
> > > > %Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id, 0.0 wa, 0.7 hi, 0.0
> > > > si, 0.0 st
> > > > %Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id, 0.0 wa, 0.3 hi, 0.0
> > > > si, 0.0 st
> > > > %Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id, 0.0 wa, 0.7 hi, 0.0
> > > > si, 0.0 st
> > > > %Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
> > > > si, 0.0 st
> > > > %Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id, 0.0 wa, 0.7 hi, 0.3
> > > > si, 0.0 st
> > > > %Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
> > > > si, 0.0 st
> > > > %Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id, 0.0 wa, 0.7 hi, 3.7
> > > > si, 0.0 st
> > > > %Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id, 0.0 wa, 0.3 hi, 0.3
> > > > si, 0.0 st
> > > > %Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id, 0.0 wa, 1.0 hi, 4.0
> > > > si, 0.0 st
> > > > %Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id, 0.0 wa, 0.3 hi, 0.0
> > > > si, 0.0 st
> > > > %Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id, 0.0 wa, 0.7 hi, 4.0
> > > > si, 0.0 st
> > > > %Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id, 0.0 wa, 0.3 hi, 0.3
> > > > si, 0.0 st
> > > > %Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id, 0.0 wa, 0.7 hi, 1.3
> > > > si, 0.0 st
> > > > %Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id, 0.0 wa, 0.3 hi, 0.3
> > > > si, 0.0 st
> > > > %Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id, 0.0 wa, 0.7 hi, 3.7
> > > > si, 0.0 st
> > > > %Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id, 0.0 wa, 0.3 hi, 0.0
> > > > si, 0.0 st
> > > > MiB Mem : 64107.4 total, 46764.9 free, 11733.6 used, 5608.8
> > > > buff/cache
> > > > MiB Swap: 32248.0 total, 32248.0 free, 0.0 used. 51670.0 avail
> > > > Mem
> > > >
> > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> > > > COMMAND
> > > > 213370 knot 20 0 34.0g 9.6g 41784 S 644.9 15.3 1319:12
> > > > /usr/sbin/knotd
> > > >
> > > > Thanks again for this great product.
> > > >
> > > > Jonathan
> > >
> > >
> > >
> > > --
> > > --\--
> > > Jonathan Hewlett
> > > ^Senior\ Systems\ Engineer\n$
> > > CentralNic Group plc (LSE:CNIC)
> > > https://www.centralnic.com <https://www.centralnic.com/>
> > > Tel: +44.7548243020
> > > CentralNic Group plc is a company registered in England and Wales with company number 8576358. Registered Offices: Saddlers House, Gutter Lane, London EC2V 6AE.
> >
> >
> >
> > --
> > --\--
> > Jonathan Hewlett
> > ^Senior\ Systems\ Engineer\n$
> > CentralNic Group plc (LSE:CNIC)
> > https://www.centralnic.com <https://www.centralnic.com/>
> > Tel: +44.7548243020
> > CentralNic Group plc is a company registered in England and Wales with company number 8576358. Registered Offices: Saddlers House, Gutter Lane, London EC2V 6AE.
>
>
>
> --
> --\--
> Jonathan Hewlett
> ^Senior\ Systems\ Engineer\n$
> CentralNic Group plc (LSE:CNIC)
> https://www.centralnic.com <https://www.centralnic.com/>
> Tel: +44.7548243020
> CentralNic Group plc is a company registered in England and Wales with company number 8576358. Registered Offices: Saddlers House, Gutter Lane, London EC2V 6AE.