iptables -A INPUT -m state --state NEW -m udp -p udp --dport 123 -m hashlimit --hashlimit-upto 100/second --hashlimit-burst 115 --hashlimit-mode srcip --hashlimit-srcmask 32 --hashlimit-name ntp -j ACCEPT
And second:
iptables -A INPUT -m state --state NEW -m udp -p udp --dport 123 -m hashlimit --hashlimit-above 100/second --hashlimit-burst 115 --hashlimit-mode srcip --hashlimit-srcmask 32 --hashlimit-name ntp -j DROP
Then check with iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp – anywhere anywhere state NEW udp dpt:ntp limit: up to 100/sec burst 115 mode srcip
DROP udp – anywhere anywhere state NEW udp dpt:ntp limit: above 100/sec burst 115 mode srcip
When you run: iptables-save it should stick at every reboot.
I checked with: chronyc -n clients |sort -rn -k 3,3 | head
And the real bad clients seem to have vanished.
Can somebody confirm this? As I do not want to overload my router with firewall-rules. Looks like Linux is better at this. I just want them to go away!
So, please tell me if this works. My friend in Leuven uses this to fight bad clients.
I don’t really understand what this solves that rate limiting in Chrony isn’t already doing? Or is the goal to stick your head in the sand by not seeing them in Chrony, even though they’re still pounding on your router?
Chrony first sends them a KoD packet, telling them to knock it off. Then it stops responding and ignores them for a while, which from the client perspective looks like the server went completely dead. These clients may still show up in Chrony’s stats still, but rest assured, they’re not getting any response or signs of life.
I’d say it’s better to let Chrony handle it. When it sends a KoD packet, any half decent client will heed it and back off. Bad clients that don’t heed it, well, they’re gonna keep hammering you either way, consuming your inbound bandwidth and burdening your router. Blocking them at the router won’t free up any bandwidth, and just means your router will have to do even more processing.
You cannot prevent packets coming to your server. This is out of your control. What these rules do is block/drop packets if a mad-gone client, that ignored KoD, is sending too many of them to your server. From the client’s POV, it’s as if your server is unreachable/not there. Are clients smart enough to notice this and redirect their hunger to some other server? That I do not know.
Quite the contrary in my experience. I drop incoming traffic to some of my servers’ addresses for one day a week in iptables to try to encourage people to update their configurations. Here’s the incoming and outgoing traffic for one of those obsolete addresses (131.111.8.171) over the course of a week:
Is that to say your NTP server buckled under the load, but the router’s not even breaking a sweat? I would’ve thought the router was the weak link in the chain, and adding more rules for the router to process means more processing for everything that goes through. Or doesn’t go through, for that matter.
Or is it more for aesthetic purposes, to clean up the graph?
Yeah, these are broken NTP clients. Still, they are unwanted for me so I drop them. My rules are on the server itself. The router’s firewall only has the ports 123(udp) and 4460(tcp) open. The server’s FW drops them. I lowered the setting from 100 to 50 packets/s and while I can see more is dropped, I do not have clients that increase their rate when you drop them, else I’d see that in the amount of packets dropped in the FW which will also increase by a lot.
From my understanding, that’s precisely what it’s supposed to do. That is, aside from a small “leak” to prevent bad clients from doubling down by increasing the rate of their requests when they get no response.
But, it would be interesting to see some numbers from your router’s perspective. Since Chrony no longer sees this traffic, have they actually gone away, or are you merely putting your head in the sand while the problem persists? It should be possible to enable logging for specific firewall rules. Or does something like tcpdump have the ability to measure packets per second maybe?
My point is, are they still trying, even thought they’re not reaching Chrony? Because if so, your router still has to deal with them.
It might be cheaper for your router to simply pass everything through, since many routers have hardware acceleration for forwarding traffic, bypassing the CPU. Once firewall rules are introduced, the CPU gets involved, and every single packet arriving has to be compared to every firewall rule. Your server may have less of a challenge filtering this traffic.
So again, it would be interesting to see some numbers from the router, rather than Chrony.
Maybe good to add that at least in the case of NTPd:
When a client connects (requests time) from the server it’s IP is written to a list (the MRU list);
Ratelimit is applied to clients in this MRU list who exceed the ratelimit settings as defined in discard…
The thing is, historically the default size of the MRU list has been only 600 IPs.
As you can imagine, this is woefully inadequate for busy servers that serve millions of clients…
I know that @davehart has proposed (a long time ago) to increase MRU list to 32,000 entries for such busy servers. Please see the following link if you want to know more.
@Bas I can imagine that having too few entries in the list does not really help identifying/tracking all abusive clients. This is also valid for chrony, which has a similar option: “clientloglimit”.
Default chrony setting for clientloglimit is 4096 IPs. Maximum setting is 16.7 million IPs. Beware that this maximum setting consumes 2GB of RAM. Each entry consumes 128 bytes.
As I understand the Chrony docs, clientloglimit is set in bytes. So, clientloglimit 33554432 for 32MB. With 128 bytes per client, that’s enough for 262144 clients.
Indeed. chrony does a very good job at keeping the response rates sane, there’s no need to get iptables involved for rate limiting. The “ratelimit” config directive handles chrony’s rate limiting. There are options to tune ratelimit’s behaviour, but the defaults are actually fairly good.
Then there’s the “clientloglimit” directive. I have set it to 268435456 on my servers. That may be considered a fairly large value, but my servers aren’t tight on memory and I don’t mind allocating 256 MB for this purpose. Besides, some of my servers serve 40-50k requests per second at busier periods, so a larger clientloglimit may be useful.
Edit/addendum/note to future self regarding setting clientloglimit to 256M (~2 million IPs): Requests per second may give an indication of how big the clientloglimit should be set to, but ultimately the key is the unique IP addresses served within some specific time period. I captured 10 minutes of traffic on my two busiest servers and checked how long it takes to reach 2M unique IP addresses. The server that serves the China zone took about 4 minutes to reach 2M unique IP addresses at 33290 qps. It appears that 10 minutes was a slightly too short measuring period for my server that serves the Singapore zone, because within 10 minutes I accumulated only 1946861 unique IP addresses at 25672 qps. Traffic on both of those servers was a little bit quieter today than usually. All in all, the 256M clientloglimit does not seem particularly excessive in my case, but if you’re not doing 10k+ qps a smaller clientloglimit will likely suffice. YMMV, do your own measurements.
