Server under DDoS or a traffic flood: first steps
First confirm that it really is a flood (a distributed denial-of-service, or DDoS, attack) and not a broken deploy or a slow database. Then contact your hosting provider early: a large attack fills your network link before your server sees it, and only the upstream network can filter it. On the server, rate limits in nginx and the kernel's SYN cookies help against smaller application-level and SYN floods.
Do this first.
- Note the time and take a quick snapshot of connections and logs (commands below). Your host will ask for this.
- Contact your hosting provider or upstream network now, with the target IP address, port and start time. Ask what mitigation they offer.
- Keep a console or out-of-band connection open. SSH may become unusable during the attack.
- Don't turn on extra debug logging. Writing millions of log lines can take the server down faster than the attack.
Symptoms
- The site or service is slow or unreachable, while the server's own CPU and disk look normal or are busy only with the network.
- A sudden jump in traffic, connections or requests from many addresses, often to a single URL or port.
- The kernel log shows
nf_conntrack: table full, dropping packetorpossible SYN flooding on port 443. - Your provider tells you about an attack, or has null-routed (blackholed) your IP address to protect its network.
Rule out the ordinary causes too. A bot or crawler gone wild, a marketing campaign or a slow backend can look similar (see High load or out of memory).
Confirm: what is hitting you
# Socket summary and TCP state counts
ss -s
ss -Htan | awk '{print $1}' | sort | uniq -c | sort -rn
# Half-open connections (a SYN flood shows thousands here)
ss -Htn state syn-recv | wc -l
# Top 20 client addresses with established connections to ports 80/443
ss -Htn state established '( sport = :80 or sport = :443 )' \
| awk '{print $4}' | sed 's/:[0-9]*$//' | sort | uniq -c | sort -rn | head -20
# Kernel messages and the connection-tracking table
sudo dmesg -T | grep -iE 'syn flooding|conntrack' | tail
sudo sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
On older systems with net-tools, netstat -ant | awk '{print $6}' | sort | uniq -c gives the same state counts. With a state filter, ss leaves out the State column, so the client address is in column 4. The conntrack lines work only when the connection-tracking module is loaded.
For HTTP floods, check what is being requested. Only read the end of a large log:
# Top client IPs and top URLs in the last 100,000 requests
sudo tail -n 100000 /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
sudo tail -n 100000 /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
# Requests per minute (time field of the combined format)
sudo tail -n 100000 /var/log/nginx/access.log | awk '{print substr($4, 2, 17)}' | uniq -c | tail
A few addresses sending most of the requests can be blocked on the server. Thousands of addresses each sending a little, or traffic that fills your bandwidth, need help from upstream.
Contact your host or upstream
CISA's DDoS guidance puts this near the top of the list: tell your ISP or hosting provider, give them the attacking addresses and the target, and ask them to filter. They may be able to reroute traffic through a scrubbing service, or filter by port and packet size. Collect timestamps, addresses, sample log lines and, if you can, a short packet capture (sudo tcpdump -ni eth0 -c 2000 -w /tmp/flood.pcap; adjust the interface name). Many providers include basic DDoS filtering and sell stronger options. Ask what applies to your plan.
Options to consider, depending on your budget and setup:
- Your provider's network filtering or scrubbing. Often the fastest option, because nothing in DNS changes.
- A CDN or reverse proxy with DDoS protection in front of your website. It absorbs and filters HTTP traffic. You point DNS at the service, and the origin server should then accept web traffic only from the service's address ranges, or attackers can bypass it.
- A dedicated DDoS scrubbing service for non-web services or whole IP ranges, usually by contract.
- More capacity (bigger link, more servers behind a load balancer) for floods that are only slightly above your normal capacity.
On the server: nginx limits
Per-IP limits stop a single address from taking all workers. Put this in the http block:
limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;
limit_conn_zone $binary_remote_addr zone=peraddr:10m;
limit_req_status 429;
limit_conn_status 429;
limit_req_log_level warn;
Then add this in the server or location blocks you want to protect:
limit_req zone=perip burst=20 nodelay;
limit_conn peraddr 20;
sudo nginx -t && sudo systemctl reload nginx # reloads the service
On a 64-bit server, one megabyte of limit_req zone holds about 8,000 addresses, so 10 MB holds about 80,000. Choose limits above what a real visitor needs. A page with 40 images over HTTP/2 counts each request, so test first with limit_req_dry_run on; and watch the error log. Many visitors behind one office or mobile carrier NAT share one IP address.
On the server: kernel and firewall
SYN cookies let the kernel keep answering new connections when the SYN backlog fills up. The default value is 1 (used only when the queue overflows). Check it:
sysctl net.ipv4.tcp_syncookies # should print 1
sudo sysctl -w net.ipv4.tcp_syncookies=1
To drop a handful of heavy addresses for an hour with nftables, use a separate table, which works alongside firewalld or ufw:
sudo nft add table inet flood
sudo nft add set inet flood block '{ type ipv4_addr; flags timeout; }'
sudo nft add chain inet flood input '{ type filter hook input priority -10; policy accept; }'
sudo nft add rule inet flood input ip saddr @block drop
sudo nft add element inet flood block '{ 198.51.100.7 timeout 1h }'
# remove everything later:
sudo nft delete table inet flood
Blocking single addresses helps only against a few sources. It does nothing for traffic that already fills your network link.
Don't overload logging
- Buffer the access log:
access_log /var/log/nginx/access.log combined buffer=64k flush=5s; - Skip logging for the flooded static URLs with a
mapandif=condition, oraccess_log off;in that location for the duration. limit_req_log_level warn;(set above) logs rejections at warn level. With nginx's default error log level (error), they are then not written at all. That saves disk writes during a flood, but the fail2bannginx-limit-reqjail then has nothing to read. Remove the line if you rely on that jail.- Watch disk space (
df -h /var). A full disk is a second outage (see Disk full). - Don't run fail2ban with very short
findtimeon huge logs during a large flood. Banning thousands of single addresses costs CPU and won't help much.
Confirm it is under control
- The site loads from an outside network, and response times are back to normal.
ss -Htn state syn-recv | wc -land the established count are back near your usual numbers.- The access log shows status 429 for the flood sources, not for normal visitors.
- Your provider confirms whether filtering or a null route is still active.
Prevent and prepare
- Know your normal traffic levels, so that you can spot an abnormal jump.
- Write down your provider's abuse/NOC contact and what DDoS protection your plan includes, before you need it.
- Decide in advance whether you would put a CDN or scrubbing service in front, and test the DNS change.
- Keep the origin server's real IP address private if you use a proxy service. Don't send mail from it or publish it in DNS records.
- The UK NCSC suggests four steps: understand your service, understand your defences, create a response plan and test it.
When to call a professional
Call your provider as soon as bandwidth is saturated or the server is unreachable. You can't fix that from inside the server. Consider a DDoS mitigation specialist if attacks repeat, come with a ransom demand, or hit services you can't put behind a CDN. Report extortion attempts to the police. If your server is the source of attack traffic, it has probably been compromised: see Abuse complaint and Signs of compromise.
Sources
- Understanding and Responding to Distributed Denial-of-Service Attacks (PDF, March 2024), CISA, FBI and MS-ISAC
- Denial of Service (DoS) guidance, UK National Cyber Security Centre
- ss(8), Linux manual pages
- IP Sysctl (tcp_syncookies), Linux kernel documentation
- Module ngx_http_limit_req_module, nginx documentation
- Module ngx_http_limit_conn_module, nginx documentation
- Module ngx_http_log_module, nginx documentation
- nft(8), Debian manpages