Web application firewalls, DDoS protection, and bot traffic, including AI crawlers

A state agency announces new benefit rules, and its site slows to a crawl the same morning. Real visitors are part of it, but the logs show most requests coming from a few thousand addresses hammering the search page and login form. Some are attack tools, some are price-scraping bots, and a growing share are AI crawlers copying the whole site. None of this traffic is wanted, and all of it competes with the people who actually need the information. Three layers of protection deal with it: a web application firewall, DDoS protection, and bot management.

Web application firewall (WAF)

A WAF sits in front of your site and inspects each request before it reaches WordPress, Drupal, or your web server. It blocks requests that match known attack patterns, such as attempts to inject database commands, upload scripts, or probe for files that should not be public. Managed rule sets cover the common attacks (the OWASP Top 10) and are updated by the provider as new threats appear. Custom rules handle what is specific to your site, such as limiting who can reach the login page or blocking countries you never serve.

A WAF does not replace patching. It reduces the window between a flaw being published and the fix being applied, and blocks a lot of automated noise. See keeping servers patched.

DDoS protection

A distributed denial-of-service (DDoS) attack tries to overwhelm a site with traffic from many machines at once. Large network-level floods are absorbed by the cloud provider's edge network; on AWS, Shield Standard protects every account automatically. Smaller, smarter floods that look like real page requests are handled by WAF rate rules and by a CDN serving cached pages, so the CMS never sees most of the load. Government sites are a frequent DDoS target around elections, announcements, and politically charged events, so plan protection before those dates, not during.

Bots and AI crawlers

Not every bot is bad. Search engines need to crawl your site, and monitoring services check it is up. The problem is volume and intent:

A robots.txt file asks crawlers to stay away from parts of your site, and the well-known AI crawlers honor it. Many others do not. Enforcement happens at the WAF or CDN, where bot-management rules identify crawlers and limit or block them.

What you should decide

1. Your AI crawler policy. Public information sites, such as agencies and universities, often want their content findable through AI search. Others want it kept out of training data. Decide, write it down, and set robots.txt and WAF rules to match.
2. Which pages need extra protection. Login, search, forms, and any page that queries the database on every request.
3. Who may reach the admin area. Restricting the CMS admin to your network, VPN, or single sign-on blocks most attacks on it outright.
4. When your peak dates are. Elections, enrollment, budget releases, and emergencies are when protection matters most.

Notes by platform

WordPress: a WordPress-specific rule set blocks attacks on wp-login.php and xmlrpc.php, the two most attacked addresses on any WordPress site. Drupal: protect /user/login and search with rate rules, and keep Drupal's own flood control on. Cascade: published Cascade sites are static files behind a CDN, which already resists most of these attacks; see securing a Cascade site: WAF, headers, and hardening.

How eWay handles it

On environments eWay manages, a WAF runs managed OWASP Top 10 rules plus custom rules for each site, with rate and IP controls, and a WordPress-specific rule set where relevant. Always-on DDoS protection (AWS Shield, or Azure DDoS Protection on Azure) sits at the edge. WAF allow and block activity is one of the signals eWay monitors, and pre-scaling is planned ahead of known peaks rather than left to reactive autoscaling.

What to include in a support ticket

• The severity you would assign, P1 to P4 (see severity levels and response times)
• When the slowdown or attack started, and the pages affected
• Your AI crawler policy, if you want rules changed
• Upcoming peak dates

eWay customers can open a ticket in the support portal or email support@ewaycorp.com. Related reading: scaling WordPress and Drupal for traffic spikes.