Performance Zen

Category: Apache

  • Slowing The Flood: Firewall Tuning – Month 1

    On Apr 01 2026, I reset my Apache server collection tracking so that the noisiest data prior to the implementation of my new firewall configuration would be eliminated. Since then, the firewall has seen some tuning and iteration.

    • Apr 01-03 2026: Initial tuning phase of the firewall and addition of largest blocks of known Hosting Providers and other malicious actors
    • Apr 04-28 2026: Steady-state of firewall with occasional additions of new ASNs to the blocklist. During this period, all the major cloud providers (AWS, GCP, & Azure) were in a complete block state.
    • Apr 29 2026 – May 07 2026: Impose rate limiting on Amazon/AWS and Google/GCP ASNs and much stricter rate-limiting on the Microsoft/Azure ASN. As well, integrated a process to add the AbuseIPDB Top 10K IPs dynamically on a schedule.

    This process has had some noticeable effects on the volume of traffic captured in my logs. Although I pruned the data prior to Apr 01 2026, I will just say that the volume was much higher than you see at the start of the chart below.

    Vistitor Traffic – Apr 01 – May 07 2026

    I can say that the new tuned firewall deployment is working extremely well on my ancient hardware and that working alongside my friend Claude has got this system to a state where it will likely continue to support a great deal of traffic for hopefully years to come.

  • Apache and 403 Responses — HTTP/2.0 v. HTTP1.1

    I’ve spent a good part of the last two days trying to track down an issue that was bothering me. My server is tuned to send a lot of annoying bots to the scrap heap with Rewrite rules that return a 403 response. I also just converted the server to HTTP/2.0 (yeah, I know; quiet in the back).

    However, many of the bots use HTTP/1.1. What was weird is that when you look at the logs in Apache, you get the following items.

    172.232.187.115 - - [06/May/2026:18:51:26 +0000] "GET / HTTP/1.1" 403 2877 "-" "Mozilla/5.0 (iPod; U; CPU iPhone OS 3_1 like Mac OS X) AppleWebKit/534.39.5 (KHTML, like Gecko) Version/3.0.5 Mobile/8B116 Safari/6534.39.5"

172.232.187.115 - - [06/May/2026:18:51:42 +0000] "GET / HTTP/2.0" 403 90 "-" "Mozilla/5.0 (iPod; U; CPU iPhone OS 3_1 like Mac OS X) AppleWebKit/534.39.5 (KHTML, like Gecko) Version/3.0.5 Mobile/8B116 Safari/6534.39.5"

    Can anyone spot the issue? Well, if you look closely, you’ll see that HTTP/1.1 response is recorded as being much larger than that of the HTTP/2.0 response for the same 403 response.

    Guess what? This is an artifact of the way that Apache processes these requests! My friend Claude described it this way:

    For HTTP/1.1, when [F] fires, Apache generates the full default error page first (2911 bytes), logs that size via %b, then ErrorDocument substitutes it with the 44-byte response before sending. The log records the pre-substitution size.

    For HTTP/2.0, mod_http2 logs the post-substitution size (plus HTTP/2 frame overhead accounting for the extra 82 bytes above 44).

    It’s always fun to go off on a Snipe Hunt and learn a lot about the internals of software you use every day.