How to stop bots and scrapers from eating your bandwidth
A surprising slice of your traffic is robots — and not the good kind. Here is how to spot the scrapers eating your bandwidth, tell them apart from Google, and shut them out at the edge.
A big chunk of web traffic is automated, and a lot of that is bots you don't want: scrapers, credential-stuffers, content thieves. The fix isn't to block all bots (you need Google) but to tell good from bad and stop the bad ones at the edge — before they cost you bandwidth, slow your site, or scrape your prices.
If you've ever looked at your raw traffic logs and thought "who are all these visitors and why are none of them buying anything", congratulations: you've met your bots. On a typical site, a large share of requests never come from a human at all. Some of those bots are essential. Most of the rest are a tax on your bandwidth and your nerves.
Good bots, bad bots
Bots you want
- Googlebot, Bingbot — they index you
- Uptime monitors you set up
- Legit link previews (Slack, social)
Bots you don't
- Scrapers copying your content and prices
- Credential-stuffers trying stolen logins
- Spam bots hammering your forms
The goal isn't a wall. It's a bouncer — something that recognizes the regulars, waves through the legitimate visitors, and turns away the troublemakers.
How to spot the bad ones
Bad bots leave fingerprints. None of these is proof on its own, but together they paint a clear picture.
Inhuman request rates
A real person doesn't load 60 product pages in 4 seconds. A scraper does, all day.
Fake or missing identities
Bots that claim to be Googlebot but come from the wrong networks are lying. Real crawlers verify.
Boring, mechanical patterns
Hitting URLs in perfect alphabetical order, ignoring your images and CSS, never moving the mouse.
Suspicious geography
A sudden flood from a data-center range that's never bought anything is a tell.
Don't block by user-agent string alone. It's trivial to fake. A bot claiming to be Chrome on a Mac might be a script on a server in a basement.
Stopping them at the edge
The key word is edge. If you block a scraper at your origin server, it has already cost you a request, some CPU, and some bandwidth. Block it at the edge — out at the PoP nearest the bot — and the bad request never reaches you at all. Your origin stays calm and your bills stay sane.
A few layers work well together:
| Layer | What it does |
|---|---|
| Rate limiting | Caps how many requests one source can make in a window |
| Verified-bot allowlist | Confirms Googlebot is really Google, blocks the impostors |
| Proof-of-work challenge | Makes a client do a tiny puzzle — invisible to humans, expensive for bots at scale |
| Network blocklists | Drops known-bad IPs across your whole network in seconds |
Proof-of-work challenges are the polite option. A real browser solves the puzzle in a blink and the visitor never notices. A bot farm running thousands of requests suddenly has to spend real CPU on each one — which usually makes the whole operation not worth it.
You can't block every bot, and you wouldn't want to — some of them are why people find you. Spot the bad ones by behavior, not just their claimed identity, and stop them at the edge so they never cost you. Layer rate limits, verified-bot checks and challenges, and the scrapers quietly give up and go bother someone else.
See how NordicCDN does this for your site:
Mads has worked in IT — mostly hosting — since he was 16. He took an early stake in a SaaS company and helped grow it through to its acquisition by Visma, has built and run data-center networks, and served as CTO of a Danish data center. He started NordicCDN to make fast, secure infrastructure simple to use.
