Meta’s web bots pushed nine billion requests in a single quarter this year, and the outbound traffic almost never made it back to the sites the crawlers scraped. Cloudflare is now defaulting new domains to block training and agent bots on any page that shows ads. The line between “search visit” and “training corpus” has moved, and if you run a site, you probably want to sit on the “blocking” side of it.
The best apps for blocking AI web scrapers on desktop in 2026 fall into three buckets: edge services that block at the network edge before requests reach your origin, self-hosted proof-of-work firewalls that run alongside your app, and origin-side rules that filter requests after they arrive. We picked seven that work together, from a single Cloudflare toggle to Anubis (already deployed by the Linux kernel source, Codeberg, and FFmpeg).
What to look for in an AI scraper blocking tool
- User-agent and behavioral detection combined. Any tool that only checks the User-Agent string is a speed bump, not a wall.
- Ability to distinguish training bots from search crawlers. You want GPTBot blocked, Googlebot allowed.
- Low false positive rate on real humans. Proof-of-work challenges must degrade gracefully on old phones and screen readers.
- Visibility. A dashboard telling you which bots hit you last week is worth more than a black-box “we blocked stuff” claim.
- Cost that matches your traffic. A personal blog does not need a five-figure WAF contract.
Quick comparison
| Tool | Best for | Free plan | Paid | Deployment |
|---|---|---|---|---|
| Cloudflare AI Crawl Control | Any site behind Cloudflare | Included on all tiers | Bundled with Cloudflare Pro/Biz | One-click toggle |
| Dark Visitors (Known Agents) | Visibility into which AI bots visit you | Free tier | Team plans from a modest monthly fee | robots.txt + agent tracker |
| Anubis | Self-hosted proof-of-work wall | Open source, free | Free only | Reverse proxy (Go binary) |
| Nepenthes | Actively wasting scraper time | Open source, free | Free only | Tarpit container |
| Fail2Ban | Origin-side ban-by-log | Open source, free | Free only | Server daemon |
| ModSecurity + OWASP CRS | Rule-based WAF at origin | Open source, free | Free with commercial rule sets | nginx/Apache module |
| DataDome | Enterprise-grade bot management | Trial only | Enterprise contracts | Edge / API integration |
The tools
1. Cloudflare AI Crawl Control – Best one-click blocker for any site
Cloudflare AI Crawl Control is the fastest way to stop AI training bots. Any customer on any tier (including free) can go to Security → Bots in the dashboard and flip the “AI Crawls and Scrapers” toggle. It uses Cloudflare’s bot fingerprint database plus behavioral signals to identify GPTBot, ClaudeBot, Applebot Extended, PerplexityBot, and hundreds of others. From September 15, 2026, new domains default to blocking training and agent bots on pages that display ads.
Because it runs at the network edge, blocked requests never touch your origin. That saves bandwidth and origin CPU, and it doesn’t matter what stack your site is on. You can also allow specific bots (Googlebot stays green by default), charge for access via Cloudflare’s pay-per-crawl program, or write custom rules for finer control.
Where it falls short: Only useful if your site is behind Cloudflare. Cloudflare’s classifier occasionally mislabels legitimate research or archive crawlers, so check the analytics before you leave it on autopilot for something important.
Pricing:
- Free: Full toggle, block/allow list, analytics
- Paid: Bundled with Cloudflare Pro, Business, and Enterprise
Platforms: Any site fronted by Cloudflare (Windows, macOS, Linux origins all work)
Bottom line: If you’re on Cloudflare, flip this on before you read the rest of the list.
2. Dark Visitors (Known Agents) – Best for knowing which bots actually visit you
Dark Visitors (renamed Known Agents in 2026) is the visibility layer most sites are missing. It maintains a curated database of AI agents (training bots, RAG scrapers, browsing agents, LLM copilots) and gives you both a live-updating robots.txt file and an analytics dashboard that shows which ones are actually knocking. Add a snippet and you get real numbers on GPTBot, ClaudeBot, PerplexityBot, and dozens of newer agents you probably haven’t heard of.
The free tier covers the essentials: agent list, robots.txt file, basic visit tracking. Paid tiers unlock full analytics, multiple sites, and API access. It pairs well with Cloudflare: Cloudflare tells you what was blocked at the edge, Known Agents tells you what tried and (for anything Cloudflare doesn’t recognize yet) what still made it through.
Where it falls short: robots.txt is a request, not a rule. Well-behaved crawlers respect it; bad actors ignore it. This tool tells you the truth about who’s ignoring the rules, but it doesn’t force compliance on its own.
Pricing:
- Free: Robots.txt file, basic tracker, one site
- Paid: Team plans from a modest monthly fee for multi-site, full analytics, API
Platforms: Any site (snippet-based)
Bottom line: Install this even if you already have Cloudflare, because it tells you which bots are the growing problem.
3. Anubis – Best self-hosted wall against AI scrapers
Anubis is a self-hosted Web AI Firewall written in Go by TecharoHQ. It sits as a reverse proxy in front of your app and issues a JavaScript proof-of-work challenge before serving a request. Real browsers solve the challenge invisibly in a fraction of a second; naive scrapers stall out because they don’t run JS or don’t want to burn the CPU cycles at their scale.
Adoption tells the story: Codeberg, FFmpeg, the Linux kernel source, and most non-Cloudflare FOSS projects run it. Config is a single YAML file; you point Anubis at your upstream, set challenge difficulty per route, and add allowlists for good bots (Googlebot, IA_Archiver, etc.).
Where it falls short: JS-executing scrapers (headless Chromium, Playwright) can solve the PoW too; they just pay a modest CPU cost. Anubis is a strong deterrent, not an impenetrable wall. It also adds a small latency hop for every request.
Pricing:
- Free: Fully open source (MIT)
- Paid: None; commercial support available separately
Platforms: Linux (Docker, systemd), Windows, macOS (via Go binary)
Bottom line: The right pick when you run your own infra and don’t want to route through Cloudflare.
4. Nepenthes – Best for wasting scraper time
Nepenthes takes the opposite approach: rather than blocking scrapers, it drags them into an infinite maze of procedurally generated, plausible-looking pages that never end. The result is a tarpit that costs the scraper CPU and bandwidth while returning no useful training data. Made specifically as a response to AI crawlers ignoring robots.txt, it’s picked up by admins who want to actively raise the cost of scraping rather than passively decline it.
Deploy it as a Docker container in a subdomain or path, add a robots.txt directive pointing bad bots at it, and watch the crawler waste its resources. It also poisons the training data slightly: any scraper that ingests Nepenthes output brings noisy plausible-but-fake text back to whatever pipeline it feeds.
Where it falls short: Ethically borderline for some operators; you are intentionally serving misleading data. It only affects bots that don’t respect robots.txt in the first place, which is the point but also worth thinking about.
Pricing:
- Free: Open source
- Paid: None
Platforms: Linux (Docker) primarily; macOS and Windows via Docker
Bottom line: Add this once you already have blocking in place and want to make scraping actively expensive.
5. Fail2Ban – Best origin-side ban-by-log tool
Fail2Ban is the classic tool for reading server logs, spotting patterns, and pushing IP-level bans to the firewall. Every Linux admin knows it. For AI scraping specifically, a Fail2Ban filter can watch nginx or Apache logs for a User-Agent regex (GPTBot, ClaudeBot, Bytespider, etc.), or for behavioral signatures (bursts of requests-per-second from a single IP), and drop the offender into iptables.
Because the ban happens at the kernel firewall, the origin doesn’t spend cycles serving the scraper at all. Combine it with a Cloudflare setup at the edge and Anubis in the middle, and you have a defense in depth.
Where it falls short: IP-based bans are brittle against rotating residential proxies, which serious scrapers use. And filters have to be written and tuned; it’s not a one-click install.
Pricing:
- Free: Open source, in every mainstream Linux distro
- Paid: None
Platforms: Linux (native); Windows and macOS support is limited
Bottom line: The right choice for sysadmins who already write regex filters and want the ban at the packet level.
6. ModSecurity + OWASP CRS – Best rule-based WAF at origin
ModSecurity with the OWASP Core Rule Set is the open-source WAF that nginx and Apache admins deploy at the origin when they can’t or don’t want an edge service. The CRS ships with rules for common attacks (SQLi, XSS, RCE), and community-maintained AI-bot rulesets add User-Agent filters, behavioral thresholds, and known-bad IP feeds specifically for AI scrapers.
Because it’s rule-based, you keep full control. You can whitelist an internal analytics tool, block a specific user-agent globally, or write per-path rules. It also plays nicely with Fail2Ban: ModSecurity flags a request as anomalous, Fail2Ban reads the flag and bans the source.
Where it falls short: Rules require maintenance. False positives happen on custom apps. And it’s a native module for nginx or Apache, so containerized deployments need Docker images with it baked in.
Pricing:
- Free: Open source
- Paid: Commercial rule sets and support available from third parties
Platforms: Linux (nginx/Apache native), Windows (via IIS with limitations)
Bottom line: The default at-origin WAF when you’re not behind Cloudflare and want full transparency.
7. DataDome – Best enterprise-grade bot management
DataDome is where you land when a personal blog scales into a real business. Roughly 1,200 companies across America and Europe run its WAF, and it operates over 85,000 customer-specific machine learning models, meaning each protected site becomes its own detection challenge for a scraper. It integrates at the edge (via CDN or DNS), and its detection extends past AI training bots to credential stuffing, ad fraud, and scraping-as-a-service farms.
The dashboard is the strongest in this category, giving you per-request classification, geographic breakdown, and revenue-impact estimates. Response time on detection is measured in milliseconds. Enterprise-grade support is included.
Where it falls short: Pricing is not for individuals; you request a quote. Setup takes cooperation with your DNS, CDN, and origin, so it’s a project not a toggle.
Pricing:
- Free: 30-day trial
- Paid: Enterprise contracts, custom quotes
Platforms: Any origin (Windows, macOS, Linux); deployed at edge
Bottom line: The right pick when the cost of scraped content or fraud is bigger than the annual bot-management bill.
How to pick the right one
- If you want the simplest option and you’re already on Cloudflare: flip on AI Crawl Control. Done in 15 seconds.
- If you want to see the truth of who’s scraping you: install Dark Visitors (Known Agents) alongside anything else. Visibility first, blocking second.
- If you self-host and don’t want to be behind a CDN: run Anubis as your reverse proxy. Add Nepenthes to poison the ones that get through.
- If you’re a Linux admin who already lives in nginx logs: wire up Fail2Ban and ModSecurity + OWASP CRS. Free, transparent, in your control.
- If you’re a business with real revenue on the line: get a DataDome contract. Cloudflare covers the mass market; DataDome covers the edge cases.
- The realistic 2026 stack for a small-to-mid site: Cloudflare AI Crawl Control at the edge, Dark Visitors for visibility, Anubis for anything Cloudflare doesn’t catch. That combo runs for $0 in most months.
FAQ
What is the best free AI scraper blocker?
Cloudflare AI Crawl Control if your site is behind Cloudflare (the free tier includes it). If you self-host, Anubis is the strongest free open-source option. Both address the same problem from different sides of the stack.
Can AI bots bypass Cloudflare?
Yes, some can. Cloudflare’s classifier is very good at signature-based blocking but sophisticated scrapers use residential-proxy rotation and headless browsers to look human. That’s why layered defense (edge + origin + visibility) works better than any single tool.
Do I need to block AI bots on a personal blog?
That’s your call. If you want your content used to train commercial LLMs, do nothing. If you’d rather it wasn’t, flip the Cloudflare toggle or add Anubis. The most common middle-ground is Dark Visitors’ robots.txt plus a Cloudflare block for the largest training crawlers.
Will blocking AI bots hurt my search ranking?
Not if you configure the blocker correctly. All the tools on this list can allow Googlebot, Bingbot, and other traditional search crawlers while blocking training-only bots like GPTBot, ClaudeBot, and PerplexityBot. Check the allow list before deployment.
What’s the difference between Anubis and Cloudflare?
Cloudflare blocks at the network edge before requests reach your server. Anubis runs on your server as a reverse proxy and issues a proof-of-work challenge. Cloudflare is easier and covers more ground; Anubis is self-hosted, transparent, and requires no third-party.
