Here’s the uncomfortable thing about robots.txt: it has never stopped anyone. It’s a text file at the root of your site that says, politely, “please don’t crawl these paths.” Well-behaved search engines honour it. A lot of AI crawlers read it the way I read the terms and conditions, which is to say, they scroll straight past and click accept.
That’s not a bug. It was never designed to enforce anything. It’s a sign on the lawn, not a lock on the door. And for twenty-odd years that was a fine arrangement, because the bots reading your site were mostly search engines that wanted you to be indexed, so they played nice.
Then the scrapers arrived with an appetite for training data and no incentive to behave. Now you’ve got requests hammering your origins, burning backend compute, and occasionally wandering into paths you’d rather weren’t fed into someone’s model. The sign on the lawn does nothing. You need a bouncer.
Microsoft just put one in public preview. It’s an AI bot ruleset for Azure Web Application Firewall on Azure Front Door Premium, and the whole point is that it moves the decision from “please” to “no.”
What a WAF actually is, in one analogy
Before the new bit, the old bit. Azure Front Door is Microsoft’s global front door, a reverse proxy that sits in front of your application, terminates connections at the edge, and forwards the ones it likes to your origins (your App Service, your VMs, your storage, whatever lives behind it). Think of it as the lobby of a building with a lot of back offices.
A Web Application Firewall (WAF) is the security desk in that lobby. Every request that comes through the front door gets looked at before it’s allowed deeper into the building. The WAF runs rules, “block anything that looks like SQL injection,” “rate-limit this IP”, and each rule has an action: let it through, log it quietly, challenge it, or turn it away at the desk.
The key word there is before. The WAF makes its call at the edge, in the lobby, before the request ever reaches your application. That’s the difference that matters for AI bots. Your robots.txt is read by the bot after it’s already knocked on your origin’s door. The WAF decides whether the knock even gets forwarded.
What the new ruleset adds
Azure’s WAF already ships managed rulesets, bundles of rules Microsoft maintains so you don’t have to hand-write them. There’s a Default Ruleset for the OWASP-style attacks, and a Bot Manager ruleset that sorts automated traffic into buckets: verified good bots, known bad bots, and the great murky “unknown” middle.
The new preview extends that bot-detection machinery specifically to AI crawlers and agents, the likes of OpenAI, Anthropic, Google, and other AI-related user agents. Instead of one blunt “bot / not bot” verdict, you get the ability to identify AI crawlers as a category and decide, per your own policy, what happens to them.
And “what happens” is a real menu, not a toggle:
- Allow, let the bot through. Maybe you want a particular AI agent indexing your public docs.
- Block, turn it away at the desk. The request never touches your origin.
- Log, let it through but write down that it came. This is the dry-run setting, and it’s where you should start.
- Challenge, make it prove it’s not a cheap script before continuing.
The elegant part is that this is per rule, inside a policy you can scope by path. Allow Google’s crawler everywhere, block OpenAI’s, let Anthropic’s agent hit /docs but not /app. That granularity is exactly what a flat text file can never give you, because the text file isn’t making the decision, the bot is.
One honest caveat, because it’s a preview and I’d rather you hear it from me than from your logs at 2am: bot identification is pattern-based and it is evolving. Do not assume it catches every AI agent on earth on day one. A determined scraper spoofing a browser user agent from a residential IP is a harder problem than a well-identified crawler announcing itself. This closes the “polite bots that announce themselves and then ignore your wishes” gap, which is a genuinely large gap. It is not a magic AI-proof wall.
The blast radius, stated plainly
A WAF policy in Azure attaches to Front Door endpoints and domains. Whatever that policy covers, every origin behind that Front Door inherits the verdict. Turn the ruleset to Block on a policy that fronts six applications, and all six are now refusing those bots. That’s the feature. It’s also the footgun: the blast radius is “everything behind this door,” and if one of your origins quietly depends on an AI agent you forgot about, a partner integration, your own retrieval pipeline, you’ll find out the hard way.
Which is the entire reason you turn it on in Detection mode first.
Turning it on without turning anything off
A WAF policy has two modes. Prevention enforces, rules with a Block action actually block. Detection enforces nothing; it evaluates every rule and writes what would have happened to the logs. Same bouncer, same guest list, but for now he just takes notes instead of ejecting anyone.
Here’s the thing worth internalising: Detection mode is your dry run. You don’t need a clever flag to preview this change, you need to run the whole ruleset in a mode where it decides nothing and records everything, then read the record. That’s the real safety mechanism, and it’s built into the product.
In the portal the path is roughly: Front Door WAF policy → Policy settings → set Mode to Detection, then Managed rules → Add → the AI bot ruleset (it sits under the bot manager family; check the exact name in the preview, since Microsoft routinely renames things between preview and GA).
You can do the same from Azure CLI by setting the policy mode to Detection and attaching the managed ruleset. I’m deliberately not handing you a copy-paste command with invented flags here, the Front Door WAF commands and their arguments shift between CLI versions, and some of what you’ll see pasted around (including a --what-if on this command) isn’t a thing on every build. Check az network front-door waf-policy update --help on the version you actually have before you run anything, and confirm the flag names against the current CLI reference. The discipline that matters isn’t a magic dry-run switch; it’s that you set Detection first, verify the mode took with a read-only show, and only move to Prevention once the logs are boring.
Whatever path you take, make sure diagnostic logging is sending the WAF log to a Log Analytics workspace, if it isn’t, Detection mode is a bouncer taking notes on paper you never read. In Front Door’s diagnostic settings that’s the WAF log category; enable it and point it at a workspace you can query.
Read the notes before you fire the bouncer
Leave it in Detection for a real week, not an afternoon. Crawlers come in waves; a Tuesday tells you nothing about the weekend batch job. Then go read what the ruleset would have done. Conceptually, you’re after four things per request the WAF flagged, the rule that matched, the action it would have taken, where the request came from, and which path it was reaching for:
Treat the column names above as a sketch, not gospel, run a bare take 10 against the table first and read the real schema off your own rows. What you’re looking for is the same regardless of field names: which AI agents are actually hitting you, how often, and, this is the part people skip, which of them you’d regret blocking. Sort by volume, then eyeball the paths. If something you rely on shows up as a would-be block, you carve it an Allow exception before you ever flip to Prevention. This is the whole discipline. Detection mode exists so your first real block isn’t a surprise outage.
When the logs stop surprising you, switch the policy to Prevention the same way, verify the mode change with a read-only show afterwards, and the notes become ejections.
The part worth sitting with
Run this exercise for a week against a site that has had a perfectly polite robots.txt the whole time, and here’s what you should expect to see. The file says “don’t.” The logs, in all likelihood, show a steady stream of AI user agents that read the don’t-crawl paths anyway. Not necessarily anything malicious, just the gap between advisory and enforced, rendered as rows in a table. That gap is the established reality the whole preview exists to address; Detection mode simply lets you measure your own version of it before you act on it.
That’s the mental model to walk away with. robots.txt is you asking nicely. A WAF rule is you deciding. For public search indexing, asking nicely was always enough, because everyone wanted the same thing. For AI crawlers, the incentives don’t line up, and “please” stopped being a control a while ago.
It’s preview. It’s Premium-only. It won’t catch a spoofed scraper hiding behind a browser string. But for the enormous category of bots that identify themselves and then ignore your wishes, it finally moves the decision to your side of the door, where you can see it, log it, and mean it.