Answer engines and your robots.txt
Paste your robots.txt. You get a verdict per crawler, the exact rule that decided it, and whether that crawler matched a group of its own or fell back to the wildcard. The precedence follows RFC 9309, including the part people get backwards: the longest matching rule wins, and a tie goes to allow.
A crawler with no group of its own is not blocked by default. The specification is explicit:
"If no matching group exists, crawlers MUST obey the group with a user-agent line with the '*' value, if present."
So if your wildcard group says Allow: /, every AI crawler you never named is already allowed. Adding an explicit group for it changes nothing. That cuts both ways: a site that thinks it is blocking AI training by not mentioning GPTBot is doing the opposite.
Within the group that applies, the longest matching rule wins, not the first one and not the last. The specification puts it in terms of octets: "The most specific match found MUST be used. The most specific match is the match that has the most octets." And when an allow and a disallow are equivalent, "the 'allow' rule SHOULD be used".
That is why Disallow: /a/ followed by Allow: /a/b/ leaves /a/b/c reachable while /a/z stays blocked. Order in the file is irrelevant.
Two more details worth knowing: user-agent matching is case-insensitive, and if several groups name the same crawler their rules are combined rather than the first one winning.
OpenAI documents four tokens with distinct roles: GPTBot crawls content for training foundation models, OAI-SearchBot is what surfaces sites in ChatGPT search, OAI-AdsBot validates pages submitted as ads, and ChatGPT-User handles user-initiated actions. Blocking GPTBot while allowing OAI-SearchBot is a coherent position: no training, but still citable.
Anthropic documents three: ClaudeBot collects content that could contribute to training, Claude-User fetches pages when someone asks Claude a question, and Claude-SearchBot works on search result quality.
The remaining tokens are checked as strings, without this page claiming a purpose for each. If you need to know exactly what one does, its operator's own documentation is the only source worth trusting.
robots.txt is a request, not an access control. It is honoured by the crawlers that choose to honour it, and a rule here stops nothing that decides to ignore it. This page also parses the file you paste; it does not fetch your site, so it cannot tell you whether that file is the one actually served.