In short
- Some companies use automated programs, called bots, to visit websites. Examples are SEO tools like Ahrefs and Semrush, and AI services like ChatGPT and Claude.
- Bots can visit many pages very quickly, which can make your website slow.
- That’s why these bots may visit up to 10 pages per minute. If a bot goes faster, it has to wait 5 minutes.
- Your visitors notice nothing. Google and Bing are not limited either, so this does not affect how your website shows up in search results.
Why do we do this?
A normal visitor opens a page, reads it and then clicks on to the next one. A bot can open hundreds of pages in a minute. That takes a lot of work from the server, and your website gets slower for your real visitors. By slowing bots down, we keep your website fast.
Bots can still visit your website. They just have to take it a bit slower.
What exactly is the limit?
- Each bot may visit up to 10 pages of your website per minute.
- Every bot has its own limit. If one bot is busy, others are not affected.
- Images, styling and other supporting files don’t count. Only pages count.
- If a bot goes over the limit, it has to wait 5 minutes. After that it can continue.
Using ChatGPT, Claude or another AI assistant?
When you ask an AI assistant about your website, it visits your pages for you. The same limit applies.
- Ask about specific pages. “Check my whole website” makes the assistant visit many pages at once. “Look at example.com/contact” works better.
- The limit is shared. If many people ask ChatGPT about your website at the same time, all those visits count together.
- You can’t tell the assistant to go slower. That instruction doesn’t change how fast it visits pages.
- Does the assistant say it can’t open your website? It has probably hit the limit. Try again later and ask about fewer pages. In some cases this can take up to 30 minutes. Solving a check in your own browser won’t help, because the assistant visits your website from its own servers. Still not working after 30 minutes? Contact our support team.
Using an SEO tool on your own computer?
Tools like Screaming Frog run on your own computer. If they go too fast and keep going, our security system may flag your internet connection. For 30 minutes, you then have to complete a check before you can open your website.
Always set your tool to a slower speed before you start. We recommend 8 pages per minute at most. You’ll find instructions for popular tools below.
For technical users
What the bot sees
A bot that goes over the limit receives HTTP status '429 Too Many Requests' for 5 minutes, with a request to retry after 300 seconds.
If one IP address (the internet address of a computer) keeps sending requests after receiving the 429, 10 times within one hour, our security software Imunify360 puts that IP address on a greylist for 30 minutes. Visitors from a greylisted IP address have to complete a check, such as a captcha, before they can open the website.
Applebot, GPTBot and AhrefsBot are on Imunify’s whitelist and only receive the 429. AI assistants that fetch pages for a user, such as ChatGPT-User and Claude-User, can be greylisted.
Which bots are limited?
Bots on our crawler list, such as Meta, GPTBot, Applebot, AhrefsBot, ChatGPT-User and Claude-User. Googlebot and Bingbot are fully excluded. Files like images, CSS, JavaScript, fonts, sitemaps, robots.txt and ai.txt never count and are never blocked.
Slowing bots down with robots.txt
You can ask bots to slow down in your robots.txt file. A delay of 8 seconds stays under the limit:
User-agent: AhrefsBot Crawl-delay: 8
Not every bot follows this rule. For SEO tools, change the speed in the tool’s own settings.
Speed settings in popular SEO tools
- Ahrefs: 4xx page in sitemap error in Site Audit.
- ContentKing: Monitoring speed and frequency.
- Semrush: How to optimize your Site Audit crawl speed.
- Screaming Frog: Speed configuration. Set it to 1 thread and a maximum of 0.125 URLs per second.
Need help?
Do you keep getting a check when you open your website, or does your SEO tool keep showing errors? Contact our support team.