Firewall: Bots and Crawlers (error 429)

To provide the fastest and best experience for all our clients, we have a specific way of working with bots. Bots, such as those from well-known SEO tools like Ahrefs and Semrush, are important to you as a client, but their way of working can slow down your website considerably.

Therefore, we have determined that all these bots are allowed to crawl your site, but at a rate that does not cause them to slow down your website. A proper crawl rate is essential for your website to function optimally without being overloaded by bots.

We recommend a crawl delay of 30 seconds. This means 2 requests per minute, 120 requests per hour and 2880 requests per 24 hours.

Bot returns message "429 too many requests"

We measure how often bots like Ahrefs and Semrush visit your website and if this is too often in a short period of time then we send the http status code "429 too many requests" back to the bot. We recommend not letting bots visit your website more than 2 times per minute. How to change these settings varies quite a bit from bot to bot, but below we have put down links to the guide for some commonly used SEO bots.

If you need some help, you can always reach our support desk by email and on weekdays from 10:00 am to 4:00 pm you can also reach us by phone and via the chat in My Cloud86.