Why are Google Bots are Getting Banned?

Google Bots and other web crawlers crawl sites to index their content. The bots will attempt to scan the entire site. If the site has 404s, the bots don't know it's not available content and will attempt to scan it. If they scan enough 404s, it's possible for them to get banned depending on your 404 Detection lockout criteria. If you find this is happening you'll want to remove the IPs from the Banned Hosts field and disable 404 Detection until you're able to resolve the site 404s.

Powered by Zendesk