Are Drops In Crawl Rate A Cause For Concern?

Can moving hosts cause a drop in crawl rate? Are drops in crawl rate a cause for concern? Google provides more insights on crawl rates, server response time, and hosting changes.
Marie Aquino
February 5, 2022

In the Google SEO Office Hours February 4 episode, a user has been encountering issues with drops in crawl rate for his site when they moved to a CDN.

After the move, they have noticed that their response time have gone up from 300 millisecond to a second. From a crawl rate of 2 Million requests a day, it has dropped down to just 80,000 daily request. Did the move to the CDN impact the crawl rate?

John Mueller provided his insights on the situation. According to Mueller, from a ranking point of view, it should not change anything.

When changing hosts, what will happen on Google’s side is that the crawl rate will move into a more conservative area first and will crawl a bit less because of the bigger change in hosting – it could be a move to a CDN, a move from a CDN, or changing from one CDN to another CDN.

Over time, it could be a couple of weeks, maybe a month or so, the crawl rate will increase again to see where it would settle down. A drop in crawl rate when moving to a CDN or a change of a CDN can be normal. It does not necessarily mean that there is a problem.

If they were crawling two million pages of the site before, it’s unlikely that there would be two million changes every day. It is not necessarily that they would miss new content on the site, they would just try to prioritize again and figure out which of the pages actually need a recrawl on a daily basis. Drops in crawl rate is not necessarily a cause for concern.

An issue that Mueller find concerning is the change in average response time. This is because crawl rate is based on the average response time and server errors. If the average response time goes up significantly, the crawl rate will lower down.

From Google’s point of view, the response time is critical for the amount of crawls that is done per day because they try to limit the number of active connections that they have on the server.

If the response time is high, then they will run into the limit fairly quickly, whereas if they can quickly crawl, then there would not be a problem with the number of concurrent connections.

Valuable information on drops in crawl rate when moving hosts, how drops in crawl rate are not a cause for concern, and how server response times have an effect on crawl rate, too.

If you’ve been monitoring your crawl rate and have seen drops, hopefully these insights would be able to provide you answers and some areas to check into to be able to determine if there is an issue on your server.

Generally, it is not a cause for concern and should not affect rank.

Check out the Google SEO Office Hours episode here: