“We Improved Our Response Time, but the Crawl Rate Stayed the Same. What’s Your Advice?” Google Responds

Your Google crawl rate drops by 90%. You improve your response time, but nothing changes. What next?
SIA Team
November 17, 2021

This news item can be thought of as the ongoing situation that a participant, named Hazel, is facing.  Her question was part of the title of a piece I wrote last month: “My Company’s Site Had a 90% Drop In Crawl Requests. What Happened?” Google Responds

For your convenience, I’ve queued the video and placed it below:

Hazel’s Situation Continues

At the 20-minute, 8-second mark of the English Google SEO Office-Hours From November 12, 2021, Hazel asked about her site’s ongoing situation.

The video below is queued for you:

So, to give you a summary:

-As stated in the previous video, her site experienced a 90% drop in crawl rate

Google’s John Mueller suggested that the drop in crawl rate may be due to a server issue on Hazel’s side (which would contribute to a decrease in the server’s response time).

And now, Hazel’s team improved their response time (at one point stating that it improved from 1,400 to 500 milliseconds). 

As she said:

“…And now,  we have done many actions to improve our response time, and now the response time in the back end of the [Google Search Console] GSC has been restored to the former level, around 1,400 milliseconds to 500 milliseconds. Yeah. 

“But the crawl request, also the crawl rate, is restoring much more slowly than the response time. I remember that you said if we fix our response time quickly, then the crawl rate will be restored in several days, right? So yeah.

“So right now, the crawl rate didn’t restart like what we expected. So we’re wondering, ‘Why is that?’ And maybe is that because our response time isn’t restored to maybe lower level or something like that? 

So we kind of hope that you can give us some advice on that.

John’s response was:

“My guess is probably it just takes longer. So usually, the system that we have is very responsive in slowing down to make sure that we don’t cause problems, but is a little bit slower in ramping back up again. 

“So I suspect if it’s been a few days, that you improved that, then probably, I would give it maybe a week or longer to catch up again. What you can also do, in the Help Center in Search Console, we have a link to a form where you can request that someone from the Googlebot team take a look at the crawling of your website.

“And you can give them information about this. Especially if it’s a larger website where we have a lot of URLs to crawl, you can tell them, like, we improved our crawl rate significantly.”

“And you can give them information about this. Especially if it’s a larger website where we have a lot of URLs to crawl, you can tell them, like, ‘we improved our crawl rate significantly.’”

A Waiting Game

From listening to them discuss further, it seemed that Hazel’s team already submitted such a request to the Googlebot team, but hadn’t heard a response. John simply said that they probably review these requests from time to time, and advised more waiting time.

Sometimes, SEO–specifically, in this case, crawling–can be a waiting game. It’s unfortunate that Hazel’s site is not getting the Googlebot love that it deserves, but hopefully, that will change soon.

Source: Google Search Central YouTube channel