“My Site Uses JavaScript and Doesn’t Block Google. Why Aren’t Pages Indexed?” Googler Responds.

Have you ever been in this potentially frustrating situation? (The video below should be queued to the right spot.)

During the most recent English Google SEO office-hours, right around the 21-minute mark, a question (situation) was asked to Google’s John Mueller:

“So basically, we have a huge problem on one of our e-commerce websites. We look at listing pages [containing] the javascript that’s responsible for loading all the listing products. Googlebot cannot access it, cannot render it, apparently according to [the] Mobile-Friendly Test Tool and Search Console so we get like zero products [listed in search]…”

Not Blocked in robots.txt, Server Logs Look Fine

But, why do the Mobile-Friendly Test Tools and Search Console say that they’re blocked? 

She goes on: “…robots.txt doesn’t block it, we check server logs [and] they say that Googlebot isn’t blocked and [that] it can access the resources. Other parts of the page are absolutely fine, other JavaScript on the website’s fine.”

Compounding Things Further

Hmm…okay. 

She went on to say that her team has another site, that’s absolutely identical, that’s in a different country, but doing fine. 

And to compound this further, she said that this issue had resolved itself for a month (even though nothing was done to it), at which point they saw that more pages were indexed. 

But now, the issue has presented itself again.

Responses and Proposed Solutions

So, now it was John Mueller’s turn to provide a response. 

One of the first things John brought up was that there can be a difference between testing tools and the real indexing process. The difference has to do with time limits, or ‘deadlines.’ 

He went on to explain that, with testing tools, because Google wants to give you a result quickly, Google’s tools may indicate that things are fine. However, during the real-life indexing process, if Googlebot comes to your page, and for some reason, the JavaScript is not rendered correctly. 

John said that “…probably what happens the most there is that it’s more of a general rendering speed issue in the sense that maybe you have something set up that the JavaScript or the API responses are created in the way that they almost have session IDs associated with them, so that we can’t cache them properly or anything where essentially rendering the page takes a significant amount of resources…”

One thing John mentioned was that “…usually, also when this happens, it can be a sign that things are kind of on the edge in the sense that sometimes it works and sometimes it doesn’t work…”

Earlier, I said that the problem seemed to have resolved itself, only to recur again. Maybe that’s why John said that things are possibly ‘kind of on the edge.’

John also encouraged the joining of one of Martin Splitt’s sessions, which are usually JavaScript-focused SEO office-hours. 

(Granted, from my search, it doesn’t seem that they occur as frequently as the regular office-hours, but if you’re facing an issue like this and happen to be able to get onto one, it might be time well spent.)

Another thing John said was, “What I’ve done in the past when looking in[to] these kinds of issues is to look at webpagetest.org or in Chrome at the waterfall kind of diagram there to see [if this is] a page that has a significant amount of JavaScript and other content that needs to be loaded.”

He went on to elaborate that, “…in the cases I’ve looked at, it is more something where they’re almost like hundreds or 200 300 400 500 requests that are required to render a page.”

With more and more required resources, the more hit-or-miss the indexing process becomes. 

So, if you’re having similar difficulties with JavaScript and ranking, you may want to consider these proposed solutions. You can also do a search on YouTube to try to find possible ideas that may help you with this. 

Source: Google Search Central YouTube Channel