We all know that, once you’ve done your on-page optimization, one of the next steps to getting ranked is getting indexed.
But, what if you have web pages that were indexed in the past, and then, you take a look today, and find that they’re currently not indexed, even though they were before, and even though you know you didn’t do anything to warrant a de-indexing by Google.
You didn’t build any backlinks.
You weren’t hit with a negative SEO campaign.
You were always above-board.
Well, recently, someone described their situation to Google’s John Mueller. During the latest English Google SEO Office-Hours (the one from November 19, 2021), at around the 37-minute, 33-second mark,
Is this situation similar to yours, in the sense that you’re experiencing that pages drop out of the index?
“I'm managing a website with over 5000 URLs. We do our best to comply with search ranking and SEO factors, but we find, on average, that we get around 100 urls per month that get de-indexed. What can we apply to our pages to prevent this from happening?”
John Mueller’s response was:
“So it's kind of hard to say with just that information, because for the most part, we don't just remove things from our index, we kind of pick up new things as well.
“So if you're adding new content at the same time, and some things get dropped on along the way from our index, usually that's kind of normal and expected, because essentially, for pretty much no website [do] we index everything on the website.”
So, it’s kind of normal for that to happen.
He went on:
“It's something where, on average, what I’ve seen is somewhere between 30% and maybe 60% of a website tends to get indexed, so if you're adding hundreds of pages a month and some of those pages get dropped or some of the older pages or less relevant pages get dropped over time, that seems kind of expected.
“To minimize that, I think it's something where, essentially, you need to show to Google or to your users what the value of your website is overall, so that Google says, ‘Well, I need to make sure I keep as much as possible from this website in my index,’
“And that's usually less of a technical thing and more of something where the value that you provide is such that it's critical for Google to keep this in our index, and even in cases like that, when we think a website is fantastic, we still won't index everything.
“It's kind of just purely from a technical point of view. It's almost impossible, partially because a website tends to have a lot of...I call it technically-duplicated content where you have things like tracking parameters or maybe pagination that goes into infinite spaces or filtering parameters or search form or things like that where theoretically, we can find an infinite number of URLs on the website, and if it's possible to find an infinite number of URLs and we can only index a finite number, then we'll never be able to get everything indexed.”
So, if you find that your web pages are not indexed in Google, try some of the things listed above, particularly providing value.