Small update on Google problems

May 9th, 2006

For those who don’t have time to trawl through the comments at Matt Cutt’s blog, he has posted a tidbit of information about the current problems Google is having with pages disappearing from their index and pages going ’supplemental’. This was in response to a lot of webmasters (ourselves included) submitting sample sites experiencing problems.

“I feel comfortable saying that participation in Sitemaps is not causing this at all. One factor I saw was that several sites had a spam penalty and should consider doing a reinclusion request (I might do it through the webmaster console) but even that wasn’t a majority. There were a smattering of other reasons (one site appears to have changed its link structure to use more JavaScript), but I didn’t notice any definitive cause so far.

There will be cases where Bigdaddy has different crawl priorities, so that could partly account for things. But I was in a meeting on Wednesday with crawl/index folks, and I mentioned people giving us feedback about this. I pointed them to a file with domains that people had mentioned, and pointed them to the gmail account so that they could read the feedback in more detail.

So my (shorter) answer would be that if you’re in a potentially spammy area, you might consider doing a reinclusion request–that won’t hurt. In the mean time, I am asking someone to go through all the emails and check domains out. That person might be able to reply to all emails or just a sampling, but they are doing some replies, not only reading the feedback.”

Not really anything new here, and it seems a little worrying that Google don’t seem to have a clue what’s causing the problem (or that it’s as widespead as it is). At least it has been confirmed that it shouldn’t be an issue with Google sitemaps - so there’s one conspiracy theory we can discount now!

We’ll keep you updated with anything we hear or find.