If Googleplex Employees Don’t Understand The Webmaster Guidelines, How Can They Expect Webmasters To Adhere To Them?

Last month, in one of my posts, I had decided to include a few images to see how well they might rank for their keyphrases. I had never targeted any of the image searches, and due to one of the topics of the post it seemed like a good opportunity to do so. When I went to check later to see if the images happened to be indexed yet, they weren’t. It had only been a couple of days, and I really didn’t expect them to be there yet, so that was really no surprise. What did surprise me, however, was that as it turns out none of my images were indexed. In fact, nothing whatsoever from that subdirectory was currently in Google’s index, Image search or otherwise.

I do know that at one point the content that directory contained, which is where I put all of my support files for posts (I use FTP instead of WordPress upload functionality), was in fact indexed… which just made the fact that it was now missing an ever bigger mystery. I checked all of the usual culprits (broken links, robots.txt, .htaccess, etc.), but everything checked clean. Donna Fontenot suggested that perhaps it was something wonky with the way I was handling my subdomains, and it occurred to me that perhaps my old evil host, DreamHost, might have done something uncool to cause it to become deindexed. Since neither was something that lent itself to easy diagnoses, I decided to pop over to the Google Webmaster Help discussion on Google Groups and ask there if anyone saw something I might be missing.

After a few posts, Googler John Mueller chimed in and gave me a few tips for getting images indexed in Google Images, pointed me to some other posts on the topic, and suggested that I opt-in to “enhanced image search” in Google Webmaster Tools. I was pretty sure that I had done most of what he suggested, but went back through the images to make sure, tweaked a couple, read the blog posts he suggested, and added the “enhanced image search” (which allows people to label, or tag, your images in Google Images, from what I understand) to the Smackdown profile in Google Webmaster Tools. After about a month, still no love from Google as far as indexing the images (or anything else in that directory, for that matter). In fact, from what I can tell, Googlebot has not even looked in that directory this month. So, I posted an update.

John replied with the following:

So I took your post as an excuse to take a better look at your site 🙂 and found a few things which I’d like to share with you. In particular regarding your /images/ subdirectory I noticed that there are some things which could be somewhat problematic. These are just two examples:

– You appear have copies of other people’s sites, eg /images/viewgcache-getafreelinkfromwired.htm
– You appear to have copies of search results in an indexable way, eg /images/viewgcache-bortlebotts.htm

I’m not sure why you would have content like that hosted on your site in an indexable way, perhaps it was just accidentally placed there or meant to be blocked from indexing. I trust you wouldn’t do that on purpose, right? – John Mueller

When you do a search on Google, next to the results in most cases you will see a “Cached” link. When you click on it, you see a snapshot of the page as it was in the past, when Google crawled it. Since webpages are subject to change, this cached copy can help to explain why a certain page is showing up in the search results, even if the content doesn’t match what you searched for when you visit the page on that day. Google explains the cached contents as such:

Google takes a snapshot of each page examined as it crawls the web and caches these as a back-up in case the original page is unavailable.

The content in question on Smackdown was most definitely placed there on purpose. They are not “copies of other people’s sites”. The pages in question are cached copies of webpages or search results that relate back to the discussion on hand in each and every case. They are relevant support files to back up and demonstrate what I am talking about at the time I make my post, because I know the content I am talking about won’t be the same later.

Some of my posts discuss Google’s search results, and as such a few of them contain cached copies of searches. John hints in that post that I am in violation of Google Webmaster Guidelines by doing this:

At any rate, I’m sure you’re aware that this kind of content is not something that we would like to include in our index and which is also mentioned in our Webmaster Guidelines. – John Mueller

I went through the guidelines carefully, but for the life of me could not figure out what John was referring to. That is, until this morning, when I went though them again, and realized that the only thing he could be talking about was this like:

• Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.

Ok, fine. Here’s the problem… that guideline has nothing whatsoever to do with the cached pages I have on my site. This is a relatively new guideline, put in place of March last year. Matt Cutts explained the change on his blog when it happened. It relates to auto-generated mass produced low quality pages that some sites would use to boost their content, taking the shotgun approach to getting something ranked. In fact, the type of content that guideline suggests blocking is exactly the type of content that Google has decided to start indexing on it’s own, with very little control being given to webmasters (since now Googlebot crawls HTML search forms). What it is not referring to is a few odd pages that were cached in order to facilitate discussion.

I’m pretty sure that John is wrong on the reason why that entire subdirectory has gotten deindexed. It doesn’t make sense from the standpoint of the guidelines, and also would not explain why the other files within that same directory weren’t getting indexed either. Additionally, on some datacenters there are now entire posts, ones that are still linked to from the front page of my blog and have a few other decent links to them, that have been deindexed. It could of course be completely unrelated, but something does appear to be going on somewhere. I mean, I am more than happy to submit a reconsideration request, to have someone who can “look behind the curtain” have a peek, but why on Earth should I have to block relevant content from Google? The thing is, if a Google employee can misread a reference in the guidelines and think that may be the cause of the problem, what chance do normal everyday webmasters have in getting it right?

Quick PS: In case anyone at Google reads this, I’d thought I’d let you know… the example link for the Cached feature on this page points to a datacenter that is no longer functional, namely 216.239.53.104. Just in case someone wanted to update it to a working link. 😀