Comments on: Googlebot Creates Pages Instead of Simply Indexing Them: New FORM Crawling Algo Goes Bad

By: Ryan

Ryan — Thu, 03 Sep 2009 23:20:07 +0000

this is slightly off topic, but google now discovers search functions on sites and lets you use them from the search bar. Try browsing around in chrome and then typing in the domain name while on the site. If the site has a search feature Google has identified it will give you the option to search with that feature. Its like the open search without creating the xml file required. Kinda nifty.

By: Casdeiro

Casdeiro — Fri, 01 Aug 2008 09:47:39 +0000

As a second line of defense we might try to add a captcha to that forum form

By: Casdeiro

Casdeiro — Fri, 01 Aug 2008 09:46:22 +0000

Recently we discovered we were being overload by queries from Googlebot. I guess it’s something to do with this issue you posted about, because our phpBB2 forum’s tables are overloaded with a lot of nonsense searches. Strangely no one of this searches seems to be indexed at Google as you can see here: http://www.google.com/search?q.....url:search

Anyway, we’ll try to forbid index.php?name=PNphpBB2&file=search for any bot at ROBOTS.TXT. It should help, don’t you think?

Thanks for telling people about this issue. Overload of dynamic content site is a main collateral issue of this new Google’s behaviour.

By: If Googleplex Employees Don’t Understand The Webmaster Guidelines, How Can They Expect Webmasters To Adhere To Them? | Smackdown!

Tue, 27 May 2008 13:52:01 +0000

[...] to start indexing on it’s own, with very little control being given to webmasters (since now Googlebot crawls HTML search forms). What it is not referring to is a few odd pages that were cached in order to facilitate [...]

By: Does Google love your site too much?

Does Google love your site too much? — Tue, 27 May 2008 08:34:06 +0000

[...] quite easy to block these pages being indexed as other people have pointed out but 99.9% of webmasters won’t know about the issue. The best method to [...]

By: Michael VanDeMar

Michael VanDeMar — Sun, 25 May 2008 04:04:57 +0000

Gab, I was just quoting what Matt Cutts had said, but yes, I would take it to mean essentially the same thing as you did. “New links” would be links leading to new content, not new links pointing to content that they already knew existed.

By: Gab Goldenberg (goes to deep crawling page)

Gab Goldenberg (goes to deep crawling page) — Sun, 25 May 2008 03:22:53 +0000

Mike, I’m not sure if the terms matter and if we’re thinking the same, but in case we’re not, I thought the new crawling was aimed at discovering deep web content – like orphaned pages – on quality domains. Not at discovering links.

The difference (as I see it) is that if the purpose is to discover new links, then it can matter for links pointing outwards to other sites too. My understanding of the point of the crawling was the 80-20 rule being applied to search – 80% of the high quality stuff coming from 20% of the sites… Ergo if a site is high quality, let’s try and index ALL of it (but not necessarily who/what it links to). (More detail if you click my name; it goes to a page where I guessed what the rationale behind discovering these new pages might be.)

I mean, with circle-of-trust type algos, I guess discovering external links comes to the same thing as discovering more pages on the site you’re crawling. Depends how cynical you are about high quality sites linking out to their sponsors etc though…

All that said, your search tech knowledge is greater than mine, so I’d love to get your take!

p.s. The flip side of the canonical issues is at least those guys know they have trusted sites now!