Comments on: How To Block The Bots SEOmoz *Isn’t* Telling You About http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/ Smackdown! Fri, 10 Feb 2012 01:20:31 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Michael VanDeMar http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-53775 Michael VanDeMar Mon, 27 Dec 2010 17:27:20 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-53775 Regardless of what your misconceptions about Rand's ethical practices are, Yatin, that is a completely different bot from Linkscape, and the rogerBot you are discussing has in fact wound up in Project Honeypot's database. Regardless of what your misconceptions about Rand’s ethical practices are, Yatin, that is a completely different bot from Linkscape, and the rogerBot you are discussing has in fact wound up in Project Honeypot’s database.

]]> By: Yatin http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-53774 Yatin Mon, 27 Dec 2010 16:09:03 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-53774 but Michael, as per seomoz blog rogerbot is the name of their crawler and although i haven't checked my server logs to see visit by rogerbot, i do NOT think : a) seomoz would have two different crawlers b) Rand would risk brand equity by ignoring robots.txt like spam bots do ;) but Michael, as per seomoz blog rogerbot is the name of their crawler and although i haven’t checked my server logs to see visit by rogerbot, i do NOT think :

a) seomoz would have two different crawlers

b) Rand would risk brand equity by ignoring robots.txt like spam bots do ;)

]]> By: Michael VanDeMar http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-53770 Michael VanDeMar Sun, 26 Dec 2010 14:58:04 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-53770 Actually, Yatin, that is a different tool that they launched last May. Also, based on that tool showing up in Project Honeypot, it looks like that tool isn't obeying robots.txt either, regardless of what they say. The Linkscape tool still doesn't identify itself as such to this day. Actually, Yatin, that is a different tool that they launched last May. Also, based on that tool showing up in Project Honeypot, it looks like that tool isn’t obeying robots.txt either, regardless of what they say. The Linkscape tool still doesn’t identify itself as such to this day.

]]> By: Yatin http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-53768 Yatin Sun, 26 Dec 2010 06:49:02 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-53768 @Ryan - you can block the linkscape UA via robots.txt User-agent: rogerbot Disallow: / ( Source: http://www.seomoz.org/dp/rogerbot ) personally, i love the seomoz tools so I wouldn't be blocking it. @Ryan – you can block the linkscape UA via robots.txt

User-agent: rogerbot
Disallow: /

( Source: http://www.seomoz.org/dp/rogerbot )

personally, i love the seomoz tools so I wouldn’t be blocking it.

]]> By: Why The Renewed Interest In The Linkscape Scams And Deception..? | Smackdown! http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-32818 Why The Renewed Interest In The Linkscape Scams And Deception..? | Smackdown! Fri, 22 Jan 2010 22:15:36 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-32818 [...] I covered this back when the launch actually happened, in this Linkscape post, resulting in quite a few comments, and there was more than a little heated conversation in the [...]

[...] I covered this back when the launch actually happened, in this Linkscape post, resulting in quite a few comments, and there was more than a little heated conversation in the [...]

]]> By: Zaphod http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-24294 Zaphod Fri, 04 Sep 2009 07:24:36 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-24294 Well pardon me, I was just trying to help. Seems a GNU General Public License author can't get any respect anywhere. Please deleted this post, and the previous one I made. I surely don't want to be associated with the likes of you! Well pardon me, I was just trying to help. Seems a GNU General Public License author can’t get any respect anywhere.

Please deleted this post, and the previous one I made. I surely don’t want to be associated with the likes of you!

]]> By: Ryan http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-24269 Ryan Thu, 03 Sep 2009 22:02:29 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-24269 Sorry to resurrect an old thread, but has ANYONE figured out how to block linkscape outside of the ridiculous meta tag? Sorry to resurrect an old thread, but has ANYONE figured out how to block linkscape outside of the ridiculous meta tag?

]]> By: Zaphod http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-24231 Zaphod Thu, 03 Sep 2009 10:40:03 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-24231 Thanks for the pertinent info on these goofballs. I have also found that blocking them by user agent helps too. The updated signatures for {my spammy product} will be able to deal with these keyword scrapers even better. I did try to block them before by domain, but they weaseled around that, the give-away was their user agent showing up from other IPs. I just wanted to make sure (and this page helped me decide) that I wanted to deep-six their crawler from wherever it came from. I personally would rather toss them a 403 a few times with reasons, than give them a robots.txt . Most scrapers, and bot probes (laycat.com) just ignore robots.txt anyway. After they hit {my spammy product} 3 times, it switches them to a 503 permanently. Quite better at bandwidth saving. Thanks for the pertinent info on these goofballs. I have also found that blocking them by user agent helps too. The updated signatures for {my spammy product} will be able to deal with these keyword scrapers even better.

I did try to block them before by domain, but they weaseled around that, the give-away was their user agent showing up from other IPs. I just wanted to make sure (and this page helped me decide) that I wanted to deep-six their crawler from wherever it came from.

I personally would rather toss them a 403 a few times with reasons, than give them a robots.txt . Most scrapers, and bot probes (laycat.com) just ignore robots.txt anyway. After they hit {my spammy product} 3 times, it switches them to a 503 permanently. Quite better at bandwidth saving.

]]> By: Steve http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-10539 Steve Wed, 17 Dec 2008 23:14:08 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-10539 I have also been ordered to fight this on our European servers that currently host 32,000 websites and complaints have been filed. I think this will have seriosu repurcussions. I have also been ordered to fight this on our European servers that currently host 32,000 websites and complaints have been filed. I think this will have seriosu repurcussions.

]]> By: Lance http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-10505 Lance Wed, 22 Oct 2008 22:37:39 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-10505 Interesting enough, the project's code name was Carhole, and the registered name for dotnetdotcom.org is Jeff Albertson. Jeff Albertson just happens to be comic book guys name............ Interesting enough, the project’s code name was Carhole, and the registered name for dotnetdotcom.org is Jeff Albertson. Jeff Albertson just happens to be comic book guys name…………

]]>