Comments on: How To Block The Bots SEOmoz *Isn’t* Telling You About http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/ Smackdown! Sun, 14 Mar 2010 03:45:59 -0500 http://wordpress.org/?v=2.8.4 hourly 1 By: Why The Renewed Interest In The Linkscape Scams And Deception..? | Smackdown! http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-32818 Why The Renewed Interest In The Linkscape Scams And Deception..? | Smackdown! Fri, 22 Jan 2010 22:15:36 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-32818 [...] I covered this back when the launch actually happened, in this Linkscape post, resulting in quite a few comments, and there was more than a little heated conversation in the [...] [...] I covered this back when the launch actually happened, in this Linkscape post, resulting in quite a few comments, and there was more than a little heated conversation in the [...]

]]> By: Zaphod http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-24294 Zaphod Fri, 04 Sep 2009 07:24:36 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-24294 Well pardon me, I was just trying to help. Seems a GNU General Public License author can't get any respect anywhere. Please deleted this post, and the previous one I made. I surely don't want to be associated with the likes of you! Well pardon me, I was just trying to help. Seems a GNU General Public License author can’t get any respect anywhere.

Please deleted this post, and the previous one I made. I surely don’t want to be associated with the likes of you!

]]> By: Ryan http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-24269 Ryan Thu, 03 Sep 2009 22:02:29 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-24269 Sorry to resurrect an old thread, but has ANYONE figured out how to block linkscape outside of the ridiculous meta tag? Sorry to resurrect an old thread, but has ANYONE figured out how to block linkscape outside of the ridiculous meta tag?

]]> By: Zaphod http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-24231 Zaphod Thu, 03 Sep 2009 10:40:03 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-24231 Thanks for the pertinent info on these goofballs. I have also found that blocking them by user agent helps too. The updated signatures for {my spammy product} will be able to deal with these keyword scrapers even better. I did try to block them before by domain, but they weaseled around that, the give-away was their user agent showing up from other IPs. I just wanted to make sure (and this page helped me decide) that I wanted to deep-six their crawler from wherever it came from. I personally would rather toss them a 403 a few times with reasons, than give them a robots.txt . Most scrapers, and bot probes (laycat.com) just ignore robots.txt anyway. After they hit {my spammy product} 3 times, it switches them to a 503 permanently. Quite better at bandwidth saving. Thanks for the pertinent info on these goofballs. I have also found that blocking them by user agent helps too. The updated signatures for {my spammy product} will be able to deal with these keyword scrapers even better.

I did try to block them before by domain, but they weaseled around that, the give-away was their user agent showing up from other IPs. I just wanted to make sure (and this page helped me decide) that I wanted to deep-six their crawler from wherever it came from.

I personally would rather toss them a 403 a few times with reasons, than give them a robots.txt . Most scrapers, and bot probes (laycat.com) just ignore robots.txt anyway. After they hit {my spammy product} 3 times, it switches them to a 503 permanently. Quite better at bandwidth saving.

]]> By: Steve http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-10539 Steve Wed, 17 Dec 2008 23:14:08 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-10539 I have also been ordered to fight this on our European servers that currently host 32,000 websites and complaints have been filed. I think this will have seriosu repurcussions. I have also been ordered to fight this on our European servers that currently host 32,000 websites and complaints have been filed. I think this will have seriosu repurcussions.

]]> By: Lance http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-10505 Lance Wed, 22 Oct 2008 22:37:39 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-10505 Interesting enough, the project's code name was Carhole, and the registered name for dotnetdotcom.org is Jeff Albertson. Jeff Albertson just happens to be comic book guys name............ Interesting enough, the project’s code name was Carhole, and the registered name for dotnetdotcom.org is Jeff Albertson. Jeff Albertson just happens to be comic book guys name…………

]]> By: NotHappy http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-10480 NotHappy Tue, 21 Oct 2008 18:08:32 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-10480 Doug (Or anyone else) feel free to write a post with this info, but as Michael points out it's not a single bullet proof solution because they can buy more IP's and it doesn't stop the "other sources". But it will block the /28 IP block that has been assigned to Dotbot. The /28 is a block of 16 IP addresses, in this case from: 208.115.111.240 to 208.115.111.255 So if you firewall those 2 IP's and the 14 in between you will be blocking the Dotbot IP range. The Dotbot site itself is hosted in the http://208.115.111.242 IP. In the past i have found the best way to deal with situations like this is to target the data sources rather than trying to fight the offender. Our company is writing letters to all data sources on the LinkScrape page besides the big 3 (G, Y and MSN) notifying them we have completely blocked their crawlers from 23 webservers encompassing thousands of websites due to the data harvesting practices of SEOMoz.org who are utilizing and selling their data services. I doubt these companies will be impressed when the value of their services are lowered, user experience reduced etc because of some company in Seattle flogging $79 a month subscriptions with the data. Especially when they realize they have been blocked from sub 5,000 Alexa sites due to SEOMoz. Doug (Or anyone else) feel free to write a post with this info, but as Michael points out it’s not a single bullet proof solution because they can buy more IP’s and it doesn’t stop the “other sources”. But it will block the /28 IP block that has been assigned to Dotbot.

The /28 is a block of 16 IP addresses, in this case from:

208.115.111.240

to

208.115.111.255

So if you firewall those 2 IP’s and the 14 in between you will be blocking the Dotbot IP range. The Dotbot site itself is hosted in the http://208.115.111.242 IP.

In the past i have found the best way to deal with situations like this is to target the data sources rather than trying to fight the offender.

Our company is writing letters to all data sources on the LinkScrape page besides the big 3 (G, Y and MSN) notifying them we have completely blocked their crawlers from 23 webservers encompassing thousands of websites due to the data harvesting practices of SEOMoz.org who are utilizing and selling their data services.

I doubt these companies will be impressed when the value of their services are lowered, user experience reduced etc because of some company in Seattle flogging $79 a month subscriptions with the data. Especially when they realize they have been blocked from sub 5,000 Alexa sites due to SEOMoz.

]]> By: Michael VanDeMar http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-10479 Michael VanDeMar Tue, 21 Oct 2008 16:10:30 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-10479 Well, blocking by IP is ok as long as they don't decide to switch or add new IP's. The thing is, that's one bot, and I still firmly believe that it only accounts for a small amount of the collection process they are claiming. It's the only way that things add up. Also, since that's like closing the barn door after SEOmoz kicked it in and raped the horses, here's a possible method for dealing with what they took already: http://smackdown.blogsblogsblogs.com/2008/10/21/how-to-remove-your-website-from-linkscape-without-an-seomoz-meta-tag/ Well, blocking by IP is ok as long as they don’t decide to switch or add new IP’s. The thing is, that’s one bot, and I still firmly believe that it only accounts for a small amount of the collection process they are claiming. It’s the only way that things add up.

Also, since that’s like closing the barn door after SEOmoz kicked it in and raped the horses, here’s a possible method for dealing with what they took already:

http://smackdown.blogsblogsblo.....-meta-tag/

]]> By: Doug Heil http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-10478 Doug Heil Tue, 21 Oct 2008 16:02:59 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-10478 @Nothappy; I would appreciate it if you could please write up a blog post about how "exactly" website owners and webmasters can block this bot. If not you; someone you give the info to would suffice. I think it would be very beneficial to the industry. This bot is like any o'l rogue bot not wanted on our servers. It needs to be dealt with by a good majority. What I mean is, make things very clear so as a new webmaster/owner clearly sees what to do step by step. @Nothappy; I would appreciate it if you could please write up a blog post about how “exactly” website owners and webmasters can block this bot. If not you; someone you give the info to would suffice. I think it would be very beneficial to the industry. This bot is like any o’l rogue bot not wanted on our servers. It needs to be dealt with by a good majority. What I mean is, make things very clear so as a new webmaster/owner clearly sees what to do step by step.

]]> By: NotHappy http://smackdown.blogsblogsblogs.com/2008/10/17/how-to-block-the-bots-seomoz-isnt-telling-you-about/comment-page-2/#comment-10477 NotHappy Tue, 21 Oct 2008 15:35:52 +0000 http://smackdown.blogsblogsblogs.com/?p=132#comment-10477 Just as i thought Rand, the old "Can't say competitive intelligence" line which puts SEOMoz at the bottom of the barrel with all the other scrapers. Actually they are better, they are harvesting it for things like spam which i can identify and do something about. So looks like your Dotbot (which may be a red herring anyway) has been issued a /28 so: # iptables -A INPUT -s 208.115.111.240/28 -j DROP That will block 208.115.111.240 - 208.115.111.255 at the firewall (The Dotbot site is using 208.115.111.242 if you visit in the browser) BTW Rand, how did you receive a /28 IP allocation? Running a scraper network is not valid ARIN justification. Might have to send off an email. Anyhow if everyone issues this command: # iptables -A INPUT -s 208.115.111.240/28 -j DROP Alternatively you can block that IP range with .htaccess if you don't have root access, and i will dig in to the logs and see what this LinkScraper is doing. Sorry to say, all respect i had for SEOMoz is now gone and i will not be visiting it again. Just as i thought Rand, the old “Can’t say competitive intelligence” line which puts SEOMoz at the bottom of the barrel with all the other scrapers. Actually they are better, they are harvesting it for things like spam which i can identify and do something about.

So looks like your Dotbot (which may be a red herring anyway) has been issued a /28 so:

# iptables -A INPUT -s 208.115.111.240/28 -j DROP

That will block 208.115.111.240 – 208.115.111.255 at the firewall (The Dotbot site is using 208.115.111.242 if you visit in the browser)

BTW Rand, how did you receive a /28 IP allocation? Running a scraper network is not valid ARIN justification. Might have to send off an email.

Anyhow if everyone issues this command:

# iptables -A INPUT -s 208.115.111.240/28 -j DROP

Alternatively you can block that IP range with .htaccess if you don’t have root access, and i will dig in to the logs and see what this LinkScraper is doing.

Sorry to say, all respect i had for SEOMoz is now gone and i will not be visiting it again.

]]>