Jason sent me an email to correct me on what he was talking about. We replied to each other back and forth a couple times, and a few very interesting things were revealed in that conversation:

On Mon, 2010-01-25 at 14:46 -0800, Jason Calacanis wrote:
sorry, didn’t mean ranking for “video game walkthroughs” literally….
more like VIDEO GAME NAME walkthrough:
http://www.google.com/search?&q=call+of+duty+walkthrough

On Mon, Jan 25, 2010 at 3:38 PM, Michael VanDeMar wrote:
Yeah, realized that and pointed out that I might have misinterpreted in the next comment. I only saw one or two that you were in the top 5 though, not a slew of them. But yeah, better than the generic version.

Thing is that you still have tons of auto or near-auto generated content out there ranked, stuff that is nothing more than noise, including stuff you are boasting about ranking for:

http://www.mahalo.com/need-for-speed-prostreet-walkthrough

I commented about this last year I believe… this is not a legitimate or ethical way to go about what you are doing:

1) Auto-generate tons of pages with no content, with AdSense embedded on them, based solely on search phrases
2) See what winds up ranking
3) Go in and put valid content in afterwards, pretend that all along the content was not only valid, but actually better that what you outranked.

You want a more palatable model? How about this… there are tons of sites out there that are full of great content that no one will ever see, because they’re simply not seo’d (ie. unknown and unlinked). Why don’t you:

1) Find a way to identify the search results that are already full of crap sites
2) Identify quality sites that should be there instead, and
3) Use your pull and popularity to get *those* sites ranked.

If you find a way to do just that and somehow build a successful business model around it then that would be much, much better than what Mahalo is now.

-Michael

On Mon, 2010-01-25 at 15:43 -0800, Jason Calacanis wrote:
the part you’re leaving out is:

a) we used to noindex these and we are going to again
b) if any page gets any traffic we pay someone to build it out–so it’s only short for a couple of days.

if a page is started by a user and is short it a) won’t rank 99% of the time and b) if it happens to we see it in analytics and build it out.

not really a big deal IMO

best j

Let’s take a look a little closer to some of the things that Jason is saying here:

“we used to noindex these and we are going to again”

At one point Mahalo had the noindex tag on the pages that were fully automated, and Jason is freely admitting that they should have the nodindex tag now. These pages have no added value and should not be indexed by Google. Jason must believe this, or he would not say that they were planning to add the noindex them again. If he thought they deserved to be in the index then there is no reason to tell the search engine spiders not to include them. Now, according to Jason’s rebuttal to Aaron, the removal of the nodindex tag was an “accident” that happened in the migration to Mahalo 3.0. For those who do not know, Mahalo 3.0 was released back in November of last year. Mahalo is a template driven site. All it takes to “fix” the tag that was lost “by accident” is a coder opening up the template header and including 1 line of conditional code. In the 3 months prior to our conversation, and in the 3 weeks since, no one has bothered to do so. To this day those pages are still getting added to the index at a very large rate.

How large, you might ask? I have been checking here and there for the past few weeks, and when I looked on any given day Google was reporting from 2,000 to 20,000 new pages on Mahalo.com:

I clicked through quite a few of these pages to see what the content looked like. Almost every singe page I looked at was completely automated content. In fact, when I looked today, the only one I found that wasn’t totally automated was the Mahalo entry on Hootsuite… which happened to have 2 additional manually added sentences in it, content which “enhanced” the scraped content that surrounded it: “HootSuite is the professional Twitter client. With HootSuite, you can manage multiple Twitter profiles, pre-schedule tweets, and measure your success.” The entire “Human Powered” portion of that page is tinier than just one AdSense block located directly under it:

And that AdSense block is one of 4 located on the page. Yeah, let’s hear it for “added value” folks.

As a side note, it looks like Jason is violating Google AdSense TOS by placing more than 3 AdSense blocks on each page. Not entirely sure why they are letting him do that, since supposedly that lowers the overall value each advertiser gets… but I digress.

“if any page gets any traffic we pay someone to build it out”

Here Jason is clearly admitting that he thinks it’s fine to rank your scraped content first, and then add quality if and only if it gets traffic (and remember, 1-2 sentences of “quality” is fine). Essentially he’s saying, go ahead and spam, but if it looks like it might get enough traffic someone will actually notice, add content.

I wrote back:

On Mon, Jan 25, 2010 at 4:51 PM, Michael VanDeMar wrote:
So, you’re not denying that is indeed your model here (rank first, quality later), you’re just saying that you think that’s it’s fine to do it that way, since you pay people to build it out within a couple of days after you start to get traffic?

Your assertion that a page on your devoid of content won’t rank is completely untrue, by the way. Generally people with quality content wind up ranking because other people link to them. That’s what the whole basis of Google’s base algorithm is built upon. Your site, however, gets pages ranked solely by virtue of other internal pages linking to them. You have so much link juice, due mostly to controversy and press, ranking power that you spread around your site from one page to another, that you can rank relatively competitive phrases with no effort at all. That does not somehow make the page quality due to some sort of mystical link juice feedback. Quality attracts link juice. Link juice does not impart quality.

Take a look at these pages, all which rank with pretty much nothing but links from within Mahalo itself:

http://www.google.com/search?q=valentine%27s+day+cupcakes
#1: http://www.mahalo.com/valentines-day-cupcakes
1 external link, from a scraper: http://search.yahoo.com/search?p=link%3Ahttp%3A%2F%2Fwww.mahalo.com%2Fvalentines-day-cupcakes+-site%3Amahalo.com

http://www.google.com/search?hl=en&safe=off&num=10&q=halo+3+walkthrough&btnG=Search
#5: http://www.mahalo.com/halo-3-walkthrough
Again, only linked externally from another scraper: http://search.yahoo.com/search?p=link%3Ahttp%3A%2F%2Fwww.mahalo.com%2Fhalo-3-walkthrough+-site%3Amahalo.com

http://www.google.com/search?q=how+to+invest+online
#8: http://www.mahalo.com/how-to-invest-online
Zero links from external sites, even from scrapers: http://search.yahoo.com/search?p=link%3Ahttp%3A%2F%2Fwww.mahalo.com%2Fhow-to-invest-online+-site%3Amahalo.com

Google describes it’s ranking algorithm as something that leverages the democratic nature of the web. No one, however, is voting for your pages. Why should they rank? Simply because you paid someone to flesh them out?

-Michael

PS. Do you mind if I blog this? Or are any of your answers being said with an expectation of privacy?

On Mon, 2010-01-25 at 17:00 -0800, Jason Calacanis wrote:
Actually, most of our pages come in VERY LARGE now because we don’t allow folks to just make them on the live site any more. They have to go through Mahalo Tasks now. in the old days it was a free for all like Wikipedia/Squidoo…. we didn’t like the results.

Very few of our pages start short. So, it’s not really a strategy to put up short pages and wait… our strategy is to get a TON of people to make pages in Mahalo Tasks.

Think about it: which is a better way to build a business… make a bunch of stubs or make a bunch of high-quality pages? The later is better and it’s really not very expensive. Especially now that we have revenue sharing with our page managers.

The three pages you’re talking about all have a LARGE amount of original content, and I would say they are B+ to A+ content, no? I think google and yahoo take into account a large amount of content plus the amount of time a user sits on a page. At least i’ve been told that they look at time spent on page. as you can imagine… the time spent on a recipe, walkthrough or howto article is VERY LONG.

thanks for the feedback!

best j

Again, let’s look at what he’s saying here:

“we don’t allow folks to just make [pages] on the live site any more. They have to go through Mahalo Tasks now.”

That’s right, Jason would like us to believe that the Mahalo pages, like the one on 13 year olds and rape, all now pass an editorial review before going live. Even if that were true (which it isn’t, btw) I don’t think that I would be boasting that, based on the pages that I looked at.

in the old days it was a free for all like Wikipedia/Squidoo…. we didn’t like the results.

Now Jason Calacanis is saying that the 100% user contributed, zero ads Wikipedia is nothing more than a “free for all”, and that his 99% auto-generated content content laden with AdSense is therefore better. Gotcha.

“Very few of our pages start short.”

Either Jason thinks he’s talking about a different site, or he has absolutely no concept of what the phrase “very few” actually means. Currently when I look, Google tells me that Mahalo has 356,000 pages indexed:

As I mentioned earlier, in contrast to “very few” of these pages being “short”, the vast majority of them are nothing more than scraped content. From what I can tell it looks like less than 4% of the Mahalo.com pages getting indexed on a daily basis actually involve humans adding unique content in to them. The rest of the pages are being auto-generated by a Mahalo bot aptly named “searchclick”. If you go to any of these non-content pages and click on the “View Page History” link in the bottom of the right column you can see exactly who it is that generates the majority of this “Human Powered Search” website:

“searchclick” is of course not an actual user. It’s the name of the robot that Mahalo uses to generate all of these pages on it’s site. According to the Mahalo website,

Searchclick is not really a user at all. The name “searchclick” is used to represent a individual “search click” by a Mahalo user who has searched a particular term and Mahalo made a “created by searchclick” meaning the page had been searched for the first time.

These pages on Mahalo that are getting indexed by the thousands are nothing more than searches that users performed that no one thought valuable enough to create a topic page for in the first place. The content used to populate these pages is nothing more than scraped versions of other websites search pages, such as Youtube, Flickr, and Google itself.

Google is very clear on it’s viewpoint of indexing search pages. Matt Cutts, head of Google’s Webspam team, wrote about the subject back in 2007:

But just to close the loop on the original question on that thread and clarify that Google reserves the right to reduce the impact of search results and proxied copies of web sites on users, Vanessa also had someone add a line to the quality guidelines page. The new webmaster guideline that you’ll see on that page says “Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.” – Matt Cutts

There really is not a whole lot of room for ambiguity in what Matt is saying.

As it turns out, Mahalo can not block these pages using robots.txt. They go out of their way to make these search generated pages blend in with every other page on their site. Not only are they all located in the same root directory, there is nothing at all that allows the fact that they are auto-generated to be detected in any sort of machine readable way. Is this by accident? From the way Jason talks, you would think so. However…

The mere creation of pages is not enough to get them discovered and indexed in the search engines. The search engine spiders must have some means of locating these pages in order to know that they exist. Most sites simply rely on links from other pages on their sites, or links from other people’s sites, for their content to get noticed. There are alternate means though. For instance, Google allows webmasters to set up special sitemaps, meant for search engine spiders alone, in order to insure that sets of pages that the webmaster considers important do in fact get found. These spider-only sitemaps are in XML format, and are not meant for human visitors to use. Does Mahalo use XML sitemaps?

Yep, they sure do. In fact, they use a dynamically generated sitemap, one that is generated on the fly with a url parameter, “p=”, used to distinguish which page of the sitemap you want to view. If we actually open up Mahalo’s sitemap, what do we see?

That’s right… Mahalo’s XML sitemap is packed with tons of spammy auto-generated pages for Google find. Not only is Mahalo not blocking these pages from getting indexed, they are actually going out of their way to make sure Googlebot finds them.

While Jason’s assertion that these pages don’t directly generate a ton of traffic for him may in fact be true, they do play a very important role in his overall ranking scheme. With nothing but the power of the domain they reside on many of these very little competition, zero traffic phrases will actually result in rankings. On occasion you will find spammers scraping these rankings from Google, which then result in a very low pagerank link back to Mahalo.com. Individually each of these links adds almost no value. However, as I mentioned earlier, Mahalo has hundreds of thousands of these pages indexed. In conjunction with link juice from 1 or 2 links from his own blog often times Jason can now get some moderately competitive phrases ranked with no effort, or votes from the rest of the world, at all. For instance, currently at the top of many pages on Mahalo you will see a link to the page for Robert Kissel, who happens to be in the news currently. Search on Google for him and you can see that Mahalo page is currently in the top 10. If we look at the links pointed to this page we see that aside from interior pages on Mahalo.com, exactly 4 sites link to it, 3 of which are from scraping Google’s results:

Even if we are to believe that the Jason himself believes the naive assertion that the sheer amount of content or time a visitor spends on a page are major ranking factors, we all know that those alone will not get you ranked, even if your content truly is quality. As to Jason’s statement that the pages on Mahalo are “quality” and deserve to rank? In most of the cases that I saw, and I am talking about the “human powered” ones now, they simply aren’t. Take for instance Mahalo’s top 10 ranking for [need for speed prostreet walkthrough], if you look at the page there isn’t even a walkthrough on it:

Aside from that tiny little blurb and links scraped from the search engines, the only thing on that page is an embeded video of a walkthrough that was made by someone other than a Mahalo user… and that video doesn’t even exist anymore. If Mahalo used content that they generated and hosted themselves that would never be an issue.

At this point in the game of course, simply noindexing the pages in and of itself is not really a solution. Unless Mahalo moves the search auto-generated content into it’s own subdirectory so it can be blocked by robots.txt, noindexes and nofollows the existing pages (to prevent grabbing unwarranted juice from serps scrapers), and removes them from their sitemap, then they are in clear violation of Google’s Webmaster Guidelines. Why Google won’t actually take action on them is anyone’s guess.

Now, I covered this back when the launch actually happened, in this Linkscape post, resulting in quite a few comments, and there was more than a little heated conversation in the Sphinn thread as well. This prompted some people, both on Sebastian’s post and in the Sphinn thread on it, to ask why all of the renewed interest?

It is not extreme, its just that it isn’t new. The fact that they bought the index (partially)? That was known from the beginning. The fact that they don’t provide a satisfying way of blocking their bots (or the fact that they didn’t want to reveal their bots user agent)? Check. The fact that they make hyped statements to push Linkscape? Check. {…} I don’t get the renewed excitement. – Branko, aka SEO Scientist

Well, I guess you could say that it’s my fault. Or, you could blame it on SEOmoz themselves, or their employees, depending on how you look at it. You see, the story goes like this…

Back when SEOmoz first launched Linkscape, it would have been damn near impossible for a shop their size to have performed the feats they were claiming, all on their own. Rand was making the claim “Yes – We spidered all 30 billion pages”. He also claimed to have done it within “several weeks”. Now, even if we stretch “several” to mean something that it normally would not, say, 6 (since a 6 week update period is now what they are claiming for the tool), we’re still talking a huge amount of resources to accomplish that task. A conservative estimate of the average website, considering only html, is 25KB of text:

30,000,000,000 websites x (25 x 1024) bytes per website = 768,000,000,000,000 bytes of data (768 trillion bytes, which is 698.4TB)

(698.4TB / 45 days of crawling) x 30 days in a month = 465.6TB bandwidth per month

Now, I know that one of the reasons that Rand can get away with some of his claims is that most people just don’t grasp the sheer size of those numbers. In todays age, bandwidth is cheap, with many hosts even boasting of unmetered, or unlimited, bandwidth on their accounts, and computers are fast. But in reality the reason they can make those claims is that in all likelihood no one on a shared server or a cluster will ever hit their bandwidth limit, because their processor usage will cause them to go over their limits way before actual data transfer becomes an issue. On dedicated servers, where the resources are not shared, hosts actually care about how much bandwidth you use. For instance, last August The Planet (one of the best hosts I know of for dedicated servers) upgraded their plans to offer 10TB/month at no additional cost. Prior to that they only included 1TB with their plans. On most hosts the charges for people who go over their bandwidth allotment are usually rather steep.

This means that basically for what Rand was claiming to be 100% true, they pretty much would have needed to own their own datacenter. Now, these days, of course, there is another option. Five months ago a new company, named 80legs, came out of beta. With 80legs pretty much anyone can build their own spiders, run them on 80leg’s servers, and spider 2 billion pages per day. They can do this of course because they rent the service out to many people, it’s not just one company powering one link tool. However, 15 months ago when moz launched their tool, 80legs wasn’t an option.

So, I called them on their claims, and a bit of controversy followed from it. Moz refused to clearly identify how they were actually gathering the data, and would not release information on how to keep whatever spiders were being used off of their sites. They did release a list of fairly widespread bots, and suggested that if you wanted to keep SEOmoz from scraping your sites via robots.txt, well, then, you’re just going to have to block Google, Yahoo, and MSN as well. They also came up with their lame assed version of a “solution” to people’s concerns, and stated that people could also add an SEOmoz meta tag to their pages to keep them from being indexed (which would not, however, keep them from being crawled in the first place). Despite the fact that many webmasters made it clear that this was unacceptable, to date nothing about that situation has changed. They still do not offer a clear concise way to allow webmasters to instruct SEOmoz to not spider their site, or give people an option to keep information about their site from showing in the Linkscape data.

The thread on Sphinn went where it did, and the next day one of the admin’s decided to close the discussion, even though it was far from being resolved. No more comments were allowed. Period. End of story.

I moved on.

Fast forward 15 months. I get an email from SEOmoz, touting their new tool, which is apparently powered from the Linkscape index. So, I trot on over and take a look. There, on the front page, are their same outrageous claims… only more so. The graphic stats that in the past 45 days, they have crawled 700 Billion Links, 55 Billion URLs, and 63 Million Root Domains:

Now, I and others, when this first happened, put Rand to task for trying to interchange “crawling” with “indexing”. Therefore, when he states in that graphic that they “crawled” 700 billion links in 45 days it’s not because he’s too stupid to know the difference. The SEOmoz employees know very well that they while they may have “found” an huge amount of links in their index, they did not crawl them. This is actually aside from whether or not it was them who did the actual crawling. Of course, they do try and set toss in some confusion there, just in case someone calls them on their bullshit again, by stating that they crawled 55 billion urls at the same time, as if there is some sort of relevant distinction between a url and a link… which, for crawling purposes, there isn’t. The only real way there would be a difference is if they were trying to say that 645 billion of the links they found were mailto: or javascript: links, but even if that were the case, you wouldn’t “crawl” those anyways.

So, upon seeing this I of course get irked all over again. I went back and revisited the unresolved Sphinn thread that had gotten locked, just to refresh my memory of how the conversation went. I got to the end of the conversation, and I saw something that struck me as just a teensy bit odd:

Wtf? Apparently Scott Willoughby (note: please see update below), an employee of SEOmoz, contacted an admin or mod on Sphinn a little over 5 months ago, 9 months after the conversation ended, and had them unlock the thread, all so he could post this way out of left field comment calling me a liar, and then had them lock it again. I mean, seriously. Why the hell would someone do that? A little over 5 months ago… hm… what happened 5 months ago… wait! Wasn’t that when 80legs.com went live? I wonder…

So, off I went to look at the list of “sources” that SEOmoz had listed on Linkscape. Lo and behold, there it was:

So it seems that what happened is that in the summer of 2009 SEOmoz learned that there was a new service about to go live, one that had it existed way back when Linkscape launched would have provided an alibi to moz’s claims, one that would at least put them in the realm of feasibility. Therefore they went through the effort of having the thread re-opened, just so that someone could post one more claim that yes, they actually did crawl their own data. Of course, this still doesn’t explain a damn thing about what user agent they were (or are, for that matter) using, or how to keep those bots from hitting your site. Apparently someone in the organization felt strongly enough that it is possible to have future technology retroactively bolster bullshit claims that they actually went down the path of trying to cover their tracks that way.

I sent some messages to Sebastian about all this, since I knew he’d get a kick out of them yet again trying to confuse people about spidering vs. crawling, and that prompted him to blog about the whole thing again.

On a side note, I do want to address a recurring theme that keeps coming up in the comments throughout this whole issue. Some people are asking, if the tool is useful, who cares if they lie to promote it? Without getting into the whole argument over whether or not link intelligence is worth $800/year when the majority of it is available for free, there are both ethical as well as legal ramifications about what SEOmoz is doing. One of the biggest selling points for this is that this data is presented with SEOmoz’s own metric, something that they have dubbed as mozRank (mR). This metric is exclusive to SEOmoz, and only holds value if it’s not more made up bullshit. If they do indeed get exposed for selling snake oil, then anything sold under the pretext of “we’re experts… trust us!’ becomes worthless.

Additionally, they are still gathering this data without full disclosure on how to keep their alleged bots off of our servers, and therefore doing so without our permission. According to the Revised Code of Washington 9A.52.110 (SEOmoz is headquartered in WA), Computer trespass in the first degree:

(1) A person is guilty of computer trespass in the first degree if the person, without authorization, intentionally gains access to a computer system or electronic database of another; and

  (a) The access is made with the intent to commit another crime; or

  (b) The violation involves a computer or database maintained by a government agency.

(2) Computer trespass in the first degree is a class C felony.

So, it’s only a crime to deliberately scrape people’s content if your are doing so in conjunction with committing a crime. According to RCW 9.04.050 False, misleading, deceptive advertising:

It shall be unlawful for any person to publish, disseminate or display, or cause directly or indirectly, to be published, disseminated or displayed in any manner or by any means, including solicitation or dissemination by mail, telephone, electronic communication, or door-to-door contacts, any false, deceptive or misleading advertising, with knowledge of the facts which render the advertising false, deceptive or misleading, for any business, trade or commercial purpose or for the purpose of inducing, or which is likely to induce, directly or indirectly, the public to purchase, consume, lease, dispose of, utilize or sell any property or service, or to enter into any obligation or transaction relating thereto: PROVIDED, That nothing in this section shall apply to any radio or television broadcasting station which broadcasts, or to any publisher, printer or distributor of any newspaper, magazine, billboard or other advertising medium who publishes, prints or distributes, such advertising in good faith without knowledge of its false, deceptive or misleading character.

While many of us may take it in stride that we will get lied to when people try and sell us things, trust me, it still does not make it acceptable, and there is law that backs that up.

Update: Apparently there was a glitch in Sphinn when they migrated to new software. The comment that I accused Scott Willoughby of making 9 months after the conversation had been closed (which would have required the involvement of a Sphinn employee) was in fact a Desphinn that he made at the time the post was first submitted. This glitched caused that and 1,530 other Desphinns to all incorrectly get imported as comments… and all with the exact same timestamp, ie. 7/14/2009. Whoops.

Thank you to Michelle Robbins, Third Door Media’s Director of Technology, for discovering how that actually happened. It does prove that not all conspiracy theories are true. I do, however, stand by the rest of the post.

Lisa Barone wrote a piece talking about personal brands and false idols on the web. In it she wrote the following paragraph:

Don’t support personal brands built on smoke and mirrors. Make people work for the brands they’re trying to create. Don’t let them scoble their way in. Don’t accept that someone is important just because they act like they are or someone told you they were.

Apparently Robert is the ultra sensitive type, and didn’t take too kindly to her choice of wordage. Here is his reply:

Wow, Bob. Way to identify with the lowly masses out there.

Those who know me know that I always prefer to do manual upgrades, wiping everything out and starting over completely fresh each time, whether I have been hacked or not. This way if there was an intrusion it should still clean the hack out completely, even if I don’t know it’s there. As it happens, when I upgraded to 2.6.5 from 2.6.2 I did not do this. I merely upgraded the 2 files involved in the security portion of the WP 2.6.5 upgrade (which were wp-includes/feed.php and wp-includes/version.php). However, to date those are still the only two files from that version with a security risk according to WordPress, and I upgraded them well before I was hacked.

I noticed something was wrong earlier this week, after I wrote the post on how to easily find free photo downloads for your blog posts. I was checking to see if the post had any rankings a couple of days later, when I noticed that it wasn’t showing in Google. I don’t mean that it wasn’t ranking, either… I mean it wasn’t showing at all. I checked, and sure enough Google had definitely cached the post shortly after I had published it. It was showing when I did a site: command too. I realized that somethings was weird, however, when I saw that the description for my homepage still showed my post from November as being the snippet in the serps:

I checked that as well, and just as I thought they had already re-cached the homepage too, which means that showing the old snippet made no sense:

Neither the post itself nor the homepage were showing in the serps at all, even for exact phrases unique to those pages, phrases that were showing in Google’s cache, and therefore should have been searchable. My first thought was that I had been penalized for some reason (the conspiracy theorist in me even considered it might be the “PageRank for Sale” alt text on an image from Novembers post ), so I sent off a couple of Tweets asking people if they saw anything I might have missed.

Luckily, one of the people listening who was kind enough to respond was John Mueller. After telling me that I need to upgrade my tinfoil hat to Mu-metal one (heh, thanks John! ), he found the issue fairly quickly:

The link he gave me pointed to the text-only version of Google’s cache of my homepage, and when I scrolled down, sure enough there it was:

I upgraded last night (complete wipe and reinstall this time), but I’m a little concerned still. Since there has still been no word from WordPress about 2.6.5 being vulnerable, that may mean that it is something that they are completely unaware of, and therefore was carried over into WordPress 2.7. I did some research on other hacked blogs, and while I did find one other 2.6.5 blog and one 2.7, comparing their caches against other older cached pages on the same site it looks like both of those were hacked prior to them upgrading. If anyone else find more information about blogs that have gotten hacked after upgrading to WP 2.6.5 or above, please let me know.

Our purpose is rather simple. We want to make the internet as open as possible. Currently only a select few corporations have a complete and useful index of the web. Our goal is to change that fact by crawling the web and releasing as much information about its structure and content as possible. We plan on doing this in a manner that will cover our costs (selling our index) and releasing it for free for the benefit of all webmasters.

If, again, DotBot is owned by SEOmoz, then actual goal of collecting those webpages was the development of a commercial tool. With that in mind, Rand’s refusal to remove pages from the index that the owners do not want in there takes on a whole new level of unreasonableness. When pressed about it, this is the most Rand is willing to compromise as far as removing sites from the index:

3)SEOmoz will ONLY remove your site from DISPLAYING your data through Linkscape if you add a customized SEOmoz meta tag to each and every page on your site, and even then, only after a 30-60 day time period.

Yes, although we are looking at ways to block an entire site from being shown in the future through a registration system. And yes, we can’t block anything until we’ve re-crawled and re-indexed that page, which can take 30-60 days depending on the speed with which we crawl/re-crawl a given URL.

4)SEOmoz is “unwilling to provide a clear concise way to keep data out of Linkscape.”

That’s what you said, and I merely copied it to point out that it had an exception. I know it’s a fun soundbyte, but without the important caveat in the sentence it was in, it’s really unfair to keep using this phrase. That caveat is that we are willing to provide one clear, concise way to keep data out of Linkscape – the seomoz noindex meta tag.

So, the only way Rand will voluntarily remove your site from his index is if you agree to basically brand your website with a meta tag using his company name, and then wait 30-60 days. Unfortunately for him, that’s really not his call.

You own your website and the data it contains (assuming you did not scrape it from somewhere else, of course), and that ownership is protected under US copyright law. Anyone whose rights are violated under that law have specific remedies available to them under the Digital Millennium Copyright Act.

Now, I cannot stress this strongly enough… these remedies are not intended to harass a website owner. They should be used neither frivolously nor fraudulently, and there are penalties for filing false information. You should under no circumstances perform this process for any urls or domains that you do not explicitly own, and if a counter-notification does get filed then you should in fact follow through with a lawsuit.

For all valid claims, I am outlining an easy to follow process for requesting that your information be removed from his index.

First, verify that your content is indeed in their tool. If it is, then the next step is to contact SEOmoz directly. Give them a chance to rectify the situation within a timely manner. Send a polite request that your entire domain be completely removed from the index powering their Linkscape tool, and for a way to confirm that it has indeed been done once they have. The support email for SEOmoz is listed on the site as sitesupport@seomoz.org, or you can fax them the request at (206) 338-3797. In this request you should list who you are, the address of your domain, and your contact information. Despite Rand’s insistence that they cannot do this, it might turn out that they do in fact have the ability after all. Do not skip the step of contacting them first. For tracking purposes, you might want to CC their ISP with this initial request, to document that you did indeed attempt to resolve the issue with them first, although this is not required. If you do decide to do that, SEOmoz’s ISP is HopOne Internet Corporation. The appropriate email to use for these matters, according to HopOne’s AUP, is abuse@hopone.net, and their fax is (604) 608-2953.

If after a reasonable amount of time, say, 24 hours, they still have not removed your sites information, then you can consider sending a formal DMCA letter to their ISP, HopOne. The requirements for such a letter are very specific, and are laid out in 17 U.S.C. § 512(c)(3), ” Elements of notification”. A sample DMCA notice for this purpose might look something like this:

To: abuse@hopone.net
Subject: Notice of Copyright Infringement
The copyrighted work at issue is the the entire set of links appearing on my domain at {www.mydomain.com}, each comprised of their respective URLs, anchor texts, and attributes, including both those constituting my websites navigation, as well as those linking my website to other websites on the Internet. While I acknowledge than an individual url in and of itself may not be copyrightable, I maintain that the set of links residing on my website taken as a whole or in sections do in fact comprise a structure that is unique and my own property.

The freely accessible URL where my copyrighted material is located is accessed through the gateway page located at http://www.seomoz.org/linkscape . Since the interface that is displaying my content is only visible via an http POST request, it is necessary to enter my domain {www.mydomain.com} into the text box presented, and then press the button labeled “GO”, in order to view the infringing material. Note that while this does demonstrate the existence of the infringing material being used on the server, it is only the one open to the general public without paying a fee, although this request is for the removal of the information from the index completely, including from areas accessible only to paying members of the website.

The contact information for the company of the infringing website, as indicated by their Contact Us page, is as follows:
Office: (206) 632-3171
Fax: (206) 338-3797
sitesupport@seomoz.org
SEOmoz.org
1221 E. Pike St., Suite 200
Seattle, WA 98122

I can be reached at {your@email.com}, or via telephone at {your telephone number}. My mailing address is {your full mailing address, including street and number, any apartment number, city, state, and zip code}.

I have a good faith belief that use of the copyrighted materials described above as allegedly infringing is not authorized by the copyright owner, its agent, or the law.

I swear, under penalty of perjury, that the information in the notification is accurate and that I am the copyright owner or am authorized to act on behalf of the owner of an exclusive right that is allegedly infringed.

At your earliest convenience, please respond to this letter at my email address listed above, and let me know what actions have been taken to resolve this matter. Thank you.

My electronic signature is below:
{Put Your Name Here}

Bottom line is, it would be nice if Rand would simply step up to the plate and actually be the nice guy he wants everyone to believe that he is. Until such time as that actually happens, however, as sad as it may be, this may be our only recourse to keep him from using our information without consent.

Right in the heat of the argument, someone decided to drop a comment on my blog that struck me as a little odd for some reason:

So who’s behind dotnetdotcom.org? “few Seattle based guys” “Trust us” ? WTF!? Why are there absolutely no names on that site? – some guy called smallfish

I had looked at that site before when Rand had released all of the info as to where the data from his tool actually came from. I had dismissed it, since Rand was claiming to have 30 billion pages in his index. The download on this site was only for 3.2 million pages out the initial 11 million pages that they had collected so far, what they were calling “the first part” of their index.

Since right at that moment Rand and I were arguing about whether or not Linkscape actually had a bot of it’s own that had collected the pages in their index, it hit me. “Aha!”, I thought. “Rand is probably going to reveal at some point that they actually own the DotBot. I mean, being able to say that you collected 11 million of the pages is better than having not collected any of them, right?”

So, I trotted off to dotnetdotcom.org to take a second look, just in case that turned out to be what was happening. Once I got there, I notice that something was different. When I had visited the page on Friday, these were the stats I saw:

(Click to enlarge.)

Saturday night, however, when I went to look, this is what I saw:

(Click to enlarge.)

That’s right… smack in the middle of an argument between Rand and myself, where he was insisting that he owned a bot capable of spidering an index of the size he was boasting, one of the sources he listed (the one that no one knew who the owners really were) jumped 7 billion pages in size. Talk about your random coincidences.

Let’s take a peek behind the scenes here, and see exactly how they managed to accomplish this truly humongous task. Viewing the source on the page, we can see that the counter is driven by a Javascript routine:

(Click to enlarge.)

What it does is it starts with the date that the DotBot went online, which is June 10th, 2008, calculates the number of seconds between then and now, and uses that as the starting point for how many pages it has spidered so far. It then counts up the display at one page per second.

While obviously this would only be an estimate of how many pages were crawled, this is a perfectly reasonable way to do it. For a smaller, non-Google sized company, with a server dedicated to nothing but crawling web pages, grabbing a page a second is about what we would expect. However, what they did next is another story altogether. How did a company capable of spidering a page a second around the clock get such an amazing boost in capacity? When we view the current Javascript, we see this:

(Click to enlarge.)

All they did was add in 7 billion pages to the start number, and added in a proportional boost to the other factors they are displaying as well (domains, robots.txt, and “clogged tubes”). They even left the clock counting up at 1 page per second.

Now, to put that in perspective… 7 billion pages at 1 page per second would in fact take 7 billion seconds to spider. This is not counting any processing or indexing time, this is just the collection of the raw data itself. That is:

116.6 million minutes
or
1.94 million hours
or
81,018 days
or
221 years (not counting leap years or time travel, of course)

in order for them to actually gather all of those pages. And they’re claiming that they did it in 4 months.

Uh huh. Right.

I figured even if my hunch about this being related to the Linkscape issue was wrong, it’s still noteworthy that a company that is self professed to be worried about making the internet “as open as possible” would be trying to pull a fast one like this.

While this is a huge breach of netiquette as it pertains to crawlers, at least today Rand finally announced that they are now disclosing their sources for data. In fact, this was how he worded it to the community:

we are now disclosing our sources for data – Rand Fiskin, SEOmoz CEO and really, really open guy

Better late than never, right?

The thing is, as I was looking over the list of bots that you would need to block in order to prevent mozzers from gathering your data, I noticed this subtle, easy to miss pattern in what they were listing. You have to look really, really close to see it, and the untrained eye might never see it at all, but luckily, eventually, I did see it for myself:

That’s right folks, if you do decide to keep moz out by blocking all of the big guys (Google, Yahoo, MSN, Ask, Amazon, and Alexa), the lesser known guys (Dotnetdotcom, Grub, Page-Store, and Exalead), and that one fictional guy they threw in there (Gagablast), you still won’t have them blocked. Fear not though… after much work, I finally figured out that Rand was indeed true to his word, and that they did in fact release enough information to block the bots. All you have to do is add the following lines to your robots.txt, and you’ll be golden*:

User-Agent: *and other data sources
Disallow: /
User-Agent: Additional crawls*
Disallow: /

See? I got ya covered!

Seriously though, despite the fact that Rand and Co. are still less than forthcoming about all of the bots that are used (I’m guessing he doesn’t actually know, to be honest), something much more revealing is highlighted in this information, namely, they lied about having their own crawler.

Let’s take a quick review of some statements Rand made in the initial announcement about the tool:

• Our crawl biases towards having pages and data…
• As others who’ve invested energy into crawling the web…
• our crawl biases towards this “center”…
• Our process for crawling the web…
• Moving forward, we’ll… invest in better and faster crawling…
• In comparing our crawls against the engines…
• we’ll be releasing more information about our crawl…

You also have statements made by moz employee Nick Gerner like this:

• we’re crawling everything we can…

You even have him claiming bullshit like this:

We do prioritize the crawl according to pages we think are important. For now, and probably for the foreseeable future we’re going to rely on link endorsement to make that decision. Make good content, get good links. Keep it publicly available. We’ll get there soon enough – Nick Gerner, moz employee

That’s right, not only did they claim that they were crawling the web, they wanted us to believe that they prioritized how they crawled based on an importance algorithm!

I have to admit, Rand, it’s pretty bold to basically admit this late in the game that you guys lied through your teeth and grossly misrepresented the facts, just so you could appear to have accomplished a much bigger task than you actually did, all in the name of getting more money from webmasters. That’s a much bigger admission than saying you cloaked your bot, if you ask me. Gratz on coming clean.

The link between financial troubles and psychological problems is well documented. In the U.S. the seminal example is the Great Depression, when the suicide rate jumped from 14 to 17 for every 100,000 Americans. And today, with the threat of recession looming large, the price we pay physically may skyrocket as well. – Chris Cuomo

The Great Depression certainly wasn’t a funny time in our history, and the current financial market really isn’t a laughing matter either. While we aren’t quite as bad (not yet, anyways) as we were back then, if things did get to the point where people started killing themselves due to their circumstances I hardly think it would be something we should joke about in the mainstream media. However, one company, Woot.com, apparently thinks it is.

I was performing my daily search to see how bad Google (NASDAQ: GOOG) was doing. As others have noted, they aren’t doing so hot. This I pretty much expected. What I didn’t expect was the AdSense ad making fun of the situation off to the right:

(Click to enlarge.)

The ad reads, “Before you jump out that window, why not spend your last remaining dollars at Woot?” Considering that the ad is showing on a stock quote search, there is no room for mistaking their message. Pretty fucking tacky, Woot. Don’t get me wrong, it is funny… but really not appropriate advertising for mainstream media. Considering all the press Google has gotten in the past for accidental inappropriate ads (like like Ebay selling Black People, and low prices on Child Brides at Amazon.com), I’m surprised that they are letting something like this slide.

I swear, you can only find this stuff in the St. Petersburg, FL area. Someone showed me this recently on the Pinellas County Jail Inmate search:

From: http://www.pcsoweb.com/Inmate/

I mean, seriously, wtf?

Daniel Scocco said it best on Daily Blogging Tips last year, in his post titled, “Put Honesty and Integrity Above Everything Else“:

One slip and you will ruin your reputation for good. And there are no erasers on the Internet. – Daniel Scocco

Obviously, this is true for more than just what you put out there on your blog. It extends to all forms of communication on the internet that are indexed and searchable: Twitter, Plurk, Digg/Sphinn/Mixx/blog comments… and of course, forum posts.

Recently Michael Martinez wrote a blog post having to do with some experimentation dealing with multiple links from one page to another, and whether they all passed weight or only the first one counted (the second scenario is what mine and others’ testing showed). The results of his test aren’t really that important, and contain more words than substance, but at the end of his post he felt the need to include the following assertion:

Mr. VanDeMar has been bashing me on various occasions for at least a year. He made the comment on the SEO Scientist blog that you can safely ignore me because 9 times out of 10 I’m wrong. Frankly, after being flamed and attacked by him numerous times on SEO Refugee I just stopped trying to respond to his nonsense. People who resort to unprofessional behavior and poison pen campaigns in order to win arguments will never impress me as being knowledgable or trustworthy. Anyone who wants to believe Michael VanDeMar’s assertions about me, PageRank, or anything else is more than welcome to. But I will certainly appreciate people ignoring all the flame-bait and insults and reaching their own conclusions (about how often I may be right or wrong) based on the evidence I provide. – Michael Martinez

I did in fact make the statement that 9 times out of 10 he is wrong about what he says. This was based on interactions with him personally over the past 2 years or so, and a random sampling of posts or comments he has made that other people have seen fit to show me. I do not read his blog on a regular basis, so there is always a chance that my estimation of his bs percentage is slightly off… but I have yet to see anything to make me feel that I would be off by much.

The “flaming” that Michael refers to in his commentary started on SEO Refugee Forums a couple of years ago, because apparently trying to get him to clarify statements he makes is an offense against his sensibilities, and grounds for him to start acting like an asshole. Among the claims I was trying to get clear on were concepts such as:

• You can have off-page optimization without having links.
• You can have competitive phrases that hardly anyone else is trying to rank for.
• All links have seo value (said on his blog) and link building is useless (said in the forum) are both true statements.

When faced with these questions, Michael’s response was to immediately get belligerent and very, very wordy, as if just piling on more words and obfuscating the issues would prove his point. It was extremely similar to the tactic that Rand Fishkin used the following year when confronted about his own words, with one very clear distinction… it really and truly did appear that Michael Martinez had somehow convinced himself that his statements made sense.

I commented on Michael’s blog, trying to point out the truth to him, again, about what was actually said back then. Of course, as with most emotionally challenged people, having evidence to refute the reality they have built up in their minds is simply unacceptable, so he did delete the comment. Since I had a strong feeling he would do that, I went ahead and saved it first:

*Ahem*

Mr. VanDeMar has been bashing me on various occasions for at least a year. He made the comment on the SEO Scientist blog that you can safely ignore me because 9 times out of 10 I’m wrong. Frankly, after being flamed and attacked by him numerous times on SEO Refugee I just stopped trying to respond to his nonsense.

See, Michael, it’s complete and utter bullshit such as that causing you and I to not get along. I’d say you were simply lying through your teeth, except that I honestly think you’ve deluded yourself into believing what you post. Not that I expect you to actually allow this comment to go live, but here is the exact “flaming” you are referring to.

Started here, when I tried to get you to clarify some statements that you made and then you got belligerent, after which you refused to reply to me and started referring to me as a troll:

http://www.seorefugee.com/forums/showthread.php?p=35745

Continued here, where I disagreed with an assertion you tried to make, that Toolbar PageRank, because it was a rounded value and not an exact figure, could not be passed on:

http://www.seorefugee.com/forums/showthread.php?p=46585

From that point on, in a very troll-like manner, you simply refused to back up anything you said. Now, of course, this being the internet, all of our conversations are still out there for everyone to see… so of course, feel free to “debunk” this claim by through links or exact quotes you feel are relevant. You’ll find my questioning your claims on how someone can have off-page optimization that has nothing to do with links. You’ll see where I pointed out that you made this claim in a forum post:

I’ve seen the insults. I’ve seen the disagreements. I’ve seen the attempts to box me with meanings for my words that I didn’t associate with them. But I haven’t seen anything to persuade me that there might be a reason to care about backlinks or to believe they are relevant to SEO.

which is a statement you made a mere 3 weeks after making this claim on your blog:

Just because people have engaged in cheap link building strategies with low-value sites doesn’t mean that all links from unrelated pages are harmful or unhelpful. Quite the contrary: most such links do help, and in many cases they help better than so-called “relevant links from relevant pages”.

You’re trying to call me a troll for pointing out your completely meaningless and contradictory statements? Michael, in case you didn’t know it, your behavior is the very definition of “troll”. You come to the forums and other people’s blogs, spewing this nonsense, and you do it with the attitude that other people must be idiots. You come off insulting people from the get go.

It’s obvious to me that your brain is cooked, and that you should be pitied… but you make it very hard to emphasize, Michael, when you behave like an ass.

Why don’t you go ahead and let this comment go live, with the links and quotes intact, and allow people to decide for themselves if your claims about me are true, Michael. – Michael VanDeMar

So, Michael, if you truly meant what you said about people drawing their own conclusions about you based on the evidence you provide, then maybe you should consider not trying to actually hide some of the evidence. Just a thought.