Last month I blogged about the fact that I had noticed that Google was playing around with delivering the SERP’s via AJAX. I pointed out that due to the way that referrers work, using AJAX to generate the pages would cause all traffic coming from Google to look like it was coming from Google’s homepage instead of from a search. This means in turn that analytics packages, including Google Analytics, would no longer be able to track what keywords searched on in Google were sending traffic to the webmaster’s websites. There was a bit of a buzz about it, and Google seemed to stop the testing shortly thereafter. Google’s only reply on the subject was “sometimes we test stuff”, to point to a post from three years ago that also said, “sometimes we test stuff”, to say that they didn’t intend to break referrer tracking, and that was it.
Shortly thereafter, the tests appeared to have stopped. People stopped thinking that Google was linking to them from their homepage, and in very short time the buzz died down about it.
Yesterday afternoon, someone pointed out to me that the subject came up again during Matt Cutt’s keynote address at PubCon South. According to Lisa Barone’s live blogging efforts the conversation went something like this:
Ok, fine. So they didn’t know about the referrer issue (and didn’t give me credit for pointing it out to them before it was more than just a test, *cough* *cough*), so they stopped until they can figure out a way to fix it.
Less than 1 hour after reading that, I happened to notice an over-abundance of url’s in the serps that were getting redirected though Google’s url redirection service. For those who are unaware of what I mean by that, it is a tool that Google uses to enhance their own behind-the-scenes tracking of user behavior. You can see it most consistently when Google displays sitelinks. They redirect the clicks through their own tracking mechanism first, so that they can determine how many people are actually using those extra links. Instead of going directly to the page in question, it goes through a link like this:
If you view one of those links using a header detector, you can see that after doing whatever tracking they do on their backend, they then redirect the user via a 302 Redirect onwards to the final destination page:
User-Agent used to fetch header:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008051206 Firefox/3.0
HTTP/1.0 302 Found
Content-Type: text/html; charset=UTF-8
Date: Fri, 13 Mar 2009 15:44:42 GMT
Occasionally you will see these Google redirects in the normal serps as well, although usually not. The thing is, I was seeing them on every search I performed. It struck me as odd, until I suddenly realized that every search was being done via AJAX:
Here’s the problem, Google. That will not fix the referrer issue, which is what the issue is with every non-Google analytics package that exists. Without that, then the traffic coming from Google cannot be accurately analyzed (unless, of course, the analytics program has access to whatever it is that Google’s redirect script is recording). For one, a 302 redirect passes on the referrer of the original page, not the one of the tracking script, and for another there are no unencrypted keywords included in your tracking urls. It doesn’t take pushing the test to the live servers to figure this out, either. The engineers had to have known this beforehand. This means that if Matt was correct, and Google did indeed stop the testing until they could make it work with analytics, then the only analytics package they were worried about AJAX serps working with is Google Analytics.
9 thoughts on “Google Re-initiates Testing of AJAX SERP’s With Faulty Proposed Fix”
Matt, thanks for chiming in. Always appreciated.
“doesn’t that kind of defeat the whole purpose of doing AJAX serps in the first place?”
They’re trying to find a way to do it that’s still fast. If there were a good/clean way to preserve the referrer without the interim jump, I’d love to hear about it.
I think you’re in an experiment/bucket-test, which is what’s causing what you’re seeing. I don’t want to commit to when the JS jump page might start because the timing on that could change.
Daniel, I actually am not sure I understand that rationale even without the jump page. As one of the people that Google has consistently tested this on, I can tell you that the Google results being returned via AJAX are not appreciably faster than those returned by regular means. If the Google page was busy by default, and retrieving a new page the normal way meant loading tons of unnecessary extra source code, then I could see the justification, but Google isn’t really known for HTML bloat. Personally I think that the real reason they did this is because from a geek perspective AJAX is a way cooler delivery method, but that’s just my guess.
Either way, there really is no way to effectively preserve the referrer strings without a rewrite of all of the major browsers that exist today.
Nice thread. Has anyone reviewed the stability of Ajax since google rolled it out? I’m looking at a referrer issue for a large firm and I’m starting to think it’s a Google Ajax issue – for some URL’s. Summary: Google sends you to a proper URL w/ referrer info on the first click, each subsequent click on the google result is missing referrer info. How can I not find articles on this topic? – it appears to be a major issue.
I realize that this is probably a ‘feature’ of Ajax + browser configuration but Google is losing lots of credit for clicks – and firms are scared that they won’t be able to track their users referrers properly.. not good for business for either side.
Brendan, missing the referrer altogether is a separate issue, one that depends on which browser is being used. You find that alot with mobile devices, and as using them to browse the web becomes more and more popular you will start to see more visits without referrers. When Google glitches with it’s AJAX (which still happens from time to time), there is still a referrer, it just makes it look like the traffic is coming from Google’s homepage, as opposed to the result of someone making a search.
Thanks for the reply. I understand the issues with the different browsers but there is clearly a major glitch with Google’s Ajax when using any popular browser. I would provide the details here but I don’t want to post company URL’s.
Here’s the problem in short – find any site that redirects its base url to a landing page (ie. http://blah.com redirects to https://blah.com/home.do)
Then do natural search in google for blah.com – the first time google will send you to http://blah.com which will then redirect you to https://blah.com/home.do as expected (all headers are present)
*you have to use a header tool like Live HTTP Headers in firefox or something to see this.
Go back to the google link and click on it again – it will not send you to http://blah.com – the ajax is smart enough to bypass the real URL (even though google lists it as http://blah.com) – it will send you directly to https://blah.com/home.do without the referrer info.
Clear your cace and go through the same process again and see that the referrer gets passed only on the first attempt every time.
Granted – I completely understand that Ajax code is ‘smarter’ and more efficient this way but the fact is that google has links on it’s search page and when you click them (perhaps for the 2nd time onwards) the process doesn’t forward the referrer information.
No need to go into the tech details why this happens, it’s not good for google and it’s not good for any company that is doing search engine analysis for marketing purposes.. I consider it a major glitch and goof on Googles part. Try the same thing in Yahoo or any other search engine and they don’t have this problem. Google’s logic does not fill the referrer field properly under a certain condition – and in my companies case it’s a major concern.