Last month I blogged about the fact that I had noticed that Google was playing around with delivering the SERP’s via AJAX. I pointed out that due to the way that referrers work, using AJAX to generate the pages would cause all traffic coming from Google to look like it was coming from Google’s homepage instead of from a search. This means in turn that analytics packages, including Google Analytics, would no longer be able to track what keywords searched on in Google were sending traffic to the webmaster’s websites. There was a bit of a buzz about it, and Google seemed to stop the testing shortly thereafter. Google’s only reply on the subject was “sometimes we test stuff”, to point to a post from three years ago that also said, “sometimes we test stuff”, to say that they didn’t intend to break referrer tracking, and that was it.
Shortly thereafter, the tests appeared to have stopped. People stopped thinking that Google was linking to them from their homepage, and in very short time the buzz died down about it.
Yesterday afternoon, someone pointed out to me that the subject came up again during Matt Cutt’s keynote address at PubCon South. According to Lisa Barone’s live blogging efforts the conversation went something like this:
Brett: How about the JavaScript test.
Matt: That was really funny. The team there only thinks about speed. They want to get the results back to users as quick as humanly possible. JavaScript makes the search results a lot faster. Suppose you do a search for flowers, as you’re typing flowers, they can do a query from the back end and fold search results right into the page. You’re still in Google.com and they can pull in the results automatically. It doesn’t give you the referrer. He says the team didn’t think about the referrer aspect. So they stopped. They’ve paused it until they can find out how to keep the referrers.
Ok, fine. So they didn’t know about the referrer issue (and didn’t give me credit for pointing it out to them before it was more than just a test, *cough* *cough*), so they stopped until they can figure out a way to fix it.
Less than 1 hour after reading that, I happened to notice an over-abundance of url’s in the serps that were getting redirected though Google’s url redirection service. For those who are unaware of what I mean by that, it is a tool that Google uses to enhance their own behind-the-scenes tracking of user behavior. You can see it most consistently when Google displays sitelinks. They redirect the clicks through their own tracking mechanism first, so that they can determine how many people are actually using those extra links. Instead of going directly to the page in question, it goes through a link like this:
http://www.google.com/url?q=http://www.bad-neighborhood.com/text-link-tool.htm&ei=in…
If you view one of those links using a header detector, you can see that after doing whatever tracking they do on their backend, they then redirect the user via a 302 Redirect onwards to the final destination page:
User-Agent used to fetch header:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008051206 Firefox/3.0HTTP/1.0 302 Found
Location: http://www.bad-neighborhood.com/text-link-tool.htm
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Date: Fri, 13 Mar 2009 15:44:42 GMT
Server: gws
Content-Length: 247
Occasionally you will see these Google redirects in the normal serps as well, although usually not. The thing is, I was seeing them on every search I performed. It struck me as odd, until I suddenly realized that every search was being done via AJAX:
Here’s the problem, Google. That will not fix the referrer issue, which is what the issue is with every non-Google analytics package that exists. Without that, then the traffic coming from Google cannot be accurately analyzed (unless, of course, the analytics program has access to whatever it is that Google’s redirect script is recording). For one, a 302 redirect passes on the referrer of the original page, not the one of the tracking script, and for another there are no unencrypted keywords included in your tracking urls. It doesn’t take pushing the test to the live servers to figure this out, either. The engineers had to have known this beforehand. This means that if Matt was correct, and Google did indeed stop the testing until they could make it work with analytics, then the only analytics package they were worried about AJAX serps working with is Google Analytics.
Hi Michael, I checked with some folks at Google about this. The redirection through a url redirector was separate from any AJAX-enhanced search results; we do that url redirection for some experiments, but it’s not related to the JavaScript-enhanced search results.
The solution to the referrer problem will be coming online in the future. It uses a JavaScript-driven redirect that enables us to pass the redirect URL as the referrer. This URL will contain a “q” param that matches the user’s query.
Matt, thanks for chiming in. Always appreciated.
As to the Javascript redirect, while I do know that does in fact work for generating a new referrer (as opposed to a 301 or 302 redirect), doesn’t that kind of defeat the whole purpose of doing AJAX serps in the first place? I mean, you said before this was why they were doing them:
For what you are describing (ie. using an interim “jump” page that contains the q= parameter and redirecting that page using Javascript), it seems to me that you are going to lose whatever extra microseconds you have shaved off with going with AJAX in the first place.
Also, I know that Lisa was live blogging and may have misquoted you, but your reply at PubCon made it seem like they were totally unaware of the referrer issue, and were going to stop until they had a fix in place. Since I have been seeing the /url?q= redirect continuously since I started getting the AJAX results last Friday, and others who aren’t getting the AJAX results are not seeing the redirect, it seemed as if the two were connected and that must be the “fix”. Any idea when they are going to start testing with the Javascript jump page?
“doesn’t that kind of defeat the whole purpose of doing AJAX serps in the first place?”
They’re trying to find a way to do it that’s still fast. If there were a good/clean way to preserve the referrer without the interim jump, I’d love to hear about it.
I think you’re in an experiment/bucket-test, which is what’s causing what you’re seeing. I don’t want to commit to when the JS jump page might start because the timing on that could change.
Excellent coverage of this topic, Michael. I guess I’m still trying to understand the point also. If Google is trying to return search results faster but has to use an interim jump page, it seems like a lot effort for nothing. Unless, of course, there are additional future advantages to delivering search results in Javascript that we haven’t yet discussed.
Daniel, I actually am not sure I understand that rationale even without the jump page. As one of the people that Google has consistently tested this on, I can tell you that the Google results being returned via AJAX are not appreciably faster than those returned by regular means. If the Google page was busy by default, and retrieving a new page the normal way meant loading tons of unnecessary extra source code, then I could see the justification, but Google isn’t really known for HTML bloat. Personally I think that the real reason they did this is because from a geek perspective AJAX is a way cooler delivery method, but that’s just my guess.
Either way, there really is no way to effectively preserve the referrer strings without a rewrite of all of the major browsers that exist today.
Nice thread. Has anyone reviewed the stability of Ajax since google rolled it out? I’m looking at a referrer issue for a large firm and I’m starting to think it’s a Google Ajax issue – for some URL’s. Summary: Google sends you to a proper URL w/ referrer info on the first click, each subsequent click on the google result is missing referrer info. How can I not find articles on this topic? – it appears to be a major issue.
I realize that this is probably a ‘feature’ of Ajax + browser configuration but Google is losing lots of credit for clicks – and firms are scared that they won’t be able to track their users referrers properly.. not good for business for either side.
Brendan, missing the referrer altogether is a separate issue, one that depends on which browser is being used. You find that alot with mobile devices, and as using them to browse the web becomes more and more popular you will start to see more visits without referrers. When Google glitches with it’s AJAX (which still happens from time to time), there is still a referrer, it just makes it look like the traffic is coming from Google’s homepage, as opposed to the result of someone making a search.
Michael,
Thanks for the reply. I understand the issues with the different browsers but there is clearly a major glitch with Google’s Ajax when using any popular browser. I would provide the details here but I don’t want to post company URL’s.
Here’s the problem in short – find any site that redirects its base url to a landing page (ie. http://blah.com redirects to https://blah.com/home.do)
Then do natural search in google for blah.com – the first time google will send you to http://blah.com which will then redirect you to https://blah.com/home.do as expected (all headers are present)
*you have to use a header tool like Live HTTP Headers in firefox or something to see this.
Go back to the google link and click on it again – it will not send you to http://blah.com – the ajax is smart enough to bypass the real URL (even though google lists it as http://blah.com) – it will send you directly to https://blah.com/home.do without the referrer info.
Clear your cace and go through the same process again and see that the referrer gets passed only on the first attempt every time.
Granted – I completely understand that Ajax code is ‘smarter’ and more efficient this way but the fact is that google has links on it’s search page and when you click them (perhaps for the 2nd time onwards) the process doesn’t forward the referrer information.
No need to go into the tech details why this happens, it’s not good for google and it’s not good for any company that is doing search engine analysis for marketing purposes.. I consider it a major glitch and goof on Googles part. Try the same thing in Yahoo or any other search engine and they don’t have this problem. Google’s logic does not fill the referrer field properly under a certain condition – and in my companies case it’s a major concern.