Forum Moderators: open
I started noticing a month or two ago that Yahoo was doing better in the mornings but that Google was back ahead by noon.
For the last nearly two weeks, Yahoo has been consistently ahead of Google so that now Yahoo is about 3:2 and sometimes even 2:1 in hits compared to Google.
Now, none of this seems to make any difference in total hits to my site. And, I know that virtually all Yahoo hits I am getting are Google indexed. Still, I find it interesting that this started at about the time we (along with many, many others) dropped from 6 to 5 in the rankings. By the way, the drop made no difference to us since competitors seemed to drop the same in proportion. As a matter of fact, I can't find any incoming links better than 5 and some used to be at 6 or 7.
Anyone have any ideas/theories?
Are you talking about more than one site? What industries and/or target markets?
I just had a horrible thought. If Yahoo's Google index is a little behind Google, I wonder if I am about to see a drop in total hits? In other words, Google drops me in rank, Yahoo moves ahead in hits, Yahoogle catches up and drops in hits as well.
Just a guess.
The problem seems to be the latest version of Funnelweb. When I use a different log traffic analyzer, I get more normal results.
Just my two pennies worth.
Stats are nearly always very inaccurate. ISP's are getting better at proxy caching dynamic or static pages, it is in their interests to do so. Here in the UK we have about 250 major isp servers. Once one of our sites starts getting 5000+ uniques per day the static pages and now many of the dynamic ones are getting cached on nearly all these servers, so we begin to see a maximum of 20 odd referrers per isp server. Further traffic is difficult to detect.
In other words, you may be getting more traffic than you think. Also this plays havoc with visitor/pageview ratio's. If your home page is easily cached but internal pages are very dynamic, you will under read for home page but be accurate for internal page views. I fight a daily battle showing clients that in fact each person does not look at 5 pages of their site.... but perhaps an average of one page per visitor, which can have huge implications..... like their site is in fact not as popular as they thought!
Mhes's points are great too.
We tend to focus on "outcome" variables (enquiries per day, newsltter signups per day, revenue per day) to assess sites as well as process variables (search engine positioning, page veiws, uniques, Alexa ranking, incoming links etc)
Outcome variables are much better at assessing te value of a website as they focus on the strategic reasons your web is there. Process variables are still important, as they give clues (and only clues, circumstantial evidence usually) to WHY the trends are as they are...
But in the end surely the outcome variables are the most important by far, because that is more integrated with the business and the role the web plays in that business. Sometimes they are more dispiriting than process variables, as we can "fudge" process variables better. Outcome performance variables provide more hard cold facts...
I havent noticed any changes to Y!/G percentages, but we update our scripts often. But far more important is we are getting more sales, and enquiries! That's what counts, and soon we may start looking into the morass of more unreliable process variables to try to work our why. When we have time.. Mind you, if our sales or enquiries go down we would be doing it much quicker!
Chiyo, you are right we all look at the bottom line (of the bank statement) first. But, I believe working out how you got there is almost as important.
As long as you stick to reasonably well targeted traffic, the visitors should be proportional to the inquiries, sales & profit, taking you back to the SERP's correlating to the visitors (depending upon how sticky you are and how much alternative advertising you do).
With regard to the actual topic of the post;) I haven't seen any difference between Y! & G ratios since the week Yahoo switched from using the Y! directory.
I have seen a big increase in MSN (true Ink) referrals over the last few months but that's a whole different story ;)
The caching stuff here was good reading, too. I'm used to AOL doing it, but hadn't really even contemplated other ISP's doing the same thing.
G.
42.05% Google
24.42% Yahoo
15.08% MSN
7.99% AOL
2.49% AskJeeves
1.44% Netscape
1.11% Altavista
0.93% EarthLink
0.85% Overture
0.71% Excite
0.63% Lycos
0.55% Looksmart
0.43% IWon
0.36% ATT
0.3% CometWebSearch
0.15% C-Net
0.14% Mamma
0.11% ixquick
0.07% Freeserve
0.05% Kanoodle
0.05% Teoma
0.04% About
0.01% Profusion
0.01% Alexa
0.01% 7Search
0.01% AllTheWeb
0.01% Webfile
Compared with my data Yahoo stayed the same. MSN and AOL are slowly creeping up.
Further to my proxy cache findings. Not quite on theme but perhaps useful in determining Google traffic.
Isp's love to cache because it saves them time and money. I suspect he U.S. has many more isp's than in the UK so this may not have quite the same impact, but here in the UK it is screwing up stats big time. We also run a few New zealand sites and with only a few isp's over there the effects can be dramatic.
It came to our notice when we started sending a big UK site about 1000 visitors per day. They used industry standard stats software and said we were sending 200 odd people per day. There overall traffic is in the region of 30,000 per day so within hours everyday their home page is cached (static page) on almost every major server in the UK. Visitors from us from about 11a.m onwards were picking up the cached pages from the isp with no request going through to the company server..... until someone requested a very dynamic internal page. So their stats were showing a ratio of 15 pages to every homepage visitor, and they believed it! We sent them the ip addresses of all 1000 visitors we had sent and after they looked at their raw logs, sure enough, there they all were, showing up for deep pages, with no reference to an external referer. The scary part of it was that someone was assuming most of their traffic was coming from directly typing in their url so deduced it was their TV ads pulling in the traffic.... wrong!
We then changed the link to their site from "www.theirsite.com" to "www.theirsite.com/?id=www.oursite.co.uk
This worked quite well, though some isp's still cached this link request. Now we put a timestamp on the link request, so the url requested looks like www.theirsite.com//?time=1042207607 which is totally unique numbers throughout the day as the time changes. They now see every ip we send them.
So, the moral to this story is that if your stats software is showing 300 hits from google and 100 from yahoo this ONLY means that you PROBABLY get 3 times more people from google than yahoo. The small print in these stats packages will say this, though it is well hidden, because in effect they are admitting the information they provide is pretty useless. Depending on factors such as how many isp's are sending you traffic, how cachable your pages are and how busy your site is will effect this figure. If your visitor numbers are low, none of this will really effect the stats figures, but in the UK, once you are over around 1000 per day, the effect starts to kick in.
So, why is yahoo showing more than google? Possibly because the links from yahoo on the Google search results are cache busting. Have a look at them, they are weird. If they are then every stats is being screwed towards them.... clever stuff.
Some people think putting 'Nocache' commands on their pages will stop their pages being cached.... it only stops some browsers caching the page, not the isp proxy cache which does not parse html and will ignore it. The only reliable way to stop your pages being cached is via the http header on the server...making dynamic pages out of date after 1 second or something like that, but I suspect isp's will hate you doing this.
I'm no expert in this field so I hope the above makes sense.
What/where are the raw data logs that can be used to check validity of my statistics server results?
Hate to think that I have been chasing geese instead of targetting customers.
My Yahoo v. Google hit count turned out to be a glitch in the way a particular analysis program was reading the logs. Running a different program, I was back where I was before with statistics virtually the same as Percentages reported: 42% 24% 15% etc.
I agree with Percentages also about MSN. I have seen it come up and passing AOL.
Wednesday and Thursday have been the best traffic days I have had since the site opened, so I am doubly confident that it was a software problem. I rank high with Yahoogle and get most of my traffic from search engines.
My main purpose in using traffic analysis is to see what people are searching for and what brings them to my site. Based on what I find, I have added many pages or tweaked existing pages to fit in with key terms. Some pages I added were in Google within 2 days placed #1 for that key phrase. So, I know it works if you make it work.
The points made about stats are very good points to keep in mind. You can't just rely on what any analysis tells you. You have to keep in mind all the variables mentioned, including caching. From now on, for starters, I will be using at least two programs for analysis (Funnelweb and Absolute Log Analyser, if anyone is interested) and I will be taking a look at the raw log from time to time to be extra sure.
Answer - Possibly Yes!
I have directory sites which deliver traffic. I know how many people click the links, I record the ip addresses. 90% of the recipricants do not understand how their stats work, and/or use stats that resolve figures to isp ip addresses and not individual ip's. In other words, the stats see 20 people come from an isp server but the software is only set up to register and count the first six digits of each ip, so resolve the 20 into 1 visitor. Most stat packages are not designed for high volume accuracy, but were designed for either monitoring low numbers or 'comparative' information for guidance. Add to this the proxy server cache issues and most stats become a very vague snapshot with huge flaws.
e.g. (extreme but possible)Your stats show 5000 visitors from the U.S and 50 from New Zealand. You assume most of your traffic is from U.S.? .... wrong.... Your pages were being cached by the top 3 New Zealand isp's and 30,000 people looked at them.... they never needed to request a page from your server once the initial page requests went through.
There are still many site owners who believe 'hits' are visitors, or 'requests' are visitors, or 'hosts served' are visitors.... they can all mean very different things. Sites that claim 'audited traffic' is usually nonsense, unless every page is uncachable and software annalysis is using the raw logs in a very detailed set up.... which in my experience is extremely rare!
Basically if your pages are being cached you have no idea how many people see them, or where they came from. Your raw logs (files on your server) will list the ip's more accurately in most cases, but still not give you the full picture. Page views stats are the most misleading.
The origin of your traffic is very difficult to determine, along with the 'paths' people follow through your site.
How do you make decisions?
IMHO Visitor feedback, gut feeling and then a quick look at the stats you have for a very rough guide!
Tracking hits to cached pages
[webmasterworld.com...]
So my devotion to data analysis may be misguided.. Too bad, I always felt at home in the statistics. This does explain why I could never get a good correlation between website activity and actual sales.
There are so many threads in these forums that dissect the fine details of their site statistics - are they all fooling themselves?
I'm going to back track a bit!
I love statistics as well, and I think that realising that there are problems with the information given makes it all the more interesting. If you factor in all these variables it gives you an edge. I am sure there are wrong decisions being made by believing the stats at face value. But if you look more closely at raw logs and figure it all out the advantages over your competitors can be huge. The exciting bit is that you can discover something they don't realise. In other words, site statistics are a goldmine, as long as you are taking in the whole picture, which includes gut feelings, user feedback and experience. We analyse our site statistics every week and have made some significant discoveries. There will be a correlation between website activity and sales, and the stats will show it if you factor in the inacuraces.
Camster,
You could be right, but what about the 17% of users who turn javascript off? I'm not sure but this could cause a problem. Also, these images have to load before a count is registered, which on a big page may not happen before the user clicks through to the next page or off the site. If a visitor comes in on a deep page and never goes to the home page, will this cause problems? Or what about browsers with the images turned off? Using non cachable gifs is a good idea but I suspect other factors kick in..... worth thinking about though.
>>the 17% of users who turn javascript off?<<
Mhes.. where did you get this stat from? I know there are several estimates but this seems the highest i have seen.
Your right, I think I must have made it up! (It was 1 o'clock in the morning) :( On reflection 17% is not probable and I must be way out.
Bigdave in another current thread says:
"I have heard estimates that 7%-13% of users surf with JS off. And at least on my site 16% of the surfers browse with something other than Internet Explorer."
This is probably closer to the mark, but somewhere I'm sure I heard it could be higher, because people hate the pop ups driven by javascript, or perhaps because, as bigdave suggests, the javascript will fail in about 16% of cases, either because of incompatible browsers or turning Javascript off.
The general population with JS turned off accounts for less than 5% of all users :)
MHes, you have always been right on the ball, until now, I guess we can forgive anyone this once....LOL ;)
I browse with several different browsers, including NS 3.02 for testing purposes, and it's amazing to me to see how many different sites FAIL with anyting other than IE. Some of these sites even sell Linux computers!
I am not totally anti-JavaScript, but if you use it, you should go out of your way to make sure thay you are not giving you user a bad experience.
Test your pages with Moz, opera an NS 4.7. if you are using click on URLs to pop up an information window, do it with <a target="_blank"> instead of JS which is bloced by just about all popup blocking software out there. Popup blockers have to account for at least a few more percent of your users.
If you make your living off the web, you should be going out of your way to make sure that no one leaves your site because your site is broken. And if you site doesn't at least work with NS 3.02, your site is broken, not the browser that the user has chosen to use. It may not look as nice in 3.02, but it should at least work.
Who said that? Accurate anyway. I used to analyse stats till I was blue in the face, holding my breath for that insight to come! OK, so it is useful:), but now I mainly look at ROI. The investment can be financial, or time, both are relevent, as I must earn money at a fair rate even on my own site.
I allow for some learning on my site too lets me off the hook a bit :)
I do this with one simple metric, I know the referer of
70% of all sales, both online and offline. The other 30% I cannot guess at, so I have to largely ignore them, and analyse the rest. Many are returns also.
Anyone else do it this way? Saves a lot of worry about stats!
George
Now this gives me something I can work with. Great! Anyone able to help buid a list of typical problems?
1. The data are mostly distorted by "visitors" looking at cached copies and not being logged as referrals to the home site.
2. The data that is recorded can be considered an accurate log of visitor activity for ALL people not seeing a cached version. (?)
3. We do not know how large the total count of people seeing the site content may be, or what portions are seeing a cached/non-cached version.
4. The missing referrals will likely be clustered into particular groups such as by country or isp.
5. Cross checking referral sources from actual sales or customer contacts is a way to peg the statistics to some real world facts.
6. What other built-in problem areas do we know about?
