Welcome to WebmasterWorld Guest from 188.8.131.52
I have had good success at understanding visitor behavior on my older Websites. But this new site is different and I don't know why.
I launched the new site around mid December. Low or no-money terms used and so relatively uncompetitive. Overture shows > 1000 searches per month for each phrase. I'm on page one in Google for five of those phrases. (I'm testing them for traffic).
The problem is 93% of my visitor's referrers show up as "none".
However, when I click my site's results from Google SERPs myself , using NetTracker, I can later view my own clicks through the site - no problem, and I see the referrer (Google) and the search term.
Does not list any search terms either, except the ones I searched for myself. I can tell it was me by the search frequency.
However, Webalizer does list IP addresses from "Google English" in IP form, and these are not apparent in Netracker.
Webalizer also tells me I have had visits from 276 sites which would include Google spiders, search, and I guess user host addresses. MSN and Yahoo don't know I exist yet.
Here are examples of hosts not by IP address but reverse lookup:
Comcast Cable Communications - (My IP Address)
Asia Pacific Network Information Centre
RIPE Network Coordination Centre (Amsterdam)
Massachusetts Institute of Technology
Japan Network Information Center
TELUS Communications Inc. (BC)
Cox Communications Inc.
NetTracker support page says referrer = "None" for the following reasons:
1. You are not logging referrer information in your log file.
I am logging referrer information. In fact I just checked a few minutes before posting this. NetTracker told me the search term, the referrer, my IP address, and every page I visited and the path taken, within a few seconds of performing the Google search.
2. The visitor typed the URL of your Web site directly into the browser or used a bookmark to your site.
Unlikey, because the site is brand new. Nobody knows I'm there unless they find me in Google.
3. The visitor's Web browser did not pass referrer information to your Web server.
That is the part I don't understand. How can that happen? Is it a likely cause?
Is there any other reason that all my referrers (except when I search in Google myself) show up as "none"?
Could it be DNS lookup activity and I'm not getting any search traffic at all? Why would they come back more than once then? I don't see this on my older sites...
I have been trying to figure this out (on-and-off) for almost two weeks now...any ideas anyone? Is it something really simple that I don't know about?
joined:Mar 13, 2005
In addition, it is possible you need to turn on extended tracking. Thou it will vary from panel to panel, on mine the basic tracking is 'Generate Web Logs' which is ON but also I have an extended tracking function called 'Analog Web/FTP Log Analyzer' which, since I use a server-based stats program, is not required but perhaps in your case it is.
While some stats require only basic generation, others require the more resource-intensive method as well, and I hear say 3rd-party trackers are especially prone to this peculiarity.
Also, Nettracker correctly reports the stats from one of the other sites under the same account. I can account for the referrers on that one.
I just checked the CP:
Error Log = OFF
Transfer Log = ON
Webalizer = ON
Referrer Log = ON
Again, Netracker will correctly report my own visit through the site with my IP address and my search terms, but the other 389 visitor's referrers are classified as "none".
It could mean that these visitors are not finding me through Google search engine, but then I wonder how they know my new site and why would they visit?
Several have made more than one visit, but the logs show they hit the homepage and leave again.
I'm think about installing AWStats. Would AWStats give me information at the same resolution, i.e.Identify visitor's search term and referrer, date and time of visit, and path through site?
3. "The visitor's Web browser did not pass referrer information to your Web server."
The usual suspects:
* Norton Internet Security (NIS) and/or Norton Personal Firewall (NPF) with "Privacy Control"
* An increasing number of Firefox extensions masking referrer info
Why? Because all of those can cloak a user's browsing history, a.k.a. "referrer" info (famously misspelled "referer" inside computers).
Thing is, in addition to messing up stats, any/all of the above can *really* goof up CGIs and mod_rewrites, etc., requiring any sender of a POST request (message board post; form e-mail, etc.) to be 'on' the server. Because with no referer info, the server doesn't know who's where and thus rightly responds, "Nope. Not gonna let you do that."
For me, it boils down to server security v. individual security -- a delicate balance! (But seeing as how it's my server, I opt to protect that, and then go extra distance to redirect and then personally explain the POST thing to thwarted posters. After they uncheck Norton's PC and/or uninstall certain of FF's add-ons, all's well.)
But I digress:) Suffice it to say you'll probably see less and less referer info as time goes on, and as Norton and/or FF usage goes up, sorry.
Suffice it to say you'll probably see less and less referer info as time goes on, and as Norton and/or FF usage goes up, sorry.
Of course, besides how many users are blocking HTTP headers, how many users have JS turned off is also a good question. Using both log files and JS tagging gives you the most complete data.
Considering the site is only a couple of weeks old, I would suspect most of the hits are bot activity.
I am pretty sure that it is not Googlebot and Slurp type of stuff. I can easily id that type of activity.
My site keywords are in an area where other sites might use a visual verification (but most do not) to counter bots. My knowledge of this type of bot (if that is what you would call it) is close to nil.
If these bots were of the type that could artificially inflate overture keyword results, it could further explain the problem.
I did not mention above that I launched two other sites about two weeks before this site for different purposes. These ones are as dead as doornails, per my expectation, with no activity at all, other than indexing.
So if the answer is “bots”, then it may be more to do with the sector I’m in.
Maybe I need to get some low-tech, low-competitive keywords that hopefully bots don't care about, and perform the same test and see what happens.
Using both log files and JS tagging gives you the most complete data.
That is something I have never thought about – I need to look into it more, thanks Larry.
We do have the ability to throttle these kinds of things IF they are identifying themselves. Frequently, these kinds of sites use a tool that masks their identify by just requesting the feed with a blank user-agent string (for example, instead of sending up FeedDemon/1.0 or Googlebot, they just send up nothing). The problem with banning a blank string is that there are a bunch of perfectly valid home grown RSS readers out there that also send blank strings, and finally, these guys usually bounce around from IP address to IP address.
The above quote was used in reference to plagiarism and content theft.
So far, it best explains the results I see on my own new site. So now I wonder is this the hallmark of a scrapper visit? I suppose it is - so something to keep in mind for the future.
I wonder how they found me so quickly?
Blank User Agent Strings - Yes (93% of visitors)
Bounce from IP address to IP address - Maybe
My Web-Site in potential scraper territory - Yes
I use NetTracker Lite too.
Whenever I see alot of activity from a visitor with "none" as the referer, it usually turns out to be a bot. They don't appear in the spider logs, they look like regular visitors except that they go through many pages very quickly and often.
I usually end up blocking the IP address in htaccess if they don't stop within a day or two.
I make up about 20% of those visitors through my own visits and using utilities etc.
540 unique IP addresses visited once. 20 IP addresses visited twice. 12 IP addresses visited between 3 and 9 times.
It is still a bit of a mystery.
I am doing some upgrades and I will be adding some features that might generate feed back, ie adsense. It will be interesting to see what happens!