Forum Moderators: DixonJones
What is the ratio of SE visits recorded / total visits for sites you monitor?
What tool do you use?
For 1 of my key sites I'm seeing this decline substatially and wonder if others are seeing the same.
Key site with 15K visits / month, my ratio was 52% in July, then 35% in Aug, then 30% in Sept, and 20% in Oct.
I use WT Log Analyzer 7.0c
Once we get a list of figures, then we can speculate as to what's going on.
I saw a substantial drop between January and March this year, but it has been kind of flat since then. Nothing unusual in the summer.
I assume that you have checked to see if your more productive keywords are now ranking you a lot lower now than they were in the early summer? That all by itself would account for the pattern.
Also, if you have added lots of third-party sites linking to you, and they're sending a lot of traffic, they will crowd the SEs' percentage down as their share gets bigger.
Also, check to see if any other stats have changed about the same time. Sometimes they will lead to surprising underlying causes.
The first thing I would do is compare a Referring Sites report (NOT SE reports) from before the change to one after this big shift you're talking about. Use a big chunk of time - at least a month of data. Look for large falloffs or increases, whether for search engines or not.
I think at least part of your problem is that you are using an oldish version of your analysis software. Every analysis package has files that it consults regarding whether to classify a given visit as a search engine visit or not, based on looking up the referring site. I in WebTrends' case, "browsers.ini" is the name of the file and there are two or three copies of it in your installation directory. (They all must be identical, by the way). Over time, this file will become more and more obsolete as the search engine landscape changes. Some search engines will actually change their domain names, and new important search engines might appear that aren't on your old list at all.
A big thing that happened in the analytics world was MSN changing its domain name a couple months ago. Instead of search.msn.com, it now uses livesearch, which will appear in logs as livesearch.* or search.live.*. So WebTrends, expecting search.msn.*, won't be finding it, and your numbers will go down.
Here is a link within WebmasterWorld for the discussion that first tipped me off that this was happening: [webmasterworld.com...]
Your problem, however, started appearing before MSN's September action, but it could be something similar.
If you want to bring your browsers.ini file up to date for current search engine identifiers, you can do some legwork on it. You'll need to look at your Referring Sites report. The Referring Sites report is more useful for this than the Referring Domains report. What I do is this: print out the full set of results for that report. Sit down with a highlighter and mark the ones that you know are search engines. Browse to the sites if you are uncertain. Once you have the list of search engines, print out the entire contents of browsers.ini, or at least the [Search Engines] section of browsers.ini. Sit down again and check each highlighted item against browsers.ini and mark those that aren't in browsers.ini. Then you have your list for updating. It won't be perfect, but it will be a huge improvement. The syntax for making additions is pretty obvious from examining browsers.ini.
In the current version of WT, the revised entry for MSN looks like this. I don't know what it will look like in a really old browsers.ini.
[MSN]
ID1=search.msn.
ID2=livesearch.
ID3=search.live.
KeywordIndicator1=q=&q
KeywordIndicator2=mt=
KeywordIndicator3=q=
And, Chewy, there is one more thing that could be screwing up your stats. It's kind of a long shot about whether it is important --- your internal table for referrer names could be full and not accepting anything new. It could have filled up a long time ago, in fact. If it's full, new players like "livesearch" will possibly be ignored in all analyses that consult the table of referrers. I don't remember how to examine or change the internal table size in LA 7, sorry. (Important explanation: internal table size is different from the length of the table displayed in reports. It's an internal database capacity question.)
Hope this helps, Chewy.
2. browsers.ini file seems to be "keywords.ini" and there is only 1 copy anywhere and the syntax is the same, so I will update that to account for MSN live etc. This is helpful but may only account for single digits of the % shift.
3. how would I go about looking for this "internal table"?
4. I clearly expect to be replacing WT 7, and am using Google Analytics as the low cost option. In part I'm trying to benchmark one against the other.
WT new versions are just way too expensive (or I am too cheap and annoyed with them) with ongoing maintenance contracts etc.
I'm just not sure Log Analyzer 7 included access to those internal tables in its administrative interface. It's prominent in recent versions, though. I'd say you don't need to worry about it at this point. Given the amount of traffic you reported for a typical month, I think you're safe.
Back to the numbers... please folks, anyone please post a few numbers as to what your ratio is.
In my case, I've now checked this with numerous tools and they all pretty much say the same thing.
On one site, Google Analytics shows 1300 out of 2200 visits to the index page shows "no data" in terms of what search terms were used. Of course, this could be "good news" in that people are typing in the domain...
900/2200 is 40%
I'm guessing these are browsers with JS and or cookies turned off.
What kind of ratios are others experiencing?
Somehow I'm trying to get a range - for instance is there a case where there are NO typeins or other ways to more clearly see what % of "no data" is a result of no-referrers due to browser settings, etc that obscure the search term?
(owner edit for clarity)
[edited by: chewy at 7:24 pm (utc) on Nov. 20, 2006]
no this is more likely users with referer logging turned off, of course only sophisticated users know how to do this, however 3rd party software may do this as the default setting
Is the advent of this increasing or decreasing? Is this substantial or on the margins? Are there a few or a lot of apps that do this?
I know the user that turns off referrers is on the margin, what I'm suss out is what is the current and future impact of these 3rd party apps (and or default browser settings?).
A - 32% down since spring (total traffic up - summer and fall banners pushed down the SE percentage)
B - 29% up since spring (ranks better lately)
C - 55% they have a good PPC campaign. PPC is about 30% of the 55%. There is some cannibalism happening.
D - 12% recognizable brand name, number of direct-types probably pushing down the very substantial SE traffic.
E - 40% they do a lot of email which pushes down the SE traffic
Other stats:
Generally, percentage of traffic with no referrer has been flat for many many months. No increase or decrease.
We had heard that some search engines were opening results in new browser windows resulting in empty referrer fields. We checked it out by going to many SE's and never found support for the rumor.
Percentages fluctuate as campaigns kick in and out, causing extra traffic that makes other pieces of the pie smaller by comparison. The underlying absolute numbers are much more stable and fluctuate according to ranks. We've looked at this many times and I feel confident about it.
We have been changing our SE definitions constantly. There are two new categories of what we define as search engines, and these categories do not get classified as such by the analytics packages. One category is ISP searches. Comcast's own search is a major player now, so is myway. They often get their feeds from Google etc. The other category is search/shop/review engines such as bizrate and shopper.com, and similar ones that are specific to a market niche like jewelryshopper.com. For some sites, these are big contributors. Most analytics programs show them in referrers reports and not in search engine reports. But we call them search engines and try to stay current.
