Strange traffic drop

Massive drop in "no referral" traffic in Urchin

[AG4Life]

I've recently been stumped by a problem on several of my websites, some hosted on different servers. They've all experienced dramatic traffic drops, but unlike previous situations, I can't tell why it is happening. I've been running these sites for more than 7 years now.

I'm using WHM/CPanel on two dedicated servers (at two different datacenters) - sites on both servers are seeing the same traffic drop. The sites vary from forums (vB, IPB) to a custom made CMS system, to Wordpress blogs. I have Urchin 5 installed on both servers to track traffic.

What is strange is that, looking at the Urchin stats at least, that drops in traffic is isolated to "(no referral)" traffic. If I remove "(no referral)" from the list of referrals, traffic remains unchanged and has actually increased. Search engine referrals are also up. On the other hand, non referral traffic for one particular site has dropped from 18,000 sessions (per day) in May, to 14,000 sessions last Thursday, and now only 11,000 sessions yesterday.

My "(no referral)" traffic has always been a bit high, 45% in May, 40% last week and now 32%, so I suspected perhaps a problem with Urchin. But the traffic drop is clearly happening, as seen through Webalizer and also significant drops in revenue.

As some of the sites that have seen the biggest drops share the same DNS host, I'm leaning towards that as a possible explanation. There are also sites on the same server that appears unaffected, so I'm inclined to rule out the (two different) servers as the problem. And with two different datacenters located in different cities, I'm also ruling that out as a possible cause.

Does anybody know what could be causing this? What would constitute so much no referral traffic, and why would it drop so suddenly? Traffic with referrals not dropping also seem very strange to me.

[AG4Life]

I've changed DNS hosts and there's no effect (as expected), so I think DNS can be ruled out too. I will then try another server with Ensim as opposed to CPanel, but I'm not too hopeful of the results.

[AG4Life]

Also just to add that traffic was normal all the way until May 29th 2008. Since then, it's gone from bad to worse.

[AG4Life]

Sorry to keep replying to my own thread, but I think I can now rule out server software related issue (CPanel/Ensim, or various versions of Apache/php/mySQL).

Having looked at the stats again in Urchin, and taking into account some strange stats in May (perhaps related to the AVG LinkScanner incident), it looks like these two stats are the only ones to have dropped:

1. (no referral) down from 15,500 per day to 9,800 sessions. All other referrers are normal
2. Under "browser", "Mozilla Compatible Agent" (5.0) down from 8,400 to 2,200 sessions. All the other browsers, like IE, Firefox, Opera, Safari ... are normal.

A drop of almost exactly 6,000 sessions for both, so perhaps both drops are related somehow. So what kind of browser or system creates no referrals and uses the Mozilla Compatible Agent (5.0) as the user agent?

[aleksl]

what kind of browser or system creates no referrals and uses the Mozilla Compatible Agent (5.0) as the user agent?

umm... a bot?

[g1smd]

Most bots can't and don't run the JavaScript code that feeds the analytics data.

[AG4Life]

I'm not using UTM (the Urchin .js script, I'm using Apache log data analysis), so that could explain the bots. I'm also suspecting that these traffic was artificial as the drop was quite sudden, although 6,000 sessions/visitors per day does sound like a lot. It appears that the user agent string only features the "Mozilla/5.0 (compatible;" part, but not the part that identifies the bot/browser (eg. "Mozilla/5.0 (compatible; IE 6.0".

I've checked the stats for bots and there doesn't seem to be any major drops or increases in this area, and there are no unexplained Mozilla/5.0 traffic in this area either (just Slurp, Googlebot ...).

I'll do more investigating, but I suspect this is a hard one to solve.

[bwnbwn]

big bill just posted a read on this as your using sessions and that is flawed. Read [webmasterworld.com...] I would suspect it was artifical bot traffic and not real.

[maximillianos]

Do you run Adsense? If so, how has your Adsense stats fluctuated during this time frame? If it has remained some what steady, you may have found your answer (bots).

[aleksl]

g1smd: Most bots can't and don't run the JavaScript code

really, hmmm? it would take me 30 minutes to build one. all you need is some code that opens your url in a browser for whatever reason, including artificially increasing user counts and ad displays, spying for trademark violations, etc.

bwnbwn, excellent post by incrediBill

[maximillianos]

Bots do run javascript. Particularly the Googlebot... I've watched it jump through my javascript spaghetti like a bat out of hell~! ;-)

[g1smd]

Run it, or just parse it?

[AG4Life]

Thanks for all the replies, that incrediBill post was extremely interesting and something that I've been thinking recently too. There has been somewhat of a drop in advertising revenue, but I will have to observe further as to if this is related to this drop in sessions or perhaps just an overall market situation (plus our server was caught up in the infamous The Planet datacenter explosion of June 2008, so it's been incredibly hard to track visitor/revenue patterns over the last few months, add to that the AVG thing as well).