homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

Google Analytics - How to tell if traffic is real or bots crawling?

 4:29 am on Dec 1, 2008 (gmt 0)


I setup Google Analytics on my website, and I've been seeing that my website gets hits based on Google Analytics.

However, my question is how can I tell if the traffic that is shown on Analytics is real or just traffic caused by bots crawling?

Thank you,




 9:14 pm on Dec 2, 2008 (gmt 0)

If i recall correctly, Analytics does not track any bots, spiders, crawlers...etc..etc...

if you are looking for that information about bots, spiders..etc...your server logs would be your best bet.


 5:23 pm on Dec 3, 2008 (gmt 0)

Your right Tony it doesnt. You need access to your servers raw logs and tool that can read them.


 9:39 pm on Dec 5, 2008 (gmt 0)

Ok, thank you...

I guess it's fine then to not show the bots in google analytics; at least this tells me any visits I get are actual traffic and not bots.


 1:30 pm on Dec 6, 2008 (gmt 0)

Well, you could still get some bots. The reason why most spiders and bots don't show up in most javascript-based data collection is that most spiders and bots ignore links to images on the pages they are spidering. And the javascript data collection method relies on a request for an image ... in the case of Google Analytics it's "_utm.gif." So if a spider is programmed to make those requests, it will show up in your Google Analytics data and in your reports. Google Images' bot is an example. I don't think Google Analytics does any hard-coded filtering that looks things up on some internal list of bots. Like other analytics programs, it relies on spiders not bothering with images.

My information could be obsolete. Let me know if you know otherwise.


 3:02 pm on Dec 6, 2008 (gmt 0)

How about bots that don't register as bots?

I have some "unknown" in my stats (not Google's - haven't signed up yet) and I think they're tentatively identified as bots because their behavior matches typical bot behavior, though they don't identify themselves as robotic user agents and they don't hit the robots.txt file.


 4:13 pm on Dec 6, 2008 (gmt 0)

Yeah, there are thousands of those out there.


 12:07 am on Dec 7, 2008 (gmt 0)

Google Analytics is based on javascript
The very large majority of bots don't process javascript, so don't show up for GA.


 3:39 am on Dec 7, 2008 (gmt 0)

Is that still true? Bots and spiders seem to be able to follow javascript navigation just fine these days. I know that a year or two ago, javascript-based links stopped bots, but not so much any more. Is it possible that bots can handle javascript-based links but not the javascript that's in analytics tags?


 5:31 am on Dec 7, 2008 (gmt 0)

Who knows?
I'm still seeing large numbers of bots that obviously don't run javascript, but there will always be some.
Its going to be site dependant as to which bots you get on your site.

As to whether the number is and will increase? Well, is there a major benefit to bots of getting the javascript from a page? Unsure.


 10:00 am on Dec 7, 2008 (gmt 0)

The very large majority of bots don't process javascript, so don't show up for GA.

This is not true. Or perhaps there is a misunderstanding with the word "bot".

Consider this example, someone creates a very simple plugin for FF that all it does it inserts a meta-refresh tag for a page the browser downloads. Now this meta tag will trigger a periodic refresh of the page and will be recorded as a human action. The browser could allow jscripts so it will pass through the GA filters but in reality this represents automation or what I could call a bot. Of course there are lots of other implementations but you get the idea.

Motives for it? Lots, say for instance you have a marketing campaign in place like ppc.


 9:47 pm on Dec 7, 2008 (gmt 0)

Very simple answer is this that Analytics does not track any bots, spiders, crawlers.


 9:08 am on Dec 8, 2008 (gmt 0)

Very simple answer is this that Analytics does not track any bots, spiders, crawlers.

This is like saying, you can tell whether a visitor is behind an elite proxy or not. The truth is, you cannot tell.


 9:05 am on Jan 13, 2009 (gmt 0)

Any analytic company/script can not guaranteed you for 100% accurate and perfect result. Wait for future.............


 1:03 pm on Jan 13, 2009 (gmt 0)

This is how I understand it, and I hope it adds the right amount of details to some of the generalities that are being stated here.

Bots may not RUN javascript in the usual sense, but many PARSE javascript for links and visit those links. Google's bot has done this for a long time. (Besides, remember that most good js-tracking systems have a 'noscript' line too.)

However most bots do not follow links that are for file types identifiable as images. This, not a javascript-following capability, is why people think javascript-tag tracking is "immune" to bots - javascript tagging generally relies on a request for a gif. Bots could easily see and follow these if they are programmed to do so, but they are usually programmed to ignore image links.

Some bots do request image file types, of course. It's a matter of how they are programmed, for example whether the programmer wants it to avoid being detectable as a bot because it doesn't request images. The analytics world had a major incident with one of these last year. Or, the bot might be an image collector, period.


 4:02 pm on Jan 14, 2009 (gmt 0)

You may find this thread [webmasterworld.com] useful.

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved