homepage Welcome to WebmasterWorld Guest from 54.205.59.78
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
Google Analytics - How to tell if traffic is real or bots crawling?
olimits7




msg:3797255
 4:29 am on Dec 1, 2008 (gmt 0)

Hi,

I setup Google Analytics on my website, and I've been seeing that my website gets hits based on Google Analytics.

However, my question is how can I tell if the traffic that is shown on Analytics is real or just traffic caused by bots crawling?

Thank you,

olimits7

 

tonynoriega




msg:3798560
 9:14 pm on Dec 2, 2008 (gmt 0)

If i recall correctly, Analytics does not track any bots, spiders, crawlers...etc..etc...

if you are looking for that information about bots, spiders..etc...your server logs would be your best bet.

incrediblehelp




msg:3799288
 5:23 pm on Dec 3, 2008 (gmt 0)

Your right Tony it doesnt. You need access to your servers raw logs and tool that can read them.

olimits7




msg:3801224
 9:39 pm on Dec 5, 2008 (gmt 0)

Ok, thank you...

I guess it's fine then to not show the bots in google analytics; at least this tells me any visits I get are actual traffic and not bots.

cgrantski




msg:3801553
 1:30 pm on Dec 6, 2008 (gmt 0)

Well, you could still get some bots. The reason why most spiders and bots don't show up in most javascript-based data collection is that most spiders and bots ignore links to images on the pages they are spidering. And the javascript data collection method relies on a request for an image ... in the case of Google Analytics it's "_utm.gif." So if a spider is programmed to make those requests, it will show up in your Google Analytics data and in your reports. Google Images' bot is an example. I don't think Google Analytics does any hard-coded filtering that looks things up on some internal list of bots. Like other analytics programs, it relies on spiders not bothering with images.

My information could be obsolete. Let me know if you know otherwise.

coaster01




msg:3801574
 3:02 pm on Dec 6, 2008 (gmt 0)

How about bots that don't register as bots?

I have some "unknown" in my stats (not Google's - haven't signed up yet) and I think they're tentatively identified as bots because their behavior matches typical bot behavior, though they don't identify themselves as robotic user agents and they don't hit the robots.txt file.

cgrantski




msg:3801595
 4:13 pm on Dec 6, 2008 (gmt 0)

Yeah, there are thousands of those out there.

leadegroot




msg:3801788
 12:07 am on Dec 7, 2008 (gmt 0)

Google Analytics is based on javascript
The very large majority of bots don't process javascript, so don't show up for GA.

cgrantski




msg:3801831
 3:39 am on Dec 7, 2008 (gmt 0)

Is that still true? Bots and spiders seem to be able to follow javascript navigation just fine these days. I know that a year or two ago, javascript-based links stopped bots, but not so much any more. Is it possible that bots can handle javascript-based links but not the javascript that's in analytics tags?

leadegroot




msg:3801850
 5:31 am on Dec 7, 2008 (gmt 0)

Who knows?
I'm still seeing large numbers of bots that obviously don't run javascript, but there will always be some.
Its going to be site dependant as to which bots you get on your site.

As to whether the number is and will increase? Well, is there a major benefit to bots of getting the javascript from a page? Unsure.

enigma1




msg:3801904
 10:00 am on Dec 7, 2008 (gmt 0)

The very large majority of bots don't process javascript, so don't show up for GA.

This is not true. Or perhaps there is a misunderstanding with the word "bot".

Consider this example, someone creates a very simple plugin for FF that all it does it inserts a meta-refresh tag for a page the browser downloads. Now this meta tag will trigger a periodic refresh of the page and will be recorded as a human action. The browser could allow jscripts so it will pass through the GA filters but in reality this represents automation or what I could call a bot. Of course there are lots of other implementations but you get the idea.

Motives for it? Lots, say for instance you have a marketing campaign in place like ppc.

jrafique




msg:3802119
 9:47 pm on Dec 7, 2008 (gmt 0)

Very simple answer is this that Analytics does not track any bots, spiders, crawlers.

enigma1




msg:3802408
 9:08 am on Dec 8, 2008 (gmt 0)

Very simple answer is this that Analytics does not track any bots, spiders, crawlers.

This is like saying, you can tell whether a visitor is behind an elite proxy or not. The truth is, you cannot tell.

lokesh




msg:3825137
 9:05 am on Jan 13, 2009 (gmt 0)

Any analytic company/script can not guaranteed you for 100% accurate and perfect result. Wait for future.............

cgrantski




msg:3825264
 1:03 pm on Jan 13, 2009 (gmt 0)

This is how I understand it, and I hope it adds the right amount of details to some of the generalities that are being stated here.

Bots may not RUN javascript in the usual sense, but many PARSE javascript for links and visit those links. Google's bot has done this for a long time. (Besides, remember that most good js-tracking systems have a 'noscript' line too.)

However most bots do not follow links that are for file types identifiable as images. This, not a javascript-following capability, is why people think javascript-tag tracking is "immune" to bots - javascript tagging generally relies on a request for a gif. Bots could easily see and follow these if they are programmed to do so, but they are usually programmed to ignore image links.

Some bots do request image file types, of course. It's a matter of how they are programmed, for example whether the programmer wants it to avoid being detectable as a bot because it doesn't request images. The analytics world had a major incident with one of these last year. Or, the bot might be an image collector, period.

oddsod




msg:3826193
 4:02 pm on Jan 14, 2009 (gmt 0)

You may find this thread [webmasterworld.com] useful.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved