Welcome to WebmasterWorld Guest from 54.161.157.73

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Google Analytics - How to tell if traffic is real or bots crawling?

     
4:29 am on Dec 1, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:May 10, 2006
posts:591
votes: 0


Hi,

I setup Google Analytics on my website, and I've been seeing that my website gets hits based on Google Analytics.

However, my question is how can I tell if the traffic that is shown on Analytics is real or just traffic caused by bots crawling?

Thank you,

olimits7

9:14 pm on Dec 2, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 8, 2006
posts:1232
votes: 0


If i recall correctly, Analytics does not track any bots, spiders, crawlers...etc..etc...

if you are looking for that information about bots, spiders..etc...your server logs would be your best bet.

5:23 pm on Dec 3, 2008 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 1, 2003
posts: 91
votes: 0


Your right Tony it doesnt. You need access to your servers raw logs and tool that can read them.
9:39 pm on Dec 5, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:May 10, 2006
posts:591
votes: 0


Ok, thank you...

I guess it's fine then to not show the bots in google analytics; at least this tells me any visits I get are actual traffic and not bots.

1:30 pm on Dec 6, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 14, 2004
posts:1107
votes: 1


Well, you could still get some bots. The reason why most spiders and bots don't show up in most javascript-based data collection is that most spiders and bots ignore links to images on the pages they are spidering. And the javascript data collection method relies on a request for an image ... in the case of Google Analytics it's "_utm.gif." So if a spider is programmed to make those requests, it will show up in your Google Analytics data and in your reports. Google Images' bot is an example. I don't think Google Analytics does any hard-coded filtering that looks things up on some internal list of bots. Like other analytics programs, it relies on spiders not bothering with images.

My information could be obsolete. Let me know if you know otherwise.

3:02 pm on Dec 6, 2008 (gmt 0)

New User

5+ Year Member

joined:Dec 2, 2008
posts: 36
votes: 0


How about bots that don't register as bots?

I have some "unknown" in my stats (not Google's - haven't signed up yet) and I think they're tentatively identified as bots because their behavior matches typical bot behavior, though they don't identify themselves as robotic user agents and they don't hit the robots.txt file.

4:13 pm on Dec 6, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 14, 2004
posts:1107
votes: 1


Yeah, there are thousands of those out there.
12:07 am on Dec 7, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 27, 2003
posts: 1642
votes: 0


Google Analytics is based on javascript
The very large majority of bots don't process javascript, so don't show up for GA.
3:39 am on Dec 7, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 14, 2004
posts:1107
votes: 1


Is that still true? Bots and spiders seem to be able to follow javascript navigation just fine these days. I know that a year or two ago, javascript-based links stopped bots, but not so much any more. Is it possible that bots can handle javascript-based links but not the javascript that's in analytics tags?
5:31 am on Dec 7, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 27, 2003
posts: 1642
votes: 0


Who knows?
I'm still seeing large numbers of bots that obviously don't run javascript, but there will always be some.
Its going to be site dependant as to which bots you get on your site.

As to whether the number is and will increase? Well, is there a major benefit to bots of getting the javascript from a page? Unsure.

10:00 am on Dec 7, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:Apr 30, 2007
posts:1394
votes: 0


The very large majority of bots don't process javascript, so don't show up for GA.

This is not true. Or perhaps there is a misunderstanding with the word "bot".

Consider this example, someone creates a very simple plugin for FF that all it does it inserts a meta-refresh tag for a page the browser downloads. Now this meta tag will trigger a periodic refresh of the page and will be recorded as a human action. The browser could allow jscripts so it will pass through the GA filters but in reality this represents automation or what I could call a bot. Of course there are lots of other implementations but you get the idea.

Motives for it? Lots, say for instance you have a marketing campaign in place like ppc.

9:47 pm on Dec 7, 2008 (gmt 0)

New User

5+ Year Member

joined:Dec 7, 2008
posts:5
votes: 0


Very simple answer is this that Analytics does not track any bots, spiders, crawlers.
9:08 am on Dec 8, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:Apr 30, 2007
posts:1394
votes: 0


Very simple answer is this that Analytics does not track any bots, spiders, crawlers.

This is like saying, you can tell whether a visitor is behind an elite proxy or not. The truth is, you cannot tell.
9:05 am on Jan 13, 2009 (gmt 0)

New User

5+ Year Member

joined:Dec 20, 2008
posts:24
votes: 0


Any analytic company/script can not guaranteed you for 100% accurate and perfect result. Wait for future.............
1:03 pm on Jan 13, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 14, 2004
posts:1107
votes: 1


This is how I understand it, and I hope it adds the right amount of details to some of the generalities that are being stated here.

Bots may not RUN javascript in the usual sense, but many PARSE javascript for links and visit those links. Google's bot has done this for a long time. (Besides, remember that most good js-tracking systems have a 'noscript' line too.)

However most bots do not follow links that are for file types identifiable as images. This, not a javascript-following capability, is why people think javascript-tag tracking is "immune" to bots - javascript tagging generally relies on a request for a gif. Bots could easily see and follow these if they are programmed to do so, but they are usually programmed to ignore image links.

Some bots do request image file types, of course. It's a matter of how they are programmed, for example whether the programmer wants it to avoid being detectable as a bot because it doesn't request images. The analytics world had a major incident with one of these last year. Or, the bot might be an image collector, period.

4:02 pm on Jan 14, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2005
posts:2259
votes: 0


You may find this thread [webmasterworld.com] useful.