homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Is it Googlebot or a user coming from Google search?

 6:10 pm on Jun 24, 2011 (gmt 0)

I have a php script which tracks users coming from search engines. It was working perfect, recently it is showing me 125K+ searches daily. This website is fairly new and I am not expecting that huge traffic from search engines within weeks of launch! (I know I am not that much lucky!).

I debugged this issue and emailed the global variables to myself. In less than 30 seconds I received over 100 emails. I am copying few snippets here.

[HTTP_USER_AGENT] => Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1

[HTTP_REFERER] => http://www.google.com/url?sa=t&source=web&cd=10&ved=0CH4QFjAJ&url=http%3A%2F%2_FMY_PAGE_URLs&rct=j&q=keyword1%keyword2%keyword3%20keyword4&ei=_csETpM06uKIAvqp7M0N&usg=AFQjCNGMXcID-G0b2KwV9-jqGVtSfaYmpA

From referral url it appears some one is searching from Google but i guess this is the Google robot. I can't understand If it is Google robot then why is it not showing in user agent?

Any clue whats cooking?

[edited by: tedster at 8:07 pm (utc) on Jun 24, 2011]



 2:35 am on Jun 25, 2011 (gmt 0)

Hardly Google!

Northwest Open Access Network NOANET-BLK2 (NET-64-146-128-0-1) -


 12:17 pm on Jun 25, 2011 (gmt 0)

Are they bad robots?


 1:02 pm on Jun 25, 2011 (gmt 0)

I've this range denied from 2008 when bot failed to comply with robots.txt and crawled pages, also probing maliciously for open directories.

64.184.179.zz - [16/May/2008:02:29:29 -0500] "GET / HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; ....../1.0 )"

I've a notation from 2009 (after making an initial reference it's not generally my practice to keep record of subsequent visits)

64.184.179.zz - - [20/Jul/2009:01:29:20 +0100] "GET / HTTP/1.1" 403 1159 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3"

Each webmaster must determine what is beneficial or detrimental to their own website (s).

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved