homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum


 5:49 am on Sep 11, 2008 (gmt 0)

I take it the "GoogleT5" is Google Transcoder version 5? - - [10/Sep/2008:21:15:58 -0400] "GET /scripts/menu.js HTTP/1.0" 403 - "http://www.apassion4jazz.net/jazz_styles.html" "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GoogleT5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; InfoPath.2; Tablet PC 2.0)"

This got blocked because of the "User-Agent:" part which I filter due to SPAMers pounding my input forms.



 8:51 am on Sep 11, 2008 (gmt 0)

No clue, but I boot anything claiming to be a browser with "HTTP:" in the user agent, and the "User-Agent:" thing as well. I whitelist the top 4 SEs first since they have HTTP paths in their user agents and then anything else with HTTP goes into the trash.

Yes, I'm a big meanie ;)


 9:29 am on Sep 11, 2008 (gmt 0)

The GoogleT5 part is apparently a new version of the Google Toolbar.

The Mountain View IP may well be a publicly accessible tool of some sort.

The User-Agent prefix always gets the thumbs down from this little meanie.



 9:37 am on Oct 18, 2008 (gmt 0)

This is a visit of Google Toolbar used component of tapestry 5. Do a Google search by: [google.com...] you will find the full Information.


Ahamed Bauani


 1:44 pm on Oct 21, 2008 (gmt 0)

Based on the behaviour I've seen from the UA starting with "User-agent", many of the sessions look like the AVG LinkScanner security pre-fetching we discussed recently.

Because of this, I've been serving a small page with a note about our site not supporting pre-fetching. The page includes CSS and image references which are never fetched by these user-agents. However, there is frequently another session immediately following from the same IP address, where the user-agent and browsing behaviour is completely normal and human-like.

However, after taking a look at Tapestry, I've modified the logic to block any UA starting with "User-agent" unless it starts with "User-agent: Mozilla/4.0 (compatible; MSIE 6.0;". This would block the UA being discussed here based on the MSIE version -- without requiring a "T5"-specific pattern, although it might be a good idea to include such a pattern.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved