homepage Welcome to WebmasterWorld Guest from 54.211.34.105
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Reasons for using Googlebot user agent?
DiscoStu

5+ Year Member



 
Msg#: 4100609 posted 7:29 pm on Mar 18, 2010 (gmt 0)

What are the reasons for wanting to occasionally switching to the Googlebot user agent? I know people do it to check for cloaking, but are there other reasons? Just curious to hear how people use it...

[edited by: incrediBILL at 8:48 pm (utc) on Mar 18, 2010]
[edit reason] fixed typo [/edit]

 

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4100609 posted 8:53 pm on Mar 18, 2010 (gmt 0)

Snooping people do this to see how Google sees your website, mostly for SEO purposes.

Some sites only send text to Google, no formatting.

Some sites load their text with extra keywords for Google to see.

However, smart sites like mine validate Googlebot and will boot you to the curb for faking so you'll never see what I'm showing Google, never, ever.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4100609 posted 1:12 am on Mar 19, 2010 (gmt 0)

However, smart sites like mine validate Googlebot and will boot you to the curb for faking so you'll never see what I'm showing Google, never, ever.

+1

sonjay

10+ Year Member



 
Msg#: 4100609 posted 2:30 am on Mar 19, 2010 (gmt 0)

At least one site (and we probably all know which one I'm talking about) shows Googlebot answers to questions, but only shows the questions to real people, along with a link to sign up for a paid subscription. Unless you tell your browser to send a Googlebot UA; then you get the answers too. That site does not validate Googlebot.

jmccormac

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



 
Msg#: 4100609 posted 9:25 am on Mar 20, 2010 (gmt 0)

Scrapers use this in an attempt to bypass controls. Some sites will whitelist the large search engine bots but apply access controls and limits to ordinary browsers.

Regards...jmcc

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved