homepage Welcome to WebmasterWorld Guest from 54.227.222.235
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Marketing and Biz Dev / Cloaking
Forum Library, Charter, Moderator: open

Cloaking Forum

    
Cloaking bots list of Ip and agents
anyone?
ildarius




msg:3631951
 2:27 am on Apr 22, 2008 (gmt 0)

Anyone knows where to find a list of ips and agents that are
used by bots?

I read the original post (Cloaking : Webmaster World Knowledge base excerpt.) here, but the links are dead.

 

volatilegx




msg:3632681
 8:40 pm on Apr 22, 2008 (gmt 0)

There are a number of resources for IP addresses and user agents of search engine spiders. You could try searching for "search engine spider ip addresses" at your favorite search engine.

ildarius




msg:3632845
 2:37 am on Apr 23, 2008 (gmt 0)

thank you, i did find quite a few pages, just thought may be you could recommend your preferred site for that (without breaking the forum rules of course)

incrediBILL




msg:3638981
 9:39 pm on Apr 30, 2008 (gmt 0)

There are a lot of lists out there but some of the lists also use legacy IPs now utilized in other ways, such as mobile proxies, etc.

When you get a list, run reverse DNS on the list and make sure all of the IPs clearly identify themselves as the crawler such as "crawl-*.googlebot.com." so you don't accidentally let obsolete IPs see the wrong pages.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Marketing and Biz Dev / Cloaking
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved